Create an endpoint

Create an endpoint to serve a model behind an HTTPS address that your applications can call. You pick a framework first. The form then asks where the model comes from, which machine to use, and how many workers to run.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service. TIR shows Create Endpoint only to people who have the create permission.
• Credit for the machine you choose. On a prepaid account, an hourly endpoint needs credit for its launch, and a committed plan needs its full price. For rates, see E2E’s pricing page.
• Your model source ready:
  • For a Hugging Face model, a Hugging Face integration. See Create a Hugging Face integration.
  • For your own files, a repository that holds the model. See Upload model files.
  • For your own image, the image’s address.

Choose the framework

1. In the side menu, under Inference, select Model Endpoints, then Create Endpoint.
2. Search with Search Framework if you need to, choose a version under the framework’s Version, and select the framework.

To pick between frameworks, see Choose a framework.

Name the endpoint

Endpoint Name starts as a random name that begins with the framework’s name. Keep it or type your own: up to 50 lowercase letters and digits, with single hyphens between them, starting and ending with a letter or digit.

Say where the model comes from

Download Type is the first block for a framework that loads a model, and its choices depend on the framework. Follow the page for the one you use:

• Download from Huggingface: Deploy a model from Hugging Face.
• Link with Model Repository: Deploy a model from Model Repository.
• Container Configuration for Custom Container: Deploy a custom container.

Choose where the downloaded model is kept

Model Cache Storage is required, whichever Download Type you chose. It has two options:

• Local drive keeps a separate copy of the model on each worker, and the copy is lost when the endpoint ends. Set Storage size in GB larger than the model’s files. It starts at 100 and takes 10 to 2000.
• Shared file-system lets several workers share one download, so extra workers start faster. Pick an existing Shared File-system and a Cache path, or create one from the link. A Shared File System is its own product with its own billing; see TIR Shared File System.

The cache path starts as /mnt/models for a local drive and /shared for a shared file system, and you can keep it. For a Hugging Face download the field is named HF_CACHE path.

Choose the machine

In Select Computing Resource, open a tab and choose a machine:

• GPU for a single machine with GPUs. This is the usual choice for a language model.
• Multi Node for a model too large for one machine. Its name gives the node count after the GPU, such as “(2x)”.
• CPU for a small model that runs well on a CPU.
• Private Cluster to run on your own cluster; choose it under Select Private Cluster. You can also add a scheduling rule that keeps the endpoint on a node group, or off it. See TIR Private Cluster.

Each machine lists the memory of one of its GPUs, such as 80 GB. Choose a machine and size whose GPU memory holds the model with room for requests. In 16-bit precision, a model’s weights take about 2 GB of GPU memory for every billion parameters. An 8 billion parameter model needs about 16 GB, and a 70 billion parameter model about 140 GB, spread over several GPUs. The size under Configuration sets the GPUs, vCPUs, and RAM that one worker gets.

A machine marked Inventory not available, or a size marked Plan not available, is out of stock in the region right now. Choose another one, or try again later.

Under Instance Pricing, choose how you pay. On-Demand is hourly pay-as-you-go and suits development and changing traffic. A savings plan commits you to a term and suits steady, long-running use. TIR asks you to confirm the plan before it creates the endpoint; see Choose a committed plan. For the longer terms, select Show more pricing.

Set the workers

In Serverless Configuration, set Active Worker, the workers that are always ready, and Max Workers, the most the endpoint can grow to. For a fixed number of workers, set both to the same number. Scale to zero, the scaling policy, and the other worker settings are in Set workers and autoscaling.

(Optional) Open Advanced Settings

Advanced Settings holds Environment Variables and, for a framework version that has them, LLM Settings, which tune how the model loads and runs:

• LLM Settings covers data type, quantization, context length, and memory.
• System Settings covers scheduler, batching, concurrency, and logging. On vLLM it also holds the tool-calling settings an agent needs.
• Tokenizer Settings covers the tokenizer path, mode, and chat template.

Which groups there are depends on the framework version. TIR sends only the settings you change from their defaults. For environment variables, see Set environment variables.

Deploy the endpoint

Select Deploy Endpoint once every required field is valid. TIR opens the endpoint’s details and emails you, with the project’s Owner and Admins in copy.

Check that the endpoint is running

The endpoint starts in Deploying while its workers download and load the model. The details page refreshes by itself. When the state reads Running, every worker is ready for requests. A large model can take 10 to 15 minutes or more.

If creating the endpoint fails, the portal shows “Unable to create endpoint” or the reason; see Troubleshooting Model Endpoints. To send a first request, see Call an endpoint with a token.