Model Endpoints FAQ

For step-by-step fixes, see Troubleshooting Model Endpoints.

Regions, access, and frameworks

Which regions have Model Endpoints?

Delhi and Chennai. An endpoint belongs to one project and one region, so you see the endpoints of the project and region you are in. See Create an endpoint.

Who in my project can use Model Endpoints?

The Owner, an Admin, a Project Manager, and a Project Lead can manage endpoints. A Member needs a policy that allows the Inference service, because Model Endpoints comes under that service’s permissions. For a Member, each permission covers one kind of action:

• Read permission opens the list and every tab.
• Create permission lets a Member create endpoints and clones.
• Update permission lets a Member start, stop, restart, and update endpoints, and opens the Serverless Configuration and Workers tabs.
• Delete permission lets a Member delete endpoints.

The portal shows only the actions your role allows. See Create an endpoint.

What is the difference between a Model Repository and a Model Endpoint?

A repository stores model files. A Model Endpoint is a running service that answers requests using files from a repository. See Deploy a model from Model Repository.

Which frameworks can I choose?

You can choose vLLM, SGLang, NVIDIA Dynamo, PyTorch, NVIDIA Triton, Custom Container, TensorRT, or vLLM-Omni. Each has several versions, and the versions change over time. See Choose a framework.

Which machine families can an endpoint use?

The families a region can offer include H200, H100, A100, L40S, L4, CPU, and Multi-Node. Which ones you can choose depends on the framework and on the free capacity right now. For rates, see E2E’s pricing page.

Why can I not choose a machine or size on the machine list?

The region is out of that machine right now. A machine that is out of stock shows Inventory not available, and a size shows Plan not available. Choose another machine or size, or try again later. See Create an endpoint.

Can I run a very large model across several machines?

Yes, on the Multi Node tab of Select Computing Resource, which is available for vLLM, SGLang, and Custom Container. See Create an endpoint.

Models and creating an endpoint

Do I need a Model Repository to deploy a Hugging Face model?

No. You need a repository only for Link with Model Repository. A Hugging Face download works on its own. See Deploy a model from Hugging Face.

Do I need a token before I create an endpoint?

Yes, if the model comes from Hugging Face. For a Hugging Face download, most frameworks ask for an HF Token Integration on the form. To call an external endpoint later, you need an auth token. See Create a Hugging Face integration and Call an endpoint with a token.

What format does the Hugging Face model ID take?

The model ID takes the form <organization>/<model-name>. An ID in any other form shows “Use format: organization/model-name”. See Deploy a model from Hugging Face.

Can I deploy my own or a fine-tuned model?

Yes, if the model works with the framework you choose. Its dependencies must be in the image or the model package, and its files must follow the framework’s layout. Use Link with Model Repository, or Custom Container with your own image. Select Validate before you deploy. See Deploy a model from Model Repository. To keep a fine-tuned model’s files in a repository, see Can I keep a fine-tuned model in a repository?.

How do I link a repository to a Model Endpoint?

There are two ways. Select Deploy Model on the repository’s row in Model Repository and pick a framework. Or select Link with Model Repository while you create an endpoint, choose the repository, and enter the Model Path. A folder looks like /v1. An empty Model Path means the top of the bucket. The endpoint downloads the model from the repository when its container starts or restarts. See Deploy a model from Model Repository.

Why can my endpoint not find the model?

Check three things. The Model Path must match the folder layout, the files must be in the repository, and the bucket name must be right. Then read the endpoint’s logs. See Deploy a model from Model Repository.

Can I deploy a LoRA adapter?

Yes, on vLLM. Choose Link with Model Repository, then PEFT Model (for LoRA-based inference) under Model Type, and give the Base Model Name. The repository must hold the adapter’s configuration file, named adapter_config.json, and a file ending in .safetensors. See Deploy a model from Model Repository.

How do I turn on tool calling on a vLLM endpoint?

Open Advanced Settings on the create or update form. Under LLM Settings, in System Settings, turn on Enable Auto Tool Choice and pick a Tool Call Parser. The list includes hermes, mistral, and pythonic, and others for Llama and Qwen models. The settings depend on the framework version. SGLang has no such setting. See Build an agent with the OpenAI Agents SDK.

Are there limits on the values in LLM Settings?

Yes. A number setting must be at least 0, the maximum model length at least 1, and the GPU memory utilization at most 1. Only the settings you change from their defaults are sent. See Create an endpoint.

How much disk does the local drive cache give each worker?

You choose it. Storage size in GB starts at 100 and takes 10 to 2000 GB. Choose a size larger than the model’s files, or use a Shared file-system cache, which several workers share. See Create an endpoint.

Why does my endpoint stay in Deploying for a long time?

A large model can take 10 to 15 minutes or more to download and load, so Deploying for a while is normal. If it lasts much longer, read Deployment Events and Worker Logs. Then check the token, the model ID, the licence, and that the machine has free capacity. See View logs and events.

Can I reuse the name of a deleted endpoint?

Yes. You can use a deleted endpoint’s name again. See Delete an endpoint.

Custom containers

Can I run my own Docker image as an endpoint?

Yes. Choose Custom Container and give a public or private image. See Deploy a custom container.

Which port must my container listen on?

On an External endpoint, HTTP is fixed to port 8080 and gRPC to port 9000. On an Internal endpoint you choose the HTTP ports, except 80 and 8082. See Deploy a custom container.

Why does my custom container show Commands and Arguments boxes instead of Entrypoint Command?

The endpoint was created before April 10, 2026. It keeps the older Commands and Arguments boxes, and you type each as a list in square brackets, as View Command Examples shows. An endpoint created since then has Serving Framework, Entrypoint Command, and Arguments rows. See Deploy a custom container.

Can I change the container of a Multi Node endpoint on its own?

No. On a Multi Node plan, change the container with Update Inference, which opens the whole form. See Update an endpoint.

Calling an endpoint

Which address do I call?

The Root Endpoint on the endpoint’s Overview and on Sample API Request. Add the route you want to it, such as /v1/chat/completions. See Call an endpoint with a token.

Which token do I send?

An auth token from Personal Access Tokens, in an Authorization: Bearer <Auth Token> header. Generate token on the endpoint creates one if you need it. To call the endpoint, a valid token from the project is all you need, whatever your portal permissions. See Call an endpoint with a token.

Can I use the OpenAI client?

Yes, for vLLM and SGLang endpoints, and for Custom Container and Finetuned ones. Set the base URL to the root endpoint plus /v1/ and the API key to your token. See Use the OpenAI client and stream responses.

Can I stream the answer?

Yes. Add stream=True to the OpenAI client call. See Use the OpenAI client and stream responses.

Why does my request fail with a wrong model name?

The model name in a request is the Hugging Face ID, /mnt/models for a model from a repository, or peft-model for a LoRA adapter. The samples on Sample API Request use the right name. See Call an endpoint with a token.

Does the chat route work with any model?

No. /v1/chat/completions works for instruct or chat models. A base model needs a chat template in the request, or you can use /v1/completions with a prompt. See Call an endpoint with a token.

Does an endpoint support gRPC?

Yes. A Custom Container endpoint takes gRPC on port 9000 when you tick gRPC. TensorRT, PyTorch, and NVIDIA Triton endpoints show a gRPC tab on Sample API Request. See Deploy a custom container.

Is WebSocket supported?

Yes, for models whose server exposes a WebSocket route, for example a streaming speech model. Use the root endpoint with wss and send the token in the Authorization header. See Call an endpoint with a token.

Can I test in the portal without code?

Yes, with the Playground, for endpoints that serve chat models. See Test in the Playground.

How do I set up async inference?

Async is set up after the endpoint is created, on its Async Invocations tab. Choose the routes and a dataset for the results there. See Set up async invocation.

Is async invocation available on every endpoint?

No. Async invocation works on every plan except Multi Node. See Set up async invocation.

Which datasets can hold async results?

Only an Object Storage dataset of the project, in the endpoint’s region, that is ready to use. You can also create one from the setup panel. See Set up async invocation.

Workers and scaling

Can an endpoint scale to zero?

Yes, on an hourly plan other than Multi Node. Set Active Worker to 0 on the create form, or Minimum Workers to 0 on the Serverless Configuration tab. The endpoint scales down to 0 when requests stop arriving and shows Idle, and the next request starts a worker. See Set workers and autoscaling.

Which plans can scale to zero?

Only hourly plans other than Multi Node. On a committed plan and on a Multi Node plan the minimum is 1. On a Multi Node plan the form asks only for Active Worker, and that number is also the maximum. See Set workers and autoscaling.

Can I scale to zero with the Custom metric?

No. The Custom metric needs at least one active worker. The create form blocks Deploy Endpoint while the metric is Custom and Active Worker is 0, and the Serverless Configuration tab says that scaling to zero is not supported with a custom metric. Choose Concurrent Request Count or Request Rate per second to scale to zero. See Set workers and autoscaling.

Can I set Max Workers to 0?

No. To pay nothing, set the minimum to 0 on an hourly plan, or stop the endpoint. See Set workers and autoscaling.

How do I keep a fixed number of workers?

Set the minimum equal to the maximum. The minimum and maximum are the only way to set the number of workers in the portal. See Set workers and autoscaling.

How many workers can an endpoint have?

Up to 64, or up to 512 on a private cluster. See Set workers and autoscaling.

Does every framework offer custom metrics?

No. vLLM, SGLang, NVIDIA Dynamo, PyTorch, and NVIDIA Triton each offer a list under Custom Metric Name, and a Custom Container takes the name you type. TensorRT offers none. See Set workers and autoscaling.

What values can Target Value take?

A whole number from 1 to 100000, or from 1 to 100 for a CPU metric. Idle Timeout (in seconds) takes 1 to 1800, and Initial Cooldown Period (in seconds) takes 30 to 86400. See Set workers and autoscaling.

How long does scaling down take?

A few seconds to a few minutes, depending on the model size, the requests in progress, and how quickly the workers stop. See Set workers and autoscaling.

What if the region has no free capacity when the endpoint scales up?

The endpoint stays at the workers it has, and the endpoint’s creator gets an email for each scale-up that failed to get a machine. See Set workers and autoscaling.

What does Total GPUs mean?

It is the number of GPUs on each worker multiplied by the number of workers currently set. It reads 0 for a Stopped or Idle endpoint. See Set workers and autoscaling.

Billing

How is an endpoint billed?

Two ways. With Hourly, you pay for the time its workers run. With Committed, you commit to a fixed term chosen from a savings plan. The price follows worker time, however many requests the endpoint answers. A Model Repository is billed on its own. See How is a repository billed?. See Choose a committed plan. For rates, see E2E’s pricing page.

Hourly or committed, which should I choose?

Use an hourly plan for development and variable traffic, and a committed plan for steady, long-running use. A committed endpoint costs the same however much you use it. See Choose a committed plan.

What do I pay for on an hourly plan?

You pay for each worker for the time it is deployed, retrying, ready, or running. Time a worker spends waiting for free capacity, unavailable, or in error costs nothing. The bill follows the number of workers over time, so when autoscaling adds or removes a worker its hours start or stop at that moment. See Set workers and autoscaling.

Is there a minimum billing duration?

No, not on an hourly plan: you pay for the time workers run. A worker that runs 45 minutes is charged for exactly those 45 minutes. A committed term is the minimum for its reservation. See Choose a committed plan.

Do I pay for Max Workers?

No, not on an hourly plan. You pay for the workers that run, and Max Workers only caps how far the endpoint can grow. On a committed plan, workers above the committed count are billed at the hourly rate of the same machine. See Set workers and autoscaling.

Am I charged while the model downloads?

Yes, on an hourly plan. The time a worker spends starting up and loading the model is part of its running time. Stop the endpoint to end the charge. See Stop, start, and restart an endpoint.

Am I charged while the endpoint is stopped or idle?

No, not for compute. Compute billing stops while the endpoint is Stopped, and while it is Idle at zero workers. The Idle Timeout is different: idle workers stay running for it after the last request, and you are charged for that time. See Set workers and autoscaling.

Which account is billed?

The project owner’s account, whoever created the endpoint. See Create an endpoint.

Why can I not stop my committed endpoint?

A committed endpoint runs for its whole term, so only an hourly endpoint can be stopped. Use an hourly plan if you need to stop an endpoint when it is idle. See Choose a committed plan.

Can I change what happens when my committed term ends?

Yes. Select Update Inference, and choose again in the committed-plan confirmation:

• Auto renewal of this committed plan
• Auto start hourly billing for this machine
• Auto deletion of this committed machine

See Choose a committed plan.

Why can I not scale below my committed workers?

A committed plan’s worker count is the fewest workers the endpoint can run. A lower minimum is refused with “Cannot downscale below the committed plan’s worker count of <n>”. Keep the minimum at or above that count. See Choose a committed plan.

Do I get a refund if I delete a committed endpoint before its term ends?

No. The committed charge is paid up front for the whole term, so it is not refunded when you delete early. Deleting a committed endpoint ends it at once. Delete Service reminds you that the charges have already been deducted. See Delete an endpoint.

Is an endpoint on a private cluster billed by the endpoint?

No. You pay for the cluster’s nodes instead. See TIR Private Cluster.

Is a Shared File System cache billed separately?

Yes. A Shared File System is its own product with its own billing. See TIR Shared File System.

Does async invocation cost extra?

Yes. A small CPU worker handles the queue and is billed only while the queue has requests. The endpoint’s own workers are billed as usual. See Set up async invocation.

Where do I see what I used?

In Account Usage, with Inference Usage chosen under Resources. See Analyse usage.

Network and alerts

Can an endpoint reach other systems?

Yes: APIs, databases, object storage, and monitoring, as far as its network and security-group rules allow. A Reserved VPC IP gives private access. See Use a private network and a security group.

Does a security group open ports on my endpoint?

No. Adding a security group leaves the endpoint’s ports as they are. The group’s inbound rules only define which IPs can reach the ports the endpoint already has open. See Use a private network and a security group.

Which port does the private address use?

Port 8080, always. The Private Endpoint sample calls http://<vpc-ip>:8080, whatever port the container listens on, so the security group’s inbound rule is for TCP port 8080. See Use a private network and a security group.

Can I attach more than one VPC IP to an endpoint?

No. An endpoint takes one VPC IP at a time. The VPC IP must be in the endpoint’s region, and the VPC must use E2E-provided ranges. See Use a private network and a security group.

Which alerts can I set on an endpoint?

You can set any alert from Alert Management except an ephemeral-storage alert, which watches temporary disk space that is lost on restart. GPU temperature and GPU power alerts work only for GPU endpoints. See Monitor an endpoint.

Stopping, updating, and deleting

Can I stop an endpoint?

Yes, if it is on an hourly plan. A stopped endpoint keeps its settings and address, and Start Service brings it back. A committed endpoint runs until its term ends. See Stop, start, and restart an endpoint.

Can I copy an endpoint?

Yes. Clone Inference starts a new endpoint from the same settings. See Clone an endpoint.

Does deleting an endpoint delete my model files?

No. The Model Repository entry and its files stay as they are when you delete an endpoint. See Delete an endpoint.