--- title: "Model Endpoints FAQ" description: "Short answers about Model Endpoint frameworks, models, calling an endpoint, scaling, billing, and access, with links to the full procedure." content_type: "faq" product: "model-endpoints" platform: "TIR" url: "/tir/model-endpoints/faq/" --- # Model Endpoints FAQ For step-by-step fixes, see [Troubleshooting Model Endpoints](/tir/model-endpoints/troubleshooting/model-endpoints/). ## Regions, access, and frameworks ### Which regions have Model Endpoints? Delhi and Chennai. An endpoint belongs to one project and one region, so you see the endpoints of the project and region you are in. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ### Who in my project can use Model Endpoints? The Owner, an Admin, a Project Manager, and a Project Lead can manage endpoints. A Member needs a policy that allows the Inference service, because Model Endpoints comes under that service's permissions. For a Member, each permission covers one kind of action: - Read permission opens the list and every tab. - Create permission lets a Member create endpoints and clones. - Update permission lets a Member start, stop, restart, and update endpoints, and opens the **Serverless Configuration** and **Workers** tabs. - Delete permission lets a Member delete endpoints. The portal shows only the actions your role allows. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ### What is the difference between a Model Repository and a Model Endpoint? A repository stores model files. A Model Endpoint is a running service that answers requests using files from a repository. See [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). ### Which frameworks can I choose? You can choose vLLM, SGLang, NVIDIA Dynamo, PyTorch, NVIDIA Triton, Custom Container, TensorRT, or vLLM-Omni. Each has several versions, and the versions change over time. See [Choose a framework](/tir/model-endpoints/how-to/choose-a-framework/). ### Which machine families can an endpoint use? The families a region can offer include H200, H100, A100, L40S, L4, CPU, and Multi-Node. Which ones you can choose depends on the framework and on the free capacity right now. For rates, see [E2E's pricing page](https://www.e2enetworks.com/pricing). ### Why can I not choose a machine or size on the machine list? The region is out of that machine right now. A machine that is out of stock shows **Inventory not available**, and a size shows **Plan not available**. Choose another machine or size, or try again later. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ### Can I run a very large model across several machines? Yes, on the **Multi Node** tab of **Select Computing Resource**, which is available for vLLM, SGLang, and Custom Container. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ## Models and creating an endpoint ### Do I need a Model Repository to deploy a Hugging Face model? No. You need a repository only for **Link with Model Repository**. A Hugging Face download works on its own. See [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/). ### Do I need a token before I create an endpoint? Yes, if the model comes from Hugging Face. For a Hugging Face download, most frameworks ask for an **HF Token Integration** on the form. To call an external endpoint later, you need an auth token. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/) and [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/). ### What format does the Hugging Face model ID take? The model ID takes the form `/`. An ID in any other form shows "Use format: organization/model-name". See [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/). ### Can I deploy my own or a fine-tuned model? Yes, if the model works with the framework you choose. Its dependencies must be in the image or the model package, and its files must follow the framework's layout. Use **Link with Model Repository**, or **Custom Container** with your own image. Select **Validate** before you deploy. See [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). To keep a fine-tuned model's files in a repository, see [Can I keep a fine-tuned model in a repository?](/tir/model-repository/faq/#can-i-keep-a-fine-tuned-model-in-a-repository). ### How do I link a repository to a Model Endpoint? There are two ways. Select **Deploy Model** on the repository's row in **Model Repository** and pick a framework. Or select **Link with Model Repository** while you create an endpoint, choose the repository, and enter the **Model Path**. A folder looks like `/v1`. An empty **Model Path** means the top of the bucket. The endpoint downloads the model from the repository when its container starts or restarts. See [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). ### Why can my endpoint not find the model? Check three things. The **Model Path** must match the folder layout, the files must be in the repository, and the bucket name must be right. Then read the endpoint's logs. See [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). ### Can I deploy a LoRA adapter? Yes, on vLLM. Choose **Link with Model Repository**, then **PEFT Model (for LoRA-based inference)** under **Model Type**, and give the **Base Model Name**. The repository must hold the adapter's configuration file, named adapter_config.json, and a file ending in .safetensors. See [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). ### How do I turn on tool calling on a vLLM endpoint? Open **Advanced Settings** on the create or update form. Under **LLM Settings**, in **System Settings**, turn on **Enable Auto Tool Choice** and pick a **Tool Call Parser**. The list includes `hermes`, `mistral`, and `pythonic`, and others for Llama and Qwen models. The settings depend on the framework version. SGLang has no such setting. See [Build an agent with the OpenAI Agents SDK](/tir/model-endpoints/how-to/build-an-agent-with-the-openai-agents-sdk/). ### Are there limits on the values in LLM Settings? Yes. A number setting must be at least 0, the maximum model length at least 1, and the GPU memory utilization at most 1. Only the settings you change from their defaults are sent. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ### How much disk does the local drive cache give each worker? You choose it. **Storage size in GB** starts at 100 and takes 10 to 2000 GB. Choose a size larger than the model's files, or use a **Shared file-system** cache, which several workers share. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ### Why does my endpoint stay in Deploying for a long time? A large model can take 10 to 15 minutes or more to download and load, so **Deploying** for a while is normal. If it lasts much longer, read **Deployment Events** and **Worker Logs**. Then check the token, the model ID, the licence, and that the machine has free capacity. See [View logs and events](/tir/model-endpoints/how-to/view-logs-and-events/). ### Can I reuse the name of a deleted endpoint? Yes. You can use a deleted endpoint's name again. See [Delete an endpoint](/tir/model-endpoints/how-to/delete-an-endpoint/). ## Custom containers ### Can I run my own Docker image as an endpoint? Yes. Choose **Custom Container** and give a public or private image. See [Deploy a custom container](/tir/model-endpoints/how-to/deploy-a-custom-container/). ### Which port must my container listen on? On an **External** endpoint, HTTP is fixed to port 8080 and gRPC to port 9000. On an **Internal** endpoint you choose the HTTP ports, except 80 and 8082. See [Deploy a custom container](/tir/model-endpoints/how-to/deploy-a-custom-container/). ### Why does my custom container show Commands and Arguments boxes instead of Entrypoint Command? The endpoint was created before April 10, 2026. It keeps the older **Commands** and **Arguments** boxes, and you type each as a list in square brackets, as **View Command Examples** shows. An endpoint created since then has **Serving Framework**, **Entrypoint Command**, and **Arguments** rows. See [Deploy a custom container](/tir/model-endpoints/how-to/deploy-a-custom-container/). ### Can I change the container of a Multi Node endpoint on its own? No. On a Multi Node plan, change the container with **Update Inference**, which opens the whole form. See [Update an endpoint](/tir/model-endpoints/how-to/update-an-endpoint/). ## Calling an endpoint ### Which address do I call? The **Root Endpoint** on the endpoint's **Overview** and on **Sample API Request**. Add the route you want to it, such as `/v1/chat/completions`. See [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/). ### Which token do I send? An auth token from **Personal Access Tokens**, in an `Authorization: Bearer ` header. **Generate token** on the endpoint creates one if you need it. To call the endpoint, a valid token from the project is all you need, whatever your portal permissions. See [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/). ### Can I use the OpenAI client? Yes, for vLLM and SGLang endpoints, and for Custom Container and Finetuned ones. Set the base URL to the root endpoint plus `/v1/` and the API key to your token. See [Use the OpenAI client and stream responses](/tir/model-endpoints/how-to/use-the-openai-client-and-stream-responses/). ### Can I stream the answer? Yes. Add `stream=True` to the OpenAI client call. See [Use the OpenAI client and stream responses](/tir/model-endpoints/how-to/use-the-openai-client-and-stream-responses/). ### Why does my request fail with a wrong model name? The model name in a request is the Hugging Face ID, `/mnt/models` for a model from a repository, or `peft-model` for a LoRA adapter. The samples on **Sample API Request** use the right name. See [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/). ### Does the chat route work with any model? No. `/v1/chat/completions` works for instruct or chat models. A base model needs a chat template in the request, or you can use `/v1/completions` with a `prompt`. See [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/). ### Does an endpoint support gRPC? Yes. A Custom Container endpoint takes gRPC on port 9000 when you tick gRPC. TensorRT, PyTorch, and NVIDIA Triton endpoints show a gRPC tab on **Sample API Request**. See [Deploy a custom container](/tir/model-endpoints/how-to/deploy-a-custom-container/). ### Is WebSocket supported? Yes, for models whose server exposes a WebSocket route, for example a streaming speech model. Use the root endpoint with `wss` and send the token in the `Authorization` header. See [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/). ### Can I test in the portal without code? Yes, with the **Playground**, for endpoints that serve chat models. See [Test in the Playground](/tir/model-endpoints/how-to/test-in-the-playground/). ### How do I set up async inference? Async is set up after the endpoint is created, on its **Async Invocations** tab. Choose the routes and a dataset for the results there. See [Set up async invocation](/tir/model-endpoints/how-to/set-up-async-invocation/). ### Is async invocation available on every endpoint? No. Async invocation works on every plan except Multi Node. See [Set up async invocation](/tir/model-endpoints/how-to/set-up-async-invocation/). ### Which datasets can hold async results? Only an Object Storage dataset of the project, in the endpoint's region, that is ready to use. You can also create one from the setup panel. See [Set up async invocation](/tir/model-endpoints/how-to/set-up-async-invocation/). ## Workers and scaling ### Can an endpoint scale to zero? Yes, on an hourly plan other than Multi Node. Set **Active Worker** to 0 on the create form, or **Minimum Workers** to 0 on the **Serverless Configuration** tab. The endpoint scales down to 0 when requests stop arriving and shows **Idle**, and the next request starts a worker. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Which plans can scale to zero? Only hourly plans other than Multi Node. On a committed plan and on a Multi Node plan the minimum is 1. On a Multi Node plan the form asks only for **Active Worker**, and that number is also the maximum. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Can I scale to zero with the Custom metric? No. The **Custom** metric needs at least one active worker. The create form blocks **Deploy Endpoint** while the metric is **Custom** and **Active Worker** is 0, and the **Serverless Configuration** tab says that scaling to zero is not supported with a custom metric. Choose **Concurrent Request Count** or **Request Rate per second** to scale to zero. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Can I set Max Workers to 0? No. To pay nothing, set the minimum to 0 on an hourly plan, or stop the endpoint. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### How do I keep a fixed number of workers? Set the minimum equal to the maximum. The minimum and maximum are the only way to set the number of workers in the portal. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### How many workers can an endpoint have? Up to 64, or up to 512 on a private cluster. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Does every framework offer custom metrics? No. vLLM, SGLang, NVIDIA Dynamo, PyTorch, and NVIDIA Triton each offer a list under **Custom Metric Name**, and a Custom Container takes the name you type. TensorRT offers none. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### What values can Target Value take? A whole number from 1 to 100000, or from 1 to 100 for a CPU metric. **Idle Timeout (in seconds)** takes 1 to 1800, and **Initial Cooldown Period (in seconds)** takes 30 to 86400. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### How long does scaling down take? A few seconds to a few minutes, depending on the model size, the requests in progress, and how quickly the workers stop. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### What if the region has no free capacity when the endpoint scales up? The endpoint stays at the workers it has, and the endpoint's creator gets an email for each scale-up that failed to get a machine. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### What does Total GPUs mean? It is the number of GPUs on each worker multiplied by the number of workers currently set. It reads 0 for a **Stopped** or **Idle** endpoint. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ## Billing ### How is an endpoint billed? Two ways. With **Hourly**, you pay for the time its workers run. With **Committed**, you commit to a fixed term chosen from a savings plan. The price follows worker time, however many requests the endpoint answers. A Model Repository is billed on its own. See [How is a repository billed?](/tir/model-repository/faq/#how-is-a-repository-billed). See [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). For rates, see [E2E's pricing page](https://www.e2enetworks.com/pricing). ### Hourly or committed, which should I choose? Use an hourly plan for development and variable traffic, and a committed plan for steady, long-running use. A committed endpoint costs the same however much you use it. See [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). ### What do I pay for on an hourly plan? You pay for each worker for the time it is deployed, retrying, ready, or running. Time a worker spends waiting for free capacity, unavailable, or in error costs nothing. The bill follows the number of workers over time, so when autoscaling adds or removes a worker its hours start or stop at that moment. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Is there a minimum billing duration? No, not on an hourly plan: you pay for the time workers run. A worker that runs 45 minutes is charged for exactly those 45 minutes. A committed term is the minimum for its reservation. See [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). ### Do I pay for Max Workers? No, not on an hourly plan. You pay for the workers that run, and **Max Workers** only caps how far the endpoint can grow. On a committed plan, workers above the committed count are billed at the hourly rate of the same machine. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Am I charged while the model downloads? Yes, on an hourly plan. The time a worker spends starting up and loading the model is part of its running time. Stop the endpoint to end the charge. See [Stop, start, and restart an endpoint](/tir/model-endpoints/how-to/stop-start-and-restart-an-endpoint/). ### Am I charged while the endpoint is stopped or idle? No, not for compute. Compute billing stops while the endpoint is **Stopped**, and while it is **Idle** at zero workers. The **Idle Timeout** is different: idle workers stay running for it after the last request, and you are charged for that time. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ### Which account is billed? The project owner's account, whoever created the endpoint. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). ### Why can I not stop my committed endpoint? A committed endpoint runs for its whole term, so only an hourly endpoint can be stopped. Use an hourly plan if you need to stop an endpoint when it is idle. See [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). ### Can I change what happens when my committed term ends? Yes. Select **Update Inference**, and choose again in the committed-plan confirmation: - **Auto renewal of this committed plan** - **Auto start hourly billing for this machine** - **Auto deletion of this committed machine** See [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). ### Why can I not scale below my committed workers? A committed plan's worker count is the fewest workers the endpoint can run. A lower minimum is refused with "Cannot downscale below the committed plan's worker count of ``". Keep the minimum at or above that count. See [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). ### Do I get a refund if I delete a committed endpoint before its term ends? No. The committed charge is paid up front for the whole term, so it is not refunded when you delete early. Deleting a committed endpoint ends it at once. **Delete Service** reminds you that the charges have already been deducted. See [Delete an endpoint](/tir/model-endpoints/how-to/delete-an-endpoint/). ### Is an endpoint on a private cluster billed by the endpoint? No. You pay for the cluster's nodes instead. See [TIR Private Cluster](/tir/tir-private-cluster/). ### Is a Shared File System cache billed separately? Yes. A Shared File System is its own product with its own billing. See [TIR Shared File System](/tir/tir-sfs/). ### Does async invocation cost extra? Yes. A small CPU worker handles the queue and is billed only while the queue has requests. The endpoint's own workers are billed as usual. See [Set up async invocation](/tir/model-endpoints/how-to/set-up-async-invocation/). ### Where do I see what I used? In **Account Usage**, with **Inference Usage** chosen under **Resources**. See [Analyse usage](/tir/tir-billing/how-to/analyse-usage/). ## Network and alerts ### Can an endpoint reach other systems? Yes: APIs, databases, object storage, and monitoring, as far as its network and security-group rules allow. A Reserved VPC IP gives private access. See [Use a private network and a security group](/tir/model-endpoints/how-to/use-a-private-network-and-a-security-group/). ### Does a security group open ports on my endpoint? No. Adding a security group leaves the endpoint's ports as they are. The group's inbound rules only define which IPs can reach the ports the endpoint already has open. See [Use a private network and a security group](/tir/model-endpoints/how-to/use-a-private-network-and-a-security-group/). ### Which port does the private address use? Port 8080, always. The **Private Endpoint** sample calls `http://:8080`, whatever port the container listens on, so the security group's inbound rule is for TCP port 8080. See [Use a private network and a security group](/tir/model-endpoints/how-to/use-a-private-network-and-a-security-group/). ### Can I attach more than one VPC IP to an endpoint? No. An endpoint takes one VPC IP at a time. The VPC IP must be in the endpoint's region, and the VPC must use E2E-provided ranges. See [Use a private network and a security group](/tir/model-endpoints/how-to/use-a-private-network-and-a-security-group/). ### Which alerts can I set on an endpoint? You can set any alert from Alert Management except an ephemeral-storage alert, which watches temporary disk space that is lost on restart. GPU temperature and GPU power alerts work only for GPU endpoints. See [Monitor an endpoint](/tir/model-endpoints/how-to/monitor-an-endpoint/). ## Stopping, updating, and deleting ### Can I stop an endpoint? Yes, if it is on an hourly plan. A stopped endpoint keeps its settings and address, and **Start Service** brings it back. A committed endpoint runs until its term ends. See [Stop, start, and restart an endpoint](/tir/model-endpoints/how-to/stop-start-and-restart-an-endpoint/). ### Can I copy an endpoint? Yes. **Clone Inference** starts a new endpoint from the same settings. See [Clone an endpoint](/tir/model-endpoints/how-to/clone-an-endpoint/). ### Does deleting an endpoint delete my model files? No. The Model Repository entry and its files stay as they are when you delete an endpoint. See [Delete an endpoint](/tir/model-endpoints/how-to/delete-an-endpoint/).