--- title: "Create an endpoint" description: "Create a TIR Model Endpoint by choosing a framework, a model source, a machine, and the number of workers, then deploy it and wait for Running." content_type: "how-to" product: "model-endpoints" platform: "TIR" url: "/tir/model-endpoints/how-to/create-an-endpoint/" topic: ["create"] --- # Create an endpoint Create an endpoint to serve a model behind an HTTPS address that your applications can call. You pick a framework first. The form then asks where the model comes from, which machine to use, and how many workers to run. ## Before you begin - Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service. TIR shows **Create Endpoint** only to people who have the create permission. - Credit for the machine you choose. On a prepaid account, an hourly endpoint needs credit for its launch, and a committed plan needs its full price. For rates, see [E2E's pricing page](https://www.e2enetworks.com/pricing). - Your model source ready: - For a Hugging Face model, a Hugging Face integration. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/). - For your own files, a repository that holds the model. See [Upload model files](/tir/model-repository/how-to/upload-model-files/). - For your own image, the image's address. ## Choose the framework 1. In the side menu, under **Inference**, select **Model Endpoints**, then **Create Endpoint**. 2. Search with **Search Framework** if you need to, choose a version under the framework's **Version**, and select the framework. To pick between frameworks, see [Choose a framework](/tir/model-endpoints/how-to/choose-a-framework/). ## Name the endpoint **Endpoint Name** starts as a random name that begins with the framework's name. Keep it or type your own: up to 50 lowercase letters and digits, with single hyphens between them, starting and ending with a letter or digit. ## Say where the model comes from **Download Type** is the first block for a framework that loads a model, and its choices depend on the framework. Follow the page for the one you use: - **Download from Huggingface**: [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/). - **Link with Model Repository**: [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). - **Container Configuration** for **Custom Container**: [Deploy a custom container](/tir/model-endpoints/how-to/deploy-a-custom-container/). ## Choose where the downloaded model is kept **Model Cache Storage** is required, whichever **Download Type** you chose. It has two options: - **Local drive** keeps a separate copy of the model on each worker, and the copy is lost when the endpoint ends. Set **Storage size in GB** larger than the model's files. It starts at 100 and takes 10 to 2000. - **Shared file-system** lets several workers share one download, so extra workers start faster. Pick an existing **Shared File-system** and a **Cache path**, or create one from the link. A Shared File System is its own product with its own billing; see [TIR Shared File System](/tir/tir-sfs/). The cache path starts as `/mnt/models` for a local drive and `/shared` for a shared file system, and you can keep it. For a Hugging Face download the field is named **HF_CACHE path**. ## Choose the machine In **Select Computing Resource**, open a tab and choose a machine: - **GPU** for a single machine with GPUs. This is the usual choice for a language model. - **Multi Node** for a model too large for one machine. Its name gives the node count after the GPU, such as "(2x)". - **CPU** for a small model that runs well on a CPU. - **Private Cluster** to run on your own cluster; choose it under **Select Private Cluster**. You can also add a scheduling rule that keeps the endpoint on a node group, or off it. See [TIR Private Cluster](/tir/tir-private-cluster/). Each machine lists the memory of one of its GPUs, such as 80 GB. Choose a machine and size whose GPU memory holds the model with room for requests. In 16-bit precision, a model's weights take about 2 GB of GPU memory for every billion parameters. An 8 billion parameter model needs about 16 GB, and a 70 billion parameter model about 140 GB, spread over several GPUs. The size under **Configuration** sets the GPUs, vCPUs, and RAM that one worker gets. A machine marked **Inventory not available**, or a size marked **Plan not available**, is out of stock in the region right now. Choose another one, or try again later. Under **Instance Pricing**, choose how you pay. **On-Demand** is hourly pay-as-you-go and suits development and changing traffic. A savings plan commits you to a term and suits steady, long-running use. TIR asks you to confirm the plan before it creates the endpoint; see [Choose a committed plan](/tir/model-endpoints/how-to/choose-a-committed-plan/). For the longer terms, select **Show more pricing**. ## Set the workers In **Serverless Configuration**, set **Active Worker**, the workers that are always ready, and **Max Workers**, the most the endpoint can grow to. For a fixed number of workers, set both to the same number. Scale to zero, the scaling policy, and the other worker settings are in [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). ## (Optional) Open Advanced Settings **Advanced Settings** holds **Environment Variables** and, for a framework version that has them, **LLM Settings**, which tune how the model loads and runs: - **LLM Settings** covers data type, quantization, context length, and memory. - **System Settings** covers scheduler, batching, concurrency, and logging. On vLLM it also holds the tool-calling settings an agent needs. - **Tokenizer Settings** covers the tokenizer path, mode, and chat template. Which groups there are depends on the framework version. TIR sends only the settings you change from their defaults. For environment variables, see [Set environment variables](/tir/model-endpoints/how-to/set-environment-variables/). ## Deploy the endpoint Select **Deploy Endpoint** once every required field is valid. TIR opens the endpoint's details and emails you, with the project's Owner and Admins in copy. ## Check that the endpoint is running The endpoint starts in **Deploying** while its workers download and load the model. The details page refreshes by itself. When the state reads **Running**, every worker is ready for requests. A large model can take 10 to 15 minutes or more. If creating the endpoint fails, the portal shows "Unable to create endpoint" or the reason; see [Troubleshooting Model Endpoints](/tir/model-endpoints/troubleshooting/model-endpoints/). To send a first request, see [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/).