Model Endpoints

A Model Endpoint serves your model on one or more workers behind one HTTPS address, the root endpoint. Your applications call that address with an auth token. vLLM and SGLang endpoints answer the same chat and completion requests as the OpenAI API. Code written for OpenAI works once you change the address, the key, and the model name.

You choose a framework such as vLLM and tell the endpoint where the model comes from: Hugging Face, a Model Repository that holds your own files, or your own container image. Then you pick a machine and set how many workers run. The endpoint can keep a fixed number of workers, grow and shrink with traffic, or scale to zero when nobody calls it. An endpoint belongs to one project and one region.

Where to find it

In TIR, open your project. In the side menu, under Inference, select Model Endpoints. Your endpoints in the current project and region are listed there, and Create Endpoint starts a new one.

Create an endpoint

• Create an endpoint from Choose Framework and one form.
• Choose a framework to match the model you bring.
• Deploy a model from Hugging Face with a model ID and a Hugging Face token.
• Deploy a model from Model Repository, including a LoRA adapter.
• Deploy a custom container from a public or private image.
• Set environment variables for your model or container.
• Choose a committed plan for steady, long-running use.

Call an endpoint

• Call an endpoint with a token from cURL, your own code, or a WebSocket client.
• Use the OpenAI client and stream responses.
• Set up async invocation to queue long-running requests.
• Test in the Playground to chat with the model in the portal.

Scale and operate

• Set workers and autoscaling, including scale to zero.
• View logs and events to see what a worker is doing while it starts.
• Monitor an endpoint with hardware and serving charts.
• Update an endpoint to change its model, framework, machine, or settings.
• Stop, start, and restart an endpoint.
• Clone an endpoint to start a new one from the same settings.
• Use a private network and a security group to call the endpoint from your VPC.
• Delete an endpoint when you have finished with it.

Walk through a model or framework

Each of these takes one model or framework from an empty project to a working request.

• On vLLM: Llama 3 8B Instruct, Llama 2, Code Llama 7B, Gemma, and MPT-7B-Chat.
• Applications on a vLLM endpoint: Generate SQL from plain English with Code Llama, Call a vLLM endpoint with the OpenAI client, and Build an agent with the OpenAI Agents SDK.
• Other frameworks: Serve a model with NVIDIA Triton, Serve a PyTorch model with TorchServe, and Deploy a custom API handler on a Custom Container.

Where to start

Start with Get started with Model Endpoints, which deploys a Hugging Face model on vLLM and calls it. To create and manage endpoints from code, the TIR API reference lists the operations under Model Endpoints. The Model Endpoints FAQ answers questions about regions, access, scaling, and billing, and Troubleshooting Model Endpoints covers the messages that come up most.