Get started with Model Endpoints

Deploy a chat model from Hugging Face on vLLM, wait for it to start, and send it a first request. You need a TIR project and permission to create model endpoints in it. A Member needs a policy that grants create permission for the Inference service.

You also need:

• A Hugging Face access token saved as a Hugging Face integration. See Create a Hugging Face integration.
• The ID of a chat or instruct model on Hugging Face, in the form <organization>/<model-name>. If the model is gated, accept its licence on Hugging Face with the account that owns the token.

Create the endpoint

1. In TIR, open your project. In the side menu, under Inference, select Model Endpoints, then Create Endpoint.
2. Find vLLM, keep its suggested Version, and select vLLM.
3. Keep the suggested Endpoint Name, which starts with vllm-, or type your own in lowercase letters, digits, and hyphens.
4. Leave Download Type on Download from Huggingface. In HuggingFace Model, type the model ID, and wait for TIR to accept it.
5. In HF Token Integration, choose your integration. TIR checks that the token can read the model.
6. Leave Model Cache Storage on Local drive with its 100 GB size, which holds a model of 7 to 8 billion parameters with room to spare.
7. In Select Computing Resource, open the GPU tab and choose a card with enough GPU memory for the model. In 16-bit precision, the weights take about 2 GB of GPU memory for every billion parameters, so an 8 billion parameter model needs about 16 GB plus room for requests. Pick a size under Configuration, and choose On-Demand under Instance Pricing to pay by the hour.
8. In Serverless Configuration, leave Active Worker at 1 and set Max Workers to 1, so the endpoint keeps one worker.
9. Select Deploy Endpoint.

TIR opens the endpoint’s details. The endpoint starts in Deploying while its worker downloads and loads the model, which can take 10 to 15 minutes or more for a large model. When the state reads Running, the endpoint is ready for requests.

Call the endpoint

1. On the Overview tab, under Auth token, choose a token, or select Generate token to create one.
2. Copy the endpoint address, the root endpoint.
3. Send a chat request. Add /v1/chat/completions to the root endpoint, put the token in the Authorization header, and use the model ID as the model name:

curl -X POST "<root-endpoint>/v1/chat/completions" \
  -H "Authorization: Bearer <auth-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<organization>/<model-name>",
    "messages": [{"role": "user", "content": "Who won the world series in 2020?"}]
  }'

Replace <root-endpoint> with the address you copied, <auth-token> with the token you chose, and <organization>/<model-name> with your model ID. API Request on the endpoint’s page gives the same request with these values filled in.

Check that it works

The endpoint shows Running with its workers counted under the state, for example 1 / 1. Your request returns a JSON reply, and the model’s answer is in choices[0].message.content. If the endpoint stays in Deploying or the request is refused, see Troubleshooting Model Endpoints.

Stop the charges when you are done

An hourly endpoint is charged for the time its worker runs, including the time it spends loading the model, even when it sits idle. When you finish testing, select Stop Service on the endpoint’s details page, or under Actions in the list. The endpoint keeps its settings and address, and Start Service brings it back. To remove it for good, see Delete an endpoint.

What to do next

• Use the OpenAI client and stream responses to call the endpoint from Python.
• Set workers and autoscaling to let the endpoint grow with traffic, or scale to zero so an idle endpoint costs nothing.
• Test in the Playground to chat with the endpoint in the portal.
• Follow a walkthrough for a particular model, such as Deploy Llama 3 8B Instruct with vLLM.