Deploy Llama 3 8B Instruct with vLLM

Deploy Meta Llama 3 8B Instruct on vLLM and send it a chat request. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• Access to the model. Llama 3 is a gated model: on its Hugging Face page, Meta-Llama-3-8B-Instruct, request access and wait until the owner approves it.
• A Hugging Face access token with Read scope, from the account that was granted access, saved as a Hugging Face integration. See Create a Hugging Face integration.

Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see Deploy a model from Hugging Face.

Field	Value
Framework on Choose Framework	vLLM
Endpoint Name	llama3-infer, or a name of your own
Download Type	Download from Huggingface
HuggingFace Model	meta-llama/Meta-Llama-3-8B-Instruct
HF Token Integration	Your Hugging Face integration
Model Cache Storage	Local drive, 100 GB
Select Computing Resource	GPU tab, a card such as A100 or H100, and a size under Configuration
Active Worker and Max Workers	1 and 1

The model’s weights take about 16 GB of GPU memory, so a card with more memory than that leaves room for requests. When you choose the integration, TIR checks that your token can read the model. If it cannot, check that the owner has approved your access.

Select Deploy Endpoint. The endpoint is Deploying while the worker downloads and loads the model, which takes 10 to 15 minutes for a model of this size. The state changes to Running when the endpoint can answer.

Call the endpoint

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

Send a chat request. The model name is the Hugging Face ID you entered in the form:

curl -X POST "$ROOT_ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Write a short Python function to reverse a string."}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'

The reply is a JSON document in the OpenAI chat format, and the model’s answer is in choices[0].message.content. For other ways to call it, see Call an endpoint with a token.

Tune the output

Add these fields to the request body:

max_tokens          The most tokens the model writes. A lower value gives a faster first test.
temperature         How varied the text is.
top_p, top_k        Limit the choice to the most likely tokens.
repetition_penalty  Discourages repeated words.
stop                A string at which the text ends.

For example, send the prompt “Summarize quantum computing in 50 words.” with max_tokens 250 and temperature 0.6.

If the answer looks wrong

• An empty or cut-short answer: raise max_tokens, or set temperature between 0.7 and 1.0.
• Slow answers: lower max_tokens, or choose a faster card.
• The endpoint gets stuck before Running: choose a card with more GPU memory, such as an A100 or an H100, and read Worker Logs on Events & Logs.
• A 403 or an access error in the form or the logs: the Hugging Face account behind the token needs the model owner’s approval.

Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set Download Type to Link with Model Repository and choose your repository, and set Model Type to Full Model (for standard inference). Requests then name the model /mnt/models. See Upload model files and Deploy a model from Model Repository.

With Llama 3 answering, the next step is usually your own code: Call a vLLM endpoint with the OpenAI client uses this same model in its examples.