---
title: "Deploy Llama 3 8B Instruct with vLLM"
description: "Serve Meta Llama 3 8B Instruct from Hugging Face on vLLM as a Model Endpoint, call it with curl, and optionally serve your own weights."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/deploy-llama-3-8b-instruct-with-vllm/"
topic: ["tutorials"]
---

# Deploy Llama 3 8B Instruct with vLLM

Deploy Meta Llama 3 8B Instruct on vLLM and send it a chat request. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
- Access to the model. Llama 3 is a gated model: on its Hugging Face page, [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), request access and wait until the owner approves it.
- A Hugging Face access token with **Read** scope, from the account that was granted access, saved as a Hugging Face integration. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/).

## Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/).

| Field                                 | Value                                                                        |
| ------------------------------------- | ---------------------------------------------------------------------------- |
| Framework on **Choose Framework**     | **vLLM**                                                                     |
| **Endpoint Name**                     | `llama3-infer`, or a name of your own                                        |
| **Download Type**                     | **Download from Huggingface**                                                |
| **HuggingFace Model**                 | `meta-llama/Meta-Llama-3-8B-Instruct`                                        |
| **HF Token Integration**              | Your Hugging Face integration                                                |
| **Model Cache Storage**               | **Local drive**, 100 GB                                                      |
| **Select Computing Resource**         | **GPU** tab, a card such as A100 or H100, and a size under **Configuration** |
| **Active Worker** and **Max Workers** | 1 and 1                                                                      |

The model's weights take about 16 GB of GPU memory, so a card with more memory than that leaves room for requests. When you choose the integration, TIR checks that your token can read the model. If it cannot, check that the owner has approved your access.

Select **Deploy Endpoint**. The endpoint is **Deploying** while the worker downloads and loads the model, which takes 10 to 15 minutes for a model of this size. The state changes to **Running** when the endpoint can answer.

## Call the endpoint

On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `<root-endpoint>` is the address you copied and `<auth-token>` is the token:

```bash
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

Send a chat request. The model name is the Hugging Face ID you entered in the form:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Write a short Python function to reverse a string."}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'
```

The reply is a JSON document in the OpenAI chat format, and the model's answer is in `choices[0].message.content`. For other ways to call it, see [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/).

## Tune the output

Add these fields to the request body:

```text
max_tokens          The most tokens the model writes. A lower value gives a faster first test.
temperature         How varied the text is.
top_p, top_k        Limit the choice to the most likely tokens.
repetition_penalty  Discourages repeated words.
stop                A string at which the text ends.
```

For example, send the prompt "Summarize quantum computing in 50 words." with max_tokens 250 and temperature 0.6.

## If the answer looks wrong

- An empty or cut-short answer: raise max_tokens, or set temperature between 0.7 and 1.0.
- Slow answers: lower max_tokens, or choose a faster card.
- The endpoint gets stuck before **Running**: choose a card with more GPU memory, such as an A100 or an H100, and read **Worker Logs** on **Events & Logs**.
- A `403` or an access error in the form or the logs: the Hugging Face account behind the token needs the model owner's approval.

## Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set **Download Type** to **Link with Model Repository** and choose your repository, and set **Model Type** to **Full Model (for standard inference)**. Requests then name the model `/mnt/models`. See [Upload model files](/tir/model-repository/how-to/upload-model-files/) and [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/).

With Llama 3 answering, the next step is usually your own code: [Call a vLLM endpoint with the OpenAI client](/tir/model-endpoints/how-to/call-a-vllm-endpoint-with-the-openai-client/) uses this same model in its examples.