---
title: "Deploy Llama 2 with vLLM"
description: "Serve Meta Llama 2 7B Chat from Hugging Face on vLLM as a Model Endpoint, call it for chat and completion, and optionally serve your own weights."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/deploy-llama-2-with-vllm/"
topic: ["tutorials"]
---

# Deploy Llama 2 with vLLM

Deploy Meta Llama 2 7B Chat on vLLM, then ask it questions and have it finish sentences. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
- Access to the model. Llama 2 is a gated model: on its Hugging Face page, [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), request access and wait until the owner approves it.
- A Hugging Face access token with **Read** scope from the account that was granted access, saved as a Hugging Face integration. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/).

## Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/).

| Field                                 | Value                                                                                                   |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Framework on **Choose Framework**     | **vLLM**                                                                                                |
| **Endpoint Name**                     | `llama2-7b-chat`, or a name of your own                                                                 |
| **Download Type**                     | **Download from Huggingface**                                                                           |
| **HuggingFace Model**                 | `meta-llama/Llama-2-7b-chat-hf`                                                                         |
| **HF Token Integration**              | Your Hugging Face integration                                                                           |
| **Model Cache Storage**               | **Local drive**, 100 GB                                                                                 |
| **Select Computing Resource**         | **GPU** tab, a card with plenty of GPU memory such as the A100 80GB, and a size under **Configuration** |
| **Active Worker** and **Max Workers** | 1 and 1                                                                                                 |

Select **Deploy Endpoint**. The endpoint is **Deploying** while the worker downloads and loads the model, which can take 10 to 15 minutes. When the state reads **Running**, the endpoint is ready.

## Ask a question

On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `<root-endpoint>` is the address you copied and `<auth-token>` is the token:

```bash
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

Send a chat request. The model name is the Hugging Face ID you entered in the form:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-chat-hf",
    "messages": [
      {"role": "user", "content": "Explain the theory of relativity in simple words."}
    ],
    "max_tokens": 250,
    "temperature": 0.7
  }'
```

The reply is a JSON document in the OpenAI chat format, with the answer in `choices[0].message.content`.

## Complete a sentence

The `/v1/completions` route takes a `prompt` instead of messages, and the model continues the text you give it rather than answering a question. Start a sentence and send it:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-chat-hf",
    "prompt": "It is said that life is beautiful when",
    "max_tokens": 200,
    "top_k": 10
  }'
```

The continued text is in `choices[0].text`. This route suits the base model, `meta-llama/Llama-2-7b-hf`, which only completes text. Deploy it the same way, and put its ID in both the form and the request.

## If a call fails

Open **Events & Logs** on the details page and check that the state is **Running**. If the endpoint gets stuck before **Running**, choose a card with more GPU memory, such as an A100 or an H100. A `403` or an access error in the logs means the Hugging Face account behind the token still needs the model owner's approval. If answers are slow, lower max_tokens or choose a faster card.

## Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set **Download Type** to **Link with Model Repository** and choose your repository, and set **Model Type** to **Full Model (for standard inference)**. Requests then name the model `/mnt/models`. See [Upload model files](/tir/model-repository/how-to/upload-model-files/) and [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/).

To send the same requests from Python, see [Call a vLLM endpoint with the OpenAI client](/tir/model-endpoints/how-to/call-a-vllm-endpoint-with-the-openai-client/). For a conversation over several turns, pass the earlier questions and answers in `messages`.