Deploy Llama 2 with vLLM

Deploy Meta Llama 2 7B Chat on vLLM, then ask it questions and have it finish sentences. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• Access to the model. Llama 2 is a gated model: on its Hugging Face page, meta-llama/Llama-2-7b-chat-hf, request access and wait until the owner approves it.
• A Hugging Face access token with Read scope from the account that was granted access, saved as a Hugging Face integration. See Create a Hugging Face integration.

Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see Deploy a model from Hugging Face.

Field	Value
Framework on Choose Framework	vLLM
Endpoint Name	llama2-7b-chat, or a name of your own
Download Type	Download from Huggingface
HuggingFace Model	meta-llama/Llama-2-7b-chat-hf
HF Token Integration	Your Hugging Face integration
Model Cache Storage	Local drive, 100 GB
Select Computing Resource	GPU tab, a card with plenty of GPU memory such as the A100 80GB, and a size under Configuration
Active Worker and Max Workers	1 and 1

Select Deploy Endpoint. The endpoint is Deploying while the worker downloads and loads the model, which can take 10 to 15 minutes. When the state reads Running, the endpoint is ready.

Ask a question

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

Send a chat request. The model name is the Hugging Face ID you entered in the form:

curl -X POST "$ROOT_ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-chat-hf",
    "messages": [
      {"role": "user", "content": "Explain the theory of relativity in simple words."}
    ],
    "max_tokens": 250,
    "temperature": 0.7
  }'

The reply is a JSON document in the OpenAI chat format, with the answer in choices[0].message.content.

Complete a sentence

The /v1/completions route takes a prompt instead of messages, and the model continues the text you give it rather than answering a question. Start a sentence and send it:

curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-chat-hf",
    "prompt": "It is said that life is beautiful when",
    "max_tokens": 200,
    "top_k": 10
  }'

The continued text is in choices[0].text. This route suits the base model, meta-llama/Llama-2-7b-hf, which only completes text. Deploy it the same way, and put its ID in both the form and the request.

If a call fails

Open Events & Logs on the details page and check that the state is Running. If the endpoint gets stuck before Running, choose a card with more GPU memory, such as an A100 or an H100. A 403 or an access error in the logs means the Hugging Face account behind the token still needs the model owner’s approval. If answers are slow, lower max_tokens or choose a faster card.

Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set Download Type to Link with Model Repository and choose your repository, and set Model Type to Full Model (for standard inference). Requests then name the model /mnt/models. See Upload model files and Deploy a model from Model Repository.

To send the same requests from Python, see Call a vLLM endpoint with the OpenAI client. For a conversation over several turns, pass the earlier questions and answers in messages.