Deploy Code Llama 7B with vLLM

Deploy Code Llama 7B on vLLM and ask it to write and explain code. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• A Hugging Face access token with Read scope, saved as a Hugging Face integration. See Create a Hugging Face integration.

Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see Deploy a model from Hugging Face.

Field	Value
Framework on Choose Framework	vLLM
Endpoint Name	codellama-7b, or a name of your own
Download Type	Download from Huggingface
HuggingFace Model	codellama/CodeLlama-7b-hf
HF Token Integration	Your Hugging Face integration
Model Cache Storage	Local drive, 100 GB
Select Computing Resource	GPU tab, a card with more than 14 GB of GPU memory, and a size under Configuration
Active Worker and Max Workers	1 and 1

Select Deploy Endpoint. The state reads Deploying while the worker downloads and loads the model, which can take 10 to 15 minutes, and Running once the endpoint can answer.

Ask for code

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

This model is a base model, so it continues the text you give it. Send it a completion request, with the Hugging Face ID as the model name:

curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "codellama/CodeLlama-7b-hf",
    "prompt": "Write a Python function for Fibonacci.",
    "max_tokens": 100,
    "temperature": 0.7
  }'

The same call from Python with the requests package:

import os
import requests

url = os.environ["ROOT_ENDPOINT"] + "/v1/completions"
headers = {
    "Authorization": "Bearer " + os.environ["AUTH_TOKEN"],
    "Content-Type": "application/json",
}
payload = {
    "model": "codellama/CodeLlama-7b-hf",
    "prompt": "Explain quicksort in Python.",
    "max_tokens": 200,
    "temperature": 0.5,
}

response = requests.post(url, json=payload, headers=headers)
print(response.json()["choices"][0]["text"])

The reply is a JSON document in the OpenAI completions format, and the generated code is in choices[0].text. To give instructions in chat form instead, deploy codellama/CodeLlama-7b-Instruct-hf and send messages to /v1/chat/completions, which answers only instruct and chat models.

Tune the output

Add these fields to the request body:

max_tokens          The most tokens the model writes. Keep it modest, because a high value makes a long wait.
temperature         How varied the code is. Start between 0.3 and 0.7.
top_p, top_k        Limit the choice to the most likely tokens.
repetition_penalty  Reduces repeated lines.

Clear instructions in the prompt help most: name the language and describe the function’s inputs and outputs. The first answer after a start can be slower than the ones that follow.

Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set Download Type to Link with Model Repository and choose your repository, and set Model Type to Full Model (for standard inference). Requests then name the model /mnt/models. See Upload model files and Deploy a model from Model Repository.

For a fuller application on the instruct model, Generate SQL from plain English with Code Llama turns questions into database queries.