---
title: "Deploy Code Llama 7B with vLLM"
description: "Serve Code Llama 7B from Hugging Face on vLLM as a Model Endpoint, ask it to write code with curl or Python, and optionally serve your own fine-tuned weights."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/deploy-code-llama-7b-with-vllm/"
topic: ["tutorials"]
---

# Deploy Code Llama 7B with vLLM

Deploy Code Llama 7B on vLLM and ask it to write and explain code. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
- A Hugging Face access token with **Read** scope, saved as a Hugging Face integration. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/).

## Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/).

| Field                                 | Value                                                                                      |
| ------------------------------------- | ------------------------------------------------------------------------------------------ |
| Framework on **Choose Framework**     | **vLLM**                                                                                   |
| **Endpoint Name**                     | `codellama-7b`, or a name of your own                                                      |
| **Download Type**                     | **Download from Huggingface**                                                              |
| **HuggingFace Model**                 | codellama/CodeLlama-7b-hf                                                                  |
| **HF Token Integration**              | Your Hugging Face integration                                                              |
| **Model Cache Storage**               | **Local drive**, 100 GB                                                                    |
| **Select Computing Resource**         | **GPU** tab, a card with more than 14 GB of GPU memory, and a size under **Configuration** |
| **Active Worker** and **Max Workers** | 1 and 1                                                                                    |

Select **Deploy Endpoint**. The state reads **Deploying** while the worker downloads and loads the model, which can take 10 to 15 minutes, and **Running** once the endpoint can answer.

## Ask for code

On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `<root-endpoint>` is the address you copied and `<auth-token>` is the token:

```bash
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

This model is a base model, so it continues the text you give it. Send it a completion request, with the Hugging Face ID as the model name:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "codellama/CodeLlama-7b-hf",
    "prompt": "Write a Python function for Fibonacci.",
    "max_tokens": 100,
    "temperature": 0.7
  }'
```

The same call from Python with the `requests` package:

```python
import os
import requests

url = os.environ["ROOT_ENDPOINT"] + "/v1/completions"
headers = {
    "Authorization": "Bearer " + os.environ["AUTH_TOKEN"],
    "Content-Type": "application/json",
}
payload = {
    "model": "codellama/CodeLlama-7b-hf",
    "prompt": "Explain quicksort in Python.",
    "max_tokens": 200,
    "temperature": 0.5,
}

response = requests.post(url, json=payload, headers=headers)
print(response.json()["choices"][0]["text"])
```

The reply is a JSON document in the OpenAI completions format, and the generated code is in `choices[0].text`. To give instructions in chat form instead, deploy codellama/CodeLlama-7b-Instruct-hf and send `messages` to `/v1/chat/completions`, which answers only instruct and chat models.

## Tune the output

Add these fields to the request body:

```text
max_tokens          The most tokens the model writes. Keep it modest, because a high value makes a long wait.
temperature         How varied the code is. Start between 0.3 and 0.7.
top_p, top_k        Limit the choice to the most likely tokens.
repetition_penalty  Reduces repeated lines.
```

Clear instructions in the prompt help most: name the language and describe the function's inputs and outputs. The first answer after a start can be slower than the ones that follow.

## Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set **Download Type** to **Link with Model Repository** and choose your repository, and set **Model Type** to **Full Model (for standard inference)**. Requests then name the model `/mnt/models`. See [Upload model files](/tir/model-repository/how-to/upload-model-files/) and [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/).

For a fuller application on the instruct model, [Generate SQL from plain English with Code Llama](/tir/model-endpoints/how-to/generate-sql-from-plain-english-with-code-llama/) turns questions into database queries.