---
title: "Deploy MPT-7B-Chat with vLLM"
description: "Serve MosaicML MPT-7B-Chat from Hugging Face on vLLM as a Model Endpoint, send it one or several prompts at once, and optionally serve your own weights."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/deploy-mpt-7b-chat-with-vllm/"
topic: ["tutorials"]
---

# Deploy MPT-7B-Chat with vLLM

Deploy MosaicML MPT-7B-Chat on vLLM and send it text prompts, one at a time or several in one request. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
- A Hugging Face access token with **Read** scope, saved as a Hugging Face integration. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/).

## Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/).

| Field                                 | Value                                                                                                                            |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Framework on **Choose Framework**     | **vLLM**                                                                                                                         |
| **Endpoint Name**                     | `mpt-7b-chat`, or a name of your own                                                                                             |
| **Download Type**                     | **Download from Huggingface**                                                                                                    |
| **HuggingFace Model**                 | `mosaicml/mpt-7b-chat`                                                                                                           |
| **HF Token Integration**              | Your Hugging Face integration                                                                                                    |
| **Model Cache Storage**               | **Local drive**, 100 GB                                                                                                          |
| **Select Computing Resource**         | **GPU** tab, a card with enough GPU memory for a 7 billion parameter model, such as the A100, and a size under **Configuration** |
| **Active Worker** and **Max Workers** | 1 and 1                                                                                                                          |

Select **Deploy Endpoint**. The state reads **Deploying** while the worker downloads and loads the model, which can take 10 to 15 minutes, and **Running** when the endpoint can answer.

## Send prompts

On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `<root-endpoint>` is the address you copied and `<auth-token>` is the token:

```bash
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

Send a completion request. The model continues the text you give it, and the model name is the Hugging Face ID you entered in the form:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mosaicml/mpt-7b-chat",
    "prompt": "Here is a recipe for vegan banana bread:\n",
    "max_tokens": 100
  }'
```

To send several prompts in one request, make `prompt` a list. The reply holds one entry in `choices` for each prompt:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mosaicml/mpt-7b-chat",
    "prompt": ["here is the tea recipe", "here is the pancakes recipe"],
    "max_tokens": 100
  }'
```

The reply is a JSON document in the OpenAI completions format, with the text of the first prompt in `choices[0].text`. If the call fails, open **Events & Logs** on the details page and read **Worker Logs**.

## Tune the output

Add these fields to the request body:

```text
max_tokens          The most tokens the model writes.
min_tokens          The fewest tokens the model writes.
temperature         How varied the text is.
top_p, top_k        Limit the choice to the most likely tokens.
repetition_penalty  Discourages repeated words.
n                   The number of answers for each prompt.
```

A lower max_tokens gives a faster first test.

## Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set **Download Type** to **Link with Model Repository** and choose your repository, and set **Model Type** to **Full Model (for standard inference)**. Requests then name the model `/mnt/models`. See [Upload model files](/tir/model-repository/how-to/upload-model-files/) and [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/).

Sending prompts in a list is the quickest way to process a batch of short texts with one call. To keep long batches from holding a connection open, [set up async invocation](/tir/model-endpoints/how-to/set-up-async-invocation/) on the endpoint.