Deploy MPT-7B-Chat with vLLM

Deploy MosaicML MPT-7B-Chat on vLLM and send it text prompts, one at a time or several in one request. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• A Hugging Face access token with Read scope, saved as a Hugging Face integration. See Create a Hugging Face integration.

Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see Deploy a model from Hugging Face.

Field	Value
Framework on Choose Framework	vLLM
Endpoint Name	mpt-7b-chat, or a name of your own
Download Type	Download from Huggingface
HuggingFace Model	mosaicml/mpt-7b-chat
HF Token Integration	Your Hugging Face integration
Model Cache Storage	Local drive, 100 GB
Select Computing Resource	GPU tab, a card with enough GPU memory for a 7 billion parameter model, such as the A100, and a size under Configuration
Active Worker and Max Workers	1 and 1

Select Deploy Endpoint. The state reads Deploying while the worker downloads and loads the model, which can take 10 to 15 minutes, and Running when the endpoint can answer.

Send prompts

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

Send a completion request. The model continues the text you give it, and the model name is the Hugging Face ID you entered in the form:

curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mosaicml/mpt-7b-chat",
    "prompt": "Here is a recipe for vegan banana bread:\n",
    "max_tokens": 100
  }'

To send several prompts in one request, make prompt a list. The reply holds one entry in choices for each prompt:

curl -X POST "$ROOT_ENDPOINT/v1/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mosaicml/mpt-7b-chat",
    "prompt": ["here is the tea recipe", "here is the pancakes recipe"],
    "max_tokens": 100
  }'

The reply is a JSON document in the OpenAI completions format, with the text of the first prompt in choices[0].text. If the call fails, open Events & Logs on the details page and read Worker Logs.

Tune the output

Add these fields to the request body:

max_tokens          The most tokens the model writes.
min_tokens          The fewest tokens the model writes.
temperature         How varied the text is.
top_p, top_k        Limit the choice to the most likely tokens.
repetition_penalty  Discourages repeated words.
n                   The number of answers for each prompt.

A lower max_tokens gives a faster first test.

Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set Download Type to Link with Model Repository and choose your repository, and set Model Type to Full Model (for standard inference). Requests then name the model /mnt/models. See Upload model files and Deploy a model from Model Repository.

Sending prompts in a list is the quickest way to process a batch of short texts with one call. To keep long batches from holding a connection open, set up async invocation on the endpoint.