Call a vLLM endpoint with the OpenAI client

View as Markdown

Send requests to a vLLM endpoint with the OpenAI Python client. A vLLM endpoint answers the OpenAI Completions and Chat routes. An application written for the OpenAI client works with your endpoint when you change the address, the token, and the model name.

Before you begin

A vLLM endpoint in the Running state. To make one, see Deploy Llama 3 8B Instruct with vLLM. The examples use that model, meta-llama/Meta-Llama-3-8B-Instruct.
Python 3 on your computer or in an instance, and the OpenAI client:
```
pip install openai
```

Get the address and the token

On the endpoint’s details page, select API Request.
Copy the Root Endpoint.
In Auth token, choose a token, or select Generate Token to make one.
Set both values as variables in your terminal. <root-endpoint> is the address you copied, and <auth-token> is the token. Keep the real token out of any script you share.
```
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

The OpenAI tab of Sample API Request has the same setup with your values filled in. The client’s base_url is the root endpoint followed by /v1/.

Send a completion request

Use the Completions route to continue a piece of text:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["ROOT_ENDPOINT"] + "/v1/",
    api_key=os.environ["AUTH_TOKEN"],
)

completion = client.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    prompt="San Francisco is a",
    max_tokens=50,
)

print("Completion result:", completion.choices[0].text)

Send a chat request

Use the Chat route for questions and conversations. It works for instruct and chat models. The first example waits for the whole answer:

completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "What is artificial intelligence?"}],
)

print(completion.choices[0].message.content)

To print the answer while the model writes it, set stream=True and read the chunks:

stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "What are large language models?"}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Run both snippets in the file from the completion example, after its client = ... lines. For more on streaming and the client, see Use the OpenAI client and stream responses.

Choose request parameters

Add these to either call to shape the answer:

max_tokens          The most tokens the model writes.
temperature         How varied the answer is. A lower value gives steadier answers.
top_p, top_k        Limit the choice to the most likely tokens.
stop                A string at which the answer ends.
seed                Makes a run repeatable.
presence_penalty    Discourages words that already appeared.
frequency_penalty   Discourages words that appeared often.
repetition_penalty  Discourages repeated words.
n                   The number of answers for one prompt.
stream              Returns the answer in pieces as it is written.

These work on the Completions route:

suffix              Text added after the generated text.
logprobs            The number of log probabilities returned for each token.
echo                Returns the prompt together with the completion.
best_of             Makes several candidates and returns the best n.
logit_bias          Pushes the model toward or away from given tokens.
user                An optional name for the user who sends the request.

These are vLLM’s own additions:

ignore_eos                  Keeps writing after the end-of-sentence token.
use_beam_search             Turns on beam search.
stop_token_ids              Token IDs at which the answer ends.
skip_special_tokens         Leaves special tokens out of the text. On by default.
length_penalty              Penalizes long sequences.
min_p                       The lowest relative probability a token needs.
include_stop_str_in_output  Keeps the stop string in the text.

The list depends on the vLLM version your endpoint runs.

Check that the endpoint answers

The completion call prints the generated text, and the chat calls print the model’s answer. If the call fails, check that the endpoint is Running and that ROOT_ENDPOINT has no /v1 at the end. Also check that the model name matches the endpoint’s Hugging Face ID exactly. If the token is rejected, choose or generate another in Sample API Request.

The same code works from any application that uses the OpenAI client. To build an agent on your endpoint, see Build an agent with the OpenAI Agents SDK.