---
title: "Call a vLLM endpoint with the OpenAI client"
description: "Use the OpenAI Python client to send completion and chat requests, as a stream or as one reply, to a Model Endpoint that runs vLLM."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/call-a-vllm-endpoint-with-the-openai-client/"
topic: ["tutorials"]
---

# Call a vLLM endpoint with the OpenAI client

Send requests to a vLLM endpoint with the OpenAI Python client. A vLLM endpoint answers the OpenAI **Completions** and **Chat** routes. An application written for the OpenAI client works with your endpoint when you change the address, the token, and the model name.

## Before you begin

- A vLLM endpoint in the **Running** state. To make one, see [Deploy Llama 3 8B Instruct with vLLM](/tir/model-endpoints/how-to/deploy-llama-3-8b-instruct-with-vllm/). The examples use that model, `meta-llama/Meta-Llama-3-8B-Instruct`.
- Python 3 on your computer or in an instance, and the OpenAI client:

  ```bash
  pip install openai
  ```

## Get the address and the token

1. On the endpoint's details page, select **API Request**.
2. Copy the **Root Endpoint**.
3. In **Auth token**, choose a token, or select **Generate Token** to make one.
4. Set both values as variables in your terminal. `<root-endpoint>` is the address you copied, and `<auth-token>` is the token. Keep the real token out of any script you share.

   ```bash
   export ROOT_ENDPOINT="<root-endpoint>"
   export AUTH_TOKEN="<auth-token>"
   ```

The **OpenAI** tab of **Sample API Request** has the same setup with your values filled in. The client's base_url is the root endpoint followed by `/v1/`.

## Send a completion request

Use the Completions route to continue a piece of text:

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["ROOT_ENDPOINT"] + "/v1/",
    api_key=os.environ["AUTH_TOKEN"],
)

completion = client.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    prompt="San Francisco is a",
    max_tokens=50,
)

print("Completion result:", completion.choices[0].text)
```

## Send a chat request

Use the Chat route for questions and conversations. It works for instruct and chat models. The first example waits for the whole answer:

```python
completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "What is artificial intelligence?"}],
)

print(completion.choices[0].message.content)
```

To print the answer while the model writes it, set `stream=True` and read the chunks:

```python
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "What are large language models?"}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

Run both snippets in the file from the completion example, after its `client = ...` lines. For more on streaming and the client, see [Use the OpenAI client and stream responses](/tir/model-endpoints/how-to/use-the-openai-client-and-stream-responses/).

## Choose request parameters

Add these to either call to shape the answer:

```text
max_tokens          The most tokens the model writes.
temperature         How varied the answer is. A lower value gives steadier answers.
top_p, top_k        Limit the choice to the most likely tokens.
stop                A string at which the answer ends.
seed                Makes a run repeatable.
presence_penalty    Discourages words that already appeared.
frequency_penalty   Discourages words that appeared often.
repetition_penalty  Discourages repeated words.
n                   The number of answers for one prompt.
stream              Returns the answer in pieces as it is written.
```

These work on the Completions route:

```text
suffix              Text added after the generated text.
logprobs            The number of log probabilities returned for each token.
echo                Returns the prompt together with the completion.
best_of             Makes several candidates and returns the best n.
logit_bias          Pushes the model toward or away from given tokens.
user                An optional name for the user who sends the request.
```

These are vLLM's own additions:

```text
ignore_eos                  Keeps writing after the end-of-sentence token.
use_beam_search             Turns on beam search.
stop_token_ids              Token IDs at which the answer ends.
skip_special_tokens         Leaves special tokens out of the text. On by default.
length_penalty              Penalizes long sequences.
min_p                       The lowest relative probability a token needs.
include_stop_str_in_output  Keeps the stop string in the text.
```

The list depends on the vLLM version your endpoint runs.

## Check that the endpoint answers

The completion call prints the generated text, and the chat calls print the model's answer. If the call fails, check that the endpoint is **Running** and that ROOT_ENDPOINT has no `/v1` at the end. Also check that the model name matches the endpoint's Hugging Face ID exactly. If the token is rejected, choose or generate another in **Sample API Request**.

The same code works from any application that uses the OpenAI client. To build an agent on your endpoint, see [Build an agent with the OpenAI Agents SDK](/tir/model-endpoints/how-to/build-an-agent-with-the-openai-agents-sdk/).