---
title: "Use the OpenAI client and stream responses"
description: "Call a vLLM or SGLang Model Endpoint with the OpenAI Python package, and stream the answer as it is generated."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/use-the-openai-client-and-stream-responses/"
topic: ["call"]
---

# Use the OpenAI client and stream responses

<!-- cspell:ignore Finetuned -->

Point the OpenAI Python package at your endpoint, so you keep the code you already have for OpenAI-style APIs. The **OpenAI** tab of **Sample API Request** shows the same code with your values filled in.

## Before you begin

- A vLLM or SGLang endpoint, or a Custom Container or Finetuned one, in the **Running** state. The **OpenAI** tab is available for these.
- The endpoint's **Root Endpoint** and an auth token. See [Call an endpoint with a token](/tir/model-endpoints/how-to/call-an-endpoint-with-a-token/).
- Python with the `openai` package installed.

## Send a chat request

Set the base URL to the root endpoint plus `/v1/`, and the API key to your auth token:

```python
from openai import OpenAI

client = OpenAI(
    base_url="<root-endpoint>/v1/",
    api_key="<auth-token>",
)

response = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Who won the world series in 2020?"}],
)
print(response.choices[0].message.content)
```

- `<root-endpoint>` is the address from the endpoint's **Overview**.
- `<auth-token>` is a token from **Personal Access Tokens**.
- `<model-name>` is the Hugging Face ID of the model, `/mnt/models` for a model from a repository, or `peft-model` for a LoRA adapter.

## Stream the answer

Add `stream=True` and read the chunks as they arrive:

```python
stream = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Who won the world series in 2020?"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")
```

## Check the result

The first call prints the reply, and the second prints it piece by piece. The chat route works for instruct or chat models. A base model needs a chat template in the request.

For completion requests and the full list of request parameters, see [Call a vLLM endpoint with the OpenAI client](/tir/model-endpoints/how-to/call-a-vllm-endpoint-with-the-openai-client/). For an async call, the OpenAI sample uses `<root-endpoint>/async/v1/` as the base URL. See [Set up async invocation](/tir/model-endpoints/how-to/set-up-async-invocation/). If the call fails, see [Troubleshooting Model Endpoints](/tir/model-endpoints/troubleshooting/model-endpoints/).