Use the OpenAI client and stream responses

Point the OpenAI Python package at your endpoint, so you keep the code you already have for OpenAI-style APIs. The OpenAI tab of Sample API Request shows the same code with your values filled in.

Before you begin

• A vLLM or SGLang endpoint, or a Custom Container or Finetuned one, in the Running state. The OpenAI tab is available for these.
• The endpoint’s Root Endpoint and an auth token. See Call an endpoint with a token.
• Python with the openai package installed.

Send a chat request

Set the base URL to the root endpoint plus /v1/, and the API key to your auth token:

from openai import OpenAI

client = OpenAI(
    base_url="<root-endpoint>/v1/",
    api_key="<auth-token>",
)

response = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Who won the world series in 2020?"}],
)
print(response.choices[0].message.content)

• <root-endpoint> is the address from the endpoint’s Overview.
• <auth-token> is a token from Personal Access Tokens.
• <model-name> is the Hugging Face ID of the model, /mnt/models for a model from a repository, or peft-model for a LoRA adapter.

Stream the answer

Add stream=True and read the chunks as they arrive:

stream = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Who won the world series in 2020?"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Check the result

The first call prints the reply, and the second prints it piece by piece. The chat route works for instruct or chat models. A base model needs a chat template in the request.

For completion requests and the full list of request parameters, see Call a vLLM endpoint with the OpenAI client. For an async call, the OpenAI sample uses <root-endpoint>/async/v1/ as the base URL. See Set up async invocation. If the call fails, see Troubleshooting Model Endpoints.