VLLM with OpenAI client

vLLM provides an HTTP server that implements OpenAI’s Completions and Chat API.

Using OpenAI Completions API with vLLM

Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the openai python package:

import openai

auth_token = "$AUTH_TOKEN"  # put your auth token here...

openai.api_key = auth_token

openai.base_url = ""

completion = client.completions.create(model="meta-llama/Meta-Llama-3-8B-Instruct",
                                  prompt="San Francisco is a")
print("Completion result:", completion)

Parameters

When using the chat completion feature of the vLLM Serverless Endpoint Worker, you can customize your requests with the following parameters.

Supported Completions inputs and descriptions

Parameter	Type	Default Value	Description
model	str		The model repo that you’ve deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the Examples: Using your RunPod endpoint with OpenAI section.
prompt	Union[List[int], List[List[int]], str, List[str]]		A string, array of strings, array of tokens, or array of token arrays to be used as the input for the model.
suffix	Optional[str]	None	A string to be appended to the end of the generated text.
max_tokens	Optional[int]	16	Maximum number of tokens to generate per output sequence.
temperature	Optional[float]	1.0	Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.
top_p	Optional[float]	1.0	Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.
n	Optional[int]	1	Number of output sequences to return for the given prompt.
stream	Optional[bool]	False	Whether to stream the output.
logprobs	Optional[int]	None	Number of log probabilities to return per output token.
echo	Optional[bool]	False	Whether to echo back the prompt in addition to the completion.
stop	Optional[Union[str, List[str]]]	list	List of strings that stop the generation when they are generated. The returned output will not contain the stop strings.
seed	Optional[int]	None	Random seed to use for the generation.
presence_penalty	Optional[float]	0.0	Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.
frequency_penalty	Optional[float]	0.0	Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.
best_of	Optional[int]	None	Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n. This parameter influences the diversity of the output.
logit_bias	Optional[Dict[str, float]]	None	Dictionary of token IDs to biases.
user	Optional[str]	None	User identifier for personalizing responses. (Unsupported by vLLM)

Parameter	Type	Default Value	Description
top_k	Optional[int]	-1	Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.
ignore_eos	Optional[bool]	False	Whether to ignore the End Of Sentence token and continue generating tokens after the EOS token is generated.
use_beam_search	Optional[bool]	False	Whether to use beam search instead of sampling for generating outputs.
stop_token_ids	Optional[List[int]]	list	List of tokens that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens.
skip_special_tokens	Optional[bool]	True	Whether to skip special tokens in the output.
spaces_between_special_tokens	Optional[bool]	True	Whether to add spaces between special tokens in the output. Defaults to True.
repetition_penalty	Optional[float]	1.0	Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens.
min_p	Optional[float]	0.0	Float that represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable.
length_penalty	Optional[float]	1.0	Float that penalizes sequences based on their length. Used in beam search.
include_stop_str_in_output	Optional[bool]	False	Whether to include the stop strings in output text. Defaults to False.

Using OpenAI Chat API with vLLM for Streaming and Non-Streaming

The vLLM server is designed to support the OpenAI Chat API, allowing you to engage in dynamic conversations with the model. The chat interface is a more interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.

import openai

auth_token = "$AUTH_TOKEN"  # put your auth token here...

openai.api_key = auth_token

openai.base_url = " "

streamer = openai.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What are large language models?",
        },
    ],
    stream=True
)
for chunk in streamer:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end ="")

import openai

auth_token = "$AUTH_TOKEN"

openai.api_key = auth_token

openai.base_url = " "

completion = openai.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "what is artificial intelligence?",
        },
    ],
)
print(completion.choices[0].message.content)

Parameters

When using the chat completion feature of the vLLM Serverless Endpoint Worker, you can customize your requests with the following parameters.

Supported Chat Completions inputs and descriptions

Parameter	Type	Default Value	Description
model	str		The model repo that you’ve deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the Examples: Using your RunPod endpoint with OpenAI section.
prompt	Union[List[int], List[List[int]], str, List[str]]		A string, array of strings, array of tokens, or array of token arrays to be used as the input for the model.
suffix	Optional[str]	None	A string to be appended to the end of the generated text.
max_tokens	Optional[int]	16	Maximum number of tokens to generate per output sequence.
temperature	Optional[float]	1.0	Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.
top_p	Optional[float]	1.0	Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.
n	Optional[int]	1	Number of output sequences to return for the given prompt.
stream	Optional[bool]	False	Whether to stream the output.
logprobs	Optional[int]	None	Number of log probabilities to return per output token.
echo	Optional[bool]	False	Whether to echo back the prompt in addition to the completion.
stop	Optional[Union[str, List[str]]]	list	List of strings that stop the generation when they are generated. The returned output will not contain the stop strings.
seed	Optional[int]	None	Random seed to use for the generation.
presence_penalty	Optional[float]	0.0	Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.
frequency_penalty	Optional[float]	0.0	Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.
best_of	Optional[int]	None	Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n. This parameter influences the diversity of the output.
logit_bias	Optional[Dict[str, float]]	None	Dictionary of token IDs to biases.
user	Optional[str]	None	User identifier for personalizing responses. (Unsupported by vLLM)

Parameter	Type	Default Value	Description
top_k	Optional[int]	-1	Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.
ignore_eos	Optional[bool]	False	Whether to ignore the End Of Sentence token and continue generating tokens after the EOS token is generated.
use_beam_search	Optional[bool]	False	Whether to use beam search instead of sampling for generating outputs.
stop_token_ids	Optional[List[int]]	list	List of tokens that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens.
skip_special_tokens	Optional[bool]	True	Whether to skip special tokens in the output.
spaces_between_special_tokens	Optional[bool]	True	Whether to add spaces between special tokens in the output. Defaults to True.
repetition_penalty	Optional[float]	1.0	Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens.
min_p	Optional[float]	0.0	Float that represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable.
length_penalty	Optional[float]	1.0	Float that penalizes sequences based on their length. Used in beam search.
include_stop_str_in_output	Optional[bool]	False	Whether to include the stop strings in output text. Defaults to False.