VLLM with OpenAI client

vLLM provides an HTTP server that implements OpenAI’s Completions and Chat API.

Using OpenAI Completions API with vLLM

  • Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the openai python package:

    import openai
    
    auth_token = "$AUTH_TOKEN"  # put your auth token here...
    
    openai.api_key = auth_token
    
    openai.base_url = ""
    
    completion = client.completions.create(model="meta-llama/Meta-Llama-3-8B-Instruct",
                                      prompt="San Francisco is a")
    print("Completion result:", completion)
    

Parameters

When using the chat completion feature of the vLLM Serverless Endpoint Worker, you can customize your requests with the following parameters.

  • Supported Completions inputs and descriptions

    Parameter

    Type

    Default Value

    Description

    model

    str

    The model repo that you’ve deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the Examples: Using your RunPod endpoint with OpenAI section.

    prompt

    Union[List[int], List[List[int]], str, List[str]]

    A string, array of strings, array of tokens, or array of token arrays to be used as the input for the model.

    suffix

    Optional[str]

    None

    A string to be appended to the end of the generated text.

    max_tokens

    Optional[int]

    16

    Maximum number of tokens to generate per output sequence.

    temperature

    Optional[float]

    1.0

    Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.

    top_p

    Optional[float]

    1.0

    Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.

    n

    Optional[int]

    1

    Number of output sequences to return for the given prompt.

    stream

    Optional[bool]

    False

    Whether to stream the output.

    logprobs

    Optional[int]

    None

    Number of log probabilities to return per output token.

    echo

    Optional[bool]

    False

    Whether to echo back the prompt in addition to the completion.

    stop

    Optional[Union[str, List[str]]]

    list

    List of strings that stop the generation when they are generated. The returned output will not contain the stop strings.

    seed

    Optional[int]

    None

    Random seed to use for the generation.

    presence_penalty

    Optional[float]

    0.0

    Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

    frequency_penalty

    Optional[float]

    0.0

    Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

    best_of

    Optional[int]

    None

    Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n. This parameter influences the diversity of the output.

    logit_bias

    Optional[Dict[str, float]]

    None

    Dictionary of token IDs to biases.

    user

    Optional[str]

    None

    User identifier for personalizing responses. (Unsupported by vLLM)

    Parameter

    Type

    Default Value

    Description

    top_k

    Optional[int]

    -1

    Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.

    ignore_eos

    Optional[bool]

    False

    Whether to ignore the End Of Sentence token and continue generating tokens after the EOS token is generated.

    use_beam_search

    Optional[bool]

    False

    Whether to use beam search instead of sampling for generating outputs.

    stop_token_ids

    Optional[List[int]]

    list

    List of tokens that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens.

    skip_special_tokens

    Optional[bool]

    True

    Whether to skip special tokens in the output.

    spaces_between_special_tokens

    Optional[bool]

    True

    Whether to add spaces between special tokens in the output. Defaults to True.

    repetition_penalty

    Optional[float]

    1.0

    Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens.

    min_p

    Optional[float]

    0.0

    Float that represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable.

    length_penalty

    Optional[float]

    1.0

    Float that penalizes sequences based on their length. Used in beam search.

    include_stop_str_in_output

    Optional[bool]

    False

    Whether to include the stop strings in output text. Defaults to False.

Using OpenAI Chat API with vLLM for Streaming and Non-Streaming

  • The vLLM server is designed to support the OpenAI Chat API, allowing you to engage in dynamic conversations with the model. The chat interface is a more interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.

import openai

auth_token = "$AUTH_TOKEN"  # put your auth token here...

openai.api_key = auth_token

openai.base_url = " "

streamer = openai.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What are large language models?",
        },
    ],
    stream=True
)
for chunk in streamer:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end ="")

Parameters

When using the chat completion feature of the vLLM Serverless Endpoint Worker, you can customize your requests with the following parameters.

  • Supported Chat Completions inputs and descriptions

    Parameter

    Type

    Default Value

    Description

    model

    str

    The model repo that you’ve deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the Examples: Using your RunPod endpoint with OpenAI section.

    prompt

    Union[List[int], List[List[int]], str, List[str]]

    A string, array of strings, array of tokens, or array of token arrays to be used as the input for the model.

    suffix

    Optional[str]

    None

    A string to be appended to the end of the generated text.

    max_tokens

    Optional[int]

    16

    Maximum number of tokens to generate per output sequence.

    temperature

    Optional[float]

    1.0

    Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.

    top_p

    Optional[float]

    1.0

    Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.

    n

    Optional[int]

    1

    Number of output sequences to return for the given prompt.

    stream

    Optional[bool]

    False

    Whether to stream the output.

    logprobs

    Optional[int]

    None

    Number of log probabilities to return per output token.

    echo

    Optional[bool]

    False

    Whether to echo back the prompt in addition to the completion.

    stop

    Optional[Union[str, List[str]]]

    list

    List of strings that stop the generation when they are generated. The returned output will not contain the stop strings.

    seed

    Optional[int]

    None

    Random seed to use for the generation.

    presence_penalty

    Optional[float]

    0.0

    Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

    frequency_penalty

    Optional[float]

    0.0

    Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens.

    best_of

    Optional[int]

    None

    Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n. This parameter influences the diversity of the output.

    logit_bias

    Optional[Dict[str, float]]

    None

    Dictionary of token IDs to biases.

    user

    Optional[str]

    None

    User identifier for personalizing responses. (Unsupported by vLLM)

    Parameter

    Type

    Default Value

    Description

    top_k

    Optional[int]

    -1

    Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.

    ignore_eos

    Optional[bool]

    False

    Whether to ignore the End Of Sentence token and continue generating tokens after the EOS token is generated.

    use_beam_search

    Optional[bool]

    False

    Whether to use beam search instead of sampling for generating outputs.

    stop_token_ids

    Optional[List[int]]

    list

    List of tokens that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens.

    skip_special_tokens

    Optional[bool]

    True

    Whether to skip special tokens in the output.

    spaces_between_special_tokens

    Optional[bool]

    True

    Whether to add spaces between special tokens in the output. Defaults to True.

    repetition_penalty

    Optional[float]

    1.0

    Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens.

    min_p

    Optional[float]

    0.0

    Float that represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable.

    length_penalty

    Optional[float]

    1.0

    Float that penalizes sequences based on their length. Used in beam search.

    include_stop_str_in_output

    Optional[bool]

    False

    Whether to include the stop strings in output text. Defaults to False.