Deploy Gemma with vLLM

Deploy Google Gemma on vLLM and send it a chat request. The examples use the 2B instruction-tuned model, google/gemma-2b-it. The same steps work for google/gemma-2b, google/gemma-7b, and google/gemma-7b-it. The models whose names end in -it answer chat requests. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• Access to the model. Gemma is a gated model: on its Hugging Face page, google/gemma-2b-it, accept Google’s terms and wait until access is granted.
• A Hugging Face access token with Read scope from the account that was granted access, saved as a Hugging Face integration. See Create a Hugging Face integration.

Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see Deploy a model from Hugging Face.

Field	Value
Framework on Choose Framework	vLLM
Endpoint Name	gemma-2b-it, or a name of your own
Download Type	Download from Huggingface
HuggingFace Model	google/gemma-2b-it
HF Token Integration	Your Hugging Face integration
Model Cache Storage	Local drive, 100 GB
Select Computing Resource	GPU tab, any card, and a size under Configuration. The 2B model needs about 5 GB of GPU memory; a 7B model needs about 17 GB.
Active Worker and Max Workers	1 and 1

Select Deploy Endpoint. The state reads Deploying while the worker downloads and loads the model, and Running once it can answer, usually within 10 to 15 minutes.

Send a chat request

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

The model name is the Hugging Face ID you entered in the form:

curl -X POST "$ROOT_ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-2b-it",
    "messages": [
      {"role": "user", "content": "What is artificial intelligence?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7,
    "top_p": 0.9
  }'

The reply is a JSON document in the OpenAI chat format, with the answer in choices[0].message.content. For google/gemma-2b and google/gemma-7b, send a prompt to /v1/completions instead, and read the text in choices[0].text. If the endpoint stays in Deploying, read Worker Logs on the Events & Logs tab.

Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set Download Type to Link with Model Repository and choose your repository, and set Model Type to Full Model (for standard inference). Requests then name the model /mnt/models. See Upload model files and Deploy a model from Model Repository.

Gemma 2B is small enough to run on any GPU card. That makes it a cheap model for trying out autoscaling and scale to zero before you move a larger model onto the same settings.