---
title: "Deploy Gemma with vLLM"
description: "Serve Google Gemma 2B-it, or another Gemma size, from Hugging Face on vLLM as a Model Endpoint and call it with curl."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/deploy-gemma-with-vllm/"
topic: ["tutorials"]
---

# Deploy Gemma with vLLM

Deploy Google Gemma on vLLM and send it a chat request. The examples use the 2B instruction-tuned model, `google/gemma-2b-it`. The same steps work for `google/gemma-2b`, `google/gemma-7b`, and `google/gemma-7b-it`. The models whose names end in `-it` answer chat requests. The endpoint downloads the model from Hugging Face when it starts, so there is nothing to upload first.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
- Access to the model. Gemma is a gated model: on its Hugging Face page, [google/gemma-2b-it](https://huggingface.co/google/gemma-2b-it), accept Google's terms and wait until access is granted.
- A Hugging Face access token with **Read** scope from the account that was granted access, saved as a Hugging Face integration. See [Create a Hugging Face integration](/tir/integrations/how-to/create-a-hugging-face-integration/).

## Create the endpoint

Create a vLLM endpoint that downloads the model from Hugging Face, with these values. Leave every other field as the form fills it. For what each field does, see [Deploy a model from Hugging Face](/tir/model-endpoints/how-to/deploy-a-model-from-hugging-face/).

| Field                                 | Value                                                                                                                                 |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Framework on **Choose Framework**     | **vLLM**                                                                                                                              |
| **Endpoint Name**                     | `gemma-2b-it`, or a name of your own                                                                                                  |
| **Download Type**                     | **Download from Huggingface**                                                                                                         |
| **HuggingFace Model**                 | `google/gemma-2b-it`                                                                                                                  |
| **HF Token Integration**              | Your Hugging Face integration                                                                                                         |
| **Model Cache Storage**               | **Local drive**, 100 GB                                                                                                               |
| **Select Computing Resource**         | **GPU** tab, any card, and a size under **Configuration**. The 2B model needs about 5 GB of GPU memory; a 7B model needs about 17 GB. |
| **Active Worker** and **Max Workers** | 1 and 1                                                                                                                               |

Select **Deploy Endpoint**. The state reads **Deploying** while the worker downloads and loads the model, and **Running** once it can answer, usually within 10 to 15 minutes.

## Send a chat request

On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `<root-endpoint>` is the address you copied and `<auth-token>` is the token:

```bash
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

The model name is the Hugging Face ID you entered in the form:

```bash
curl -X POST "$ROOT_ENDPOINT/v1/chat/completions" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-2b-it",
    "messages": [
      {"role": "user", "content": "What is artificial intelligence?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7,
    "top_p": 0.9
  }'
```

The reply is a JSON document in the OpenAI chat format, with the answer in `choices[0].message.content`. For `google/gemma-2b` and `google/gemma-7b`, send a `prompt` to `/v1/completions` instead, and read the text in `choices[0].text`. If the endpoint stays in **Deploying**, read **Worker Logs** on the **Events & Logs** tab.

## Serve your own copy of the weights

For fine-tuned weights, or to load the model from storage you control, upload the files to a Model Repository. Then create the endpoint the same way with two changes: set **Download Type** to **Link with Model Repository** and choose your repository, and set **Model Type** to **Full Model (for standard inference)**. Requests then name the model `/mnt/models`. See [Upload model files](/tir/model-repository/how-to/upload-model-files/) and [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/).

Gemma 2B is small enough to run on any GPU card. That makes it a cheap model for trying out [autoscaling and scale to zero](/tir/model-endpoints/how-to/set-workers-and-autoscaling/) before you move a larger model onto the same settings.