---
title: "Serve LLM inference on a GPU node"
description: "Serve an LLM from a GPU node with vLLM or Hugging Face TGI, behind an OpenAI-compatible API."
content_type: "how-to"
product: "gpu-node"
platform: "MyAccount"
url: "/myaccount/gpu-node/how-to/serve-llm-inference-on-a-gpu-node/"
topic: ["run-and-serve"]
---

# Serve LLM inference on a GPU node

vLLM and Hugging Face TGI are both production-grade LLM inference servers you can run on a GPU
node. Each serves an OpenAI-compatible HTTP API from a Docker image with CUDA already installed,
and supports batching and quantization. Each one below starts with the fewest settings that work. For container basics, see [Run GPU workloads in Docker](/myaccount/gpu-node/how-to/run-gpu-workloads-in-docker/).

For managed model endpoints, see [TIR](/tir/).

## Pick a GPU card for the model

The card has to fit the model weights plus the
[KV cache](/myaccount/gpu-node/concepts/choose-a-gpu-card/#how-to-choose-by-workload), the memory
the model keeps for text it has already read, for your maximum context length and concurrency:

| Model size                   | Minimum card (FP16) | Comfortable card                                 |
| ---------------------------- | ------------------- | ------------------------------------------------ |
| 7B                           | L4, or A30          | L40S, or A100 40GB                               |
| 13B                          | L40S, or A100 40GB  | A100 80GB                                        |
| 34B                          | A100 80GB           | 2x A100 80GB                                     |
| 70B                          | 2x A100 80GB        | 4x A100 80GB                                     |
| 70B, long context (over 16K) | 4x A100 80GB        | 4x A100 80GB, or an H100 or H200 on [TIR](/tir/) |

Sharding across multiple cards costs latency; for latency-sensitive work prefer a single card
whose memory comfortably fits the model. See
[Choose a GPU card](/myaccount/gpu-node/concepts/choose-a-gpu-card/) for full card profiles.

## Serve a model with vLLM

vLLM is the highest-throughput option for batched generation on most cards, exposing an
OpenAI-compatible API on port `8000`:

```bash frame="none"
docker run -d --name vllm \
  --gpus all \
  --restart unless-stopped \
  -p 127.0.0.1:8000:8000 \
  -v /data/models:/root/.cache/huggingface \
  -e HF_TOKEN=<your-token> \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192
```

Replace `<your-token>` with a Hugging Face access token from an account that has accepted the
model's license.

For multi-GPU tensor parallelism, add `--shm-size=16g` and `--tensor-parallel-size <N>`, where
`<N>` is the number of GPU cards to split the model across:

```bash frame="none"
docker run -d --name vllm \
  --gpus all \
  --restart unless-stopped \
  --shm-size=16g \
  -p 127.0.0.1:8000:8000 \
  -v /data/models:/root/.cache/huggingface \
  -e HF_TOKEN=<your-token> \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192
```

Test it from the same node:

```bash frame="none"
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Hello", "max_tokens": 32}'
```

## Serve a model with TGI

Hugging Face TGI is the alternative when you want first-class streaming and Hugging Face Hub
integration; its API is on port `80` inside the container:

```bash frame="none"
docker run -d --name tgi \
  --gpus all \
  --restart unless-stopped \
  --shm-size=16g \
  -p 127.0.0.1:8080:80 \
  -v /data/models:/data \
  -e HF_TOKEN=<your-token> \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192
```

Add `--num-shard <N>` for multi-GPU sharding, where `<N>` is the number of GPU cards on a multi-card plan. Test it:

```bash frame="none"
curl http://localhost:8080/generate \
  -H "Content-Type: application/json" \
  -d '{"inputs": "Hello", "parameters": {"max_new_tokens": 32}}'
```

## Reach the inference API safely

The vLLM and TGI commands on this page both bind to `127.0.0.1`; keep the inference port off the
public internet, and reach it through:

| Path                                 | When to use it                                                           |
| ------------------------------------ | ------------------------------------------------------------------------ |
| An SSH local-forward tunnel          | Personal access, demos, internal use; nothing exposed to the internet.   |
| A load balancer in front of the node | Production traffic, TLS, rate limiting, request logging.                 |
| A same-VPC client, private IP only   | An internal service calling the model from another node in the same VPC. |

To use the model from your own computer, open an SSH tunnel. Run this on your computer, with your node's username and public IP in place of `<username>` and `<gpu-node-ip>`:

```bash frame="none"
ssh -NL 8000:localhost:8000 <username>@<gpu-node-ip>
```

For the TGI example, which listens on port `8080`, use `8080:localhost:8080` instead. The
command holds the connection open and prints nothing. While it runs, requests to
`http://localhost:8000` on your computer reach the inference server on the node. Stop the tunnel
with Ctrl+C.

If you do open the port directly, restrict the security group to specific source IPs.

## Sizing reminders

Watch GPU memory in the Monitoring tab: if it stays near 100%, lower `--max-model-len` or
concurrency. Persistent low GPU utilization usually means the request rate is the bottleneck,
not the card; raise concurrency or batch size before moving to a bigger one. For long-context
models, the KV cache dominates memory; FP8 on Hopper GPUs, on TIR, roughly halves it. See
[Troubleshooting GPU Node](/myaccount/gpu-node/troubleshooting/gpu-node/) for out-of-memory and
throttling fixes.