Serve LLM inference on a GPU node

vLLM and Hugging Face TGI are both production-grade LLM inference servers you can run on a GPU node. Each serves an OpenAI-compatible HTTP API from a Docker image with CUDA already installed, and supports batching and quantization. Each one below starts with the fewest settings that work. For container basics, see Run GPU workloads in Docker.

For managed model endpoints, see TIR.

Pick a GPU card for the model

The card has to fit the model weights plus the KV cache, the memory the model keeps for text it has already read, for your maximum context length and concurrency:

Model size	Minimum card (FP16)	Comfortable card
7B	L4, or A30	L40S, or A100 40GB
13B	L40S, or A100 40GB	A100 80GB
34B	A100 80GB	2x A100 80GB
70B	2x A100 80GB	4x A100 80GB
70B, long context (over 16K)	4x A100 80GB	4x A100 80GB, or an H100 or H200 on TIR

Sharding across multiple cards costs latency; for latency-sensitive work prefer a single card whose memory comfortably fits the model. See Choose a GPU card for full card profiles.

Serve a model with vLLM

vLLM is the highest-throughput option for batched generation on most cards, exposing an OpenAI-compatible API on port 8000:

docker run -d --name vllm \
  --gpus all \
  --restart unless-stopped \
  -p 127.0.0.1:8000:8000 \
  -v /data/models:/root/.cache/huggingface \
  -e HF_TOKEN=<your-token> \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192

Replace <your-token> with a Hugging Face access token from an account that has accepted the model’s license.

For multi-GPU tensor parallelism, add --shm-size=16g and --tensor-parallel-size <N>, where <N> is the number of GPU cards to split the model across:

docker run -d --name vllm \
  --gpus all \
  --restart unless-stopped \
  --shm-size=16g \
  -p 127.0.0.1:8000:8000 \
  -v /data/models:/root/.cache/huggingface \
  -e HF_TOKEN=<your-token> \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192

Test it from the same node:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Hello", "max_tokens": 32}'

Serve a model with TGI

Hugging Face TGI is the alternative when you want first-class streaming and Hugging Face Hub integration; its API is on port 80 inside the container:

docker run -d --name tgi \
  --gpus all \
  --restart unless-stopped \
  --shm-size=16g \
  -p 127.0.0.1:8080:80 \
  -v /data/models:/data \
  -e HF_TOKEN=<your-token> \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192

Add --num-shard <N> for multi-GPU sharding, where <N> is the number of GPU cards on a multi-card plan. Test it:

curl http://localhost:8080/generate \
  -H "Content-Type: application/json" \
  -d '{"inputs": "Hello", "parameters": {"max_new_tokens": 32}}'

Reach the inference API safely

The vLLM and TGI commands on this page both bind to 127.0.0.1; keep the inference port off the public internet, and reach it through:

Path	When to use it
An SSH local-forward tunnel	Personal access, demos, internal use; nothing exposed to the internet.
A load balancer in front of the node	Production traffic, TLS, rate limiting, request logging.
A same-VPC client, private IP only	An internal service calling the model from another node in the same VPC.

To use the model from your own computer, open an SSH tunnel. Run this on your computer, with your node’s username and public IP in place of <username> and <gpu-node-ip>:

ssh -NL 8000:localhost:8000 <username>@<gpu-node-ip>

For the TGI example, which listens on port 8080, use 8080:localhost:8080 instead. The command holds the connection open and prints nothing. While it runs, requests to http://localhost:8000 on your computer reach the inference server on the node. Stop the tunnel with Ctrl+C.

If you do open the port directly, restrict the security group to specific source IPs.

Sizing reminders

Watch GPU memory in the Monitoring tab: if it stays near 100%, lower --max-model-len or concurrency. Persistent low GPU utilization usually means the request rate is the bottleneck, not the card; raise concurrency or batch size before moving to a bigger one. For long-context models, the KV cache dominates memory; FP8 on Hopper GPUs, on TIR, roughly halves it. See Troubleshooting GPU Node for out-of-memory and throttling fixes.