Choose a framework

Choose the framework that serves your model. The framework decides how the model is loaded, which machines you can pick, and which sample requests the endpoint shows.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• A clear idea of what you are serving: a model on Hugging Face, model files you uploaded, or your own container image.

The frameworks

TIR offers eight frameworks, each in several versions. If you serve a language model from Hugging Face, vLLM is the place to start: it has the most features on TIR.

Framework	Choose it for	Where the model comes from
vLLM	Chat and text-generation models. Offers the OpenAI-style samples, the Playground, LoRA adapters, and tool calling.	Hugging Face or a Model Repository
SGLang	Language models and vision language models, with chat and completion calls.	Hugging Face or a Model Repository
NVIDIA Dynamo	Language models on NVIDIA’s own serving framework.	Hugging Face or a Model Repository
vLLM-Omni	Models from the vLLM project that work with images, audio, or video as well as text.	Hugging Face or a Model Repository
PyTorch	A PyTorch model packaged as a TorchServe .mar archive.	A Model Repository
NVIDIA Triton	A Triton model repository, served over HTTP and gRPC.	A Model Repository
TensorRT	An engine built with TensorRT-LLM.	A Model Repository
Custom Container	Your own Docker image, when your model or serving code goes beyond the other frameworks.	Your image, with optional files from a Model Repository or GitHub

For worked examples, see Serve a PyTorch model with TorchServe, Serve a model with NVIDIA Triton, and Deploy a custom container.

Choose the framework and version

1. In the side menu, under Inference, select Model Endpoints, then Create Endpoint.
2. Find your framework. To find it faster, type in Search Framework, which matches a framework’s name or its newest version.
3. Choose a version under Version.
4. Select the framework. You can still change the version at the top of the create form.

What the choice changes

• Download Type offers different options by framework. vLLM, SGLang, NVIDIA Dynamo, and vLLM-Omni offer Download from Huggingface and Link with Model Repository. Custom Container offers Skip Model Download, Link with Model Repository, and Link with Github. TensorRT, PyTorch, and NVIDIA Triton offer only Link with Model Repository.
• Multi Node machines are offered for vLLM, SGLang, and Custom Container.
• Only some frameworks run on CPU plans, and each framework offers only the GPU cards it can use.
• The Playground works with the vLLM family of frameworks.
• The sample requests on Sample API Request depend on the framework. vLLM and SGLang have cURL and OpenAI samples. Custom Container has a cURL sample for you to fill in.

You can change the framework and its version later; see Update an endpoint.

Check the framework

The framework and its version are under Framework on the endpoint’s Overview. To continue the setup, see Create an endpoint.