---
title: "Choose a framework"
description: "See the eight frameworks that Choose Framework offers for a Model Endpoint, pick a version, and find out what the choice changes on the create form."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/choose-a-framework/"
topic: ["create"]
---

# Choose a framework

Choose the framework that serves your model. The framework decides how the model is loaded, which machines you can pick, and which sample requests the endpoint shows.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
- A clear idea of what you are serving: a model on Hugging Face, model files you uploaded, or your own container image.

## The frameworks

TIR offers eight frameworks, each in several versions. If you serve a language model from Hugging Face, vLLM is the place to start: it has the most features on TIR.

| Framework        | Choose it for                                                                                                      | Where the model comes from                                        |
| ---------------- | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| vLLM             | Chat and text-generation models. Offers the OpenAI-style samples, the Playground, LoRA adapters, and tool calling. | Hugging Face or a Model Repository                                |
| SGLang           | Language models and vision language models, with chat and completion calls.                                        | Hugging Face or a Model Repository                                |
| NVIDIA Dynamo    | Language models on NVIDIA's own serving framework.                                                                 | Hugging Face or a Model Repository                                |
| vLLM-Omni        | Models from the vLLM project that work with images, audio, or video as well as text.                               | Hugging Face or a Model Repository                                |
| PyTorch          | A PyTorch model packaged as a TorchServe `.mar` archive.                                                           | A Model Repository                                                |
| NVIDIA Triton    | A Triton model repository, served over HTTP and gRPC.                                                              | A Model Repository                                                |
| TensorRT         | An engine built with TensorRT-LLM.                                                                                 | A Model Repository                                                |
| Custom Container | Your own Docker image, when your model or serving code goes beyond the other frameworks.                           | Your image, with optional files from a Model Repository or GitHub |

For worked examples, see [Serve a PyTorch model with TorchServe](/tir/model-endpoints/how-to/serve-a-model-with-pytorch/), [Serve a model with NVIDIA Triton](/tir/model-endpoints/how-to/serve-a-model-with-nvidia-triton/), and [Deploy a custom container](/tir/model-endpoints/how-to/deploy-a-custom-container/).

## Choose the framework and version

1. In the side menu, under **Inference**, select **Model Endpoints**, then **Create Endpoint**.
2. Find your framework. To find it faster, type in **Search Framework**, which matches a framework's name or its newest version.
3. Choose a version under **Version**.
4. Select the framework. You can still change the version at the top of the create form.

## What the choice changes

- **Download Type** offers different options by framework. vLLM, SGLang, NVIDIA Dynamo, and vLLM-Omni offer **Download from Huggingface** and **Link with Model Repository**. Custom Container offers **Skip Model Download**, **Link with Model Repository**, and **Link with Github**. TensorRT, PyTorch, and NVIDIA Triton offer only **Link with Model Repository**.
- **Multi Node** machines are offered for vLLM, SGLang, and Custom Container.
- Only some frameworks run on CPU plans, and each framework offers only the GPU cards it can use.
- The **Playground** works with the vLLM family of frameworks.
- The sample requests on **Sample API Request** depend on the framework. vLLM and SGLang have **cURL** and **OpenAI** samples. Custom Container has a **cURL** sample for you to fill in.

You can change the framework and its version later; see [Update an endpoint](/tir/model-endpoints/how-to/update-an-endpoint/).

## Check the framework

The framework and its version are under **Framework** on the endpoint's **Overview**. To continue the setup, see [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/).