--- title: "Serve a model with NVIDIA Triton" description: "Upload a Triton model repository to a Model Repository, deploy it on the NVIDIA Triton framework as a Model Endpoint, and call it with the Triton Python client." content_type: "how-to" product: "model-endpoints" platform: "TIR" url: "/tir/model-endpoints/how-to/serve-a-model-with-nvidia-triton/" topic: ["tutorials"] --- # Serve a model with NVIDIA Triton Serve a model with NVIDIA Triton Inference Server. Triton is open-source serving software from NVIDIA. It answers over HTTP and gRPC, batches requests, and can serve several models from one repository. You store a Triton model repository in a Model Repository, deploy it on the **NVIDIA Triton** framework, and call it with the Triton client. The steps use a small sample model, so you can see the whole path before you use your own. ## Before you begin - Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service. You also need permission to create model repositories. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/). - A terminal with `wget`, on your computer or in an instance. - The MinIO CLI (`mc`) or another tool that can upload to a bucket. See [Connect to a repository with the MinIO CLI or s3cmd](/tir/model-repository/how-to/connect-to-a-repository-with-an-s3-tool/). - Python 3 for the client call. ## Download the sample model repository The sample is Triton's `simple` model: it takes two lists of 16 integers and returns their sum and their difference. A Triton model repository has one folder for each model, with a `config.pbtxt` file and a numbered folder for each version. ```bash mkdir -p model-dir/model_repository/simple/1 cd model-dir wget -O ./model_repository/simple/1/model.graphdef https://objectstore.e2enetworks.net/iris/model_repository/simple/1/model.graphdef wget -O ./model_repository/simple/config.pbtxt https://objectstore.e2enetworks.net/iris/model_repository/simple/config.pbtxt ``` To start from more of NVIDIA's examples, clone Triton's repository and run its fetch_models.sh script. The script collects the sample models into a model_repository folder: ```bash git clone -b r23.10 https://github.com/triton-inference-server/server.git ./server/docs/examples/fetch_models.sh ``` Keep the model_repository folder under the `model-dir` folder. ## Upload it to a Model Repository 1. Create a repository with **Triton** as the **Model Type**. See [Create a repository](/tir/model-repository/how-to/create-a-repository/). Choose a name you want to keep. 2. Upload the contents of the model_repository folder, so that the `simple` folder sits at the top of the repository. See [Upload model files](/tir/model-repository/how-to/upload-model-files/). With the MinIO CLI, the command looks like this. `` is the name of the alias you set up, and `` is the repository's bucket: ```bash mc cp -r model_repository/* // ``` 3. Open the repository's **Model Files** tab and check that `simple` is listed. ## Create the endpoint Create an endpoint with these values, and leave every other field as the form fills it. For what each field does, see [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/). | Field | Value | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Framework on **Choose Framework** | **NVIDIA Triton** | | **Endpoint Name** | `triton-simple`, or a name of your own | | **Download Type** | **Link with Model Repository**, the only choice for this framework | | **Model Repository** | The repository you uploaded to | | **Model Path** | Empty, because the models sit at the top of the repository | | **Runtime Configuration** | The values TIR shows | | **Select Computing Resource** | A **CPU** plan, which is enough for the sample model, or a **GPU** card if the version you chose offers only GPU plans | | **Active Worker** and **Max Workers** | 1 and 1 | Select **Deploy Endpoint**, and wait until the state is **Running**. ## Call the endpoint On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `` is the address you copied and `` is the token: ```bash export ROOT_ENDPOINT="" export AUTH_TOKEN="" ``` 1. Check that Triton is ready: ```bash curl -H "Authorization: Bearer $AUTH_TOKEN" "$ROOT_ENDPOINT/v2/health/ready" ``` A ready server answers with HTTP status 200. 2. Install the Triton client: ```bash pip install "tritonclient[http]" gevent numpy ``` 3. Send the sample request. The client takes the root endpoint with `https://` removed: ```python import os import gevent.ssl as ssl import numpy as np import tritonclient.http as httpclient from tritonclient.http import InferenceServerClient headers = {"Authorization": "Bearer " + os.environ["AUTH_TOKEN"]} endpoint_url = os.environ["ROOT_ENDPOINT"].replace("https://", "").rstrip("/") triton_client = InferenceServerClient( url=endpoint_url, verbose=False, ssl=True, ssl_options={}, insecure=False, ssl_context_factory=ssl.create_default_context, ) input0_data = np.expand_dims(np.arange(start=0, stop=16, dtype=np.int32), axis=0) input1_data = np.full(shape=(1, 16), fill_value=-1, dtype=np.int32) inputs = [ httpclient.InferInput("INPUT0", [1, 16], "INT32"), httpclient.InferInput("INPUT1", [1, 16], "INT32"), ] inputs[0].set_data_from_numpy(input0_data, binary_data=False) inputs[1].set_data_from_numpy(input1_data, binary_data=True) outputs = [ httpclient.InferRequestedOutput("OUTPUT0", binary_data=True), httpclient.InferRequestedOutput("OUTPUT1", binary_data=False), ] results = triton_client.infer( "simple", inputs, outputs=outputs, headers=headers, ) print("OUTPUT0 (sum):", results.as_numpy("OUTPUT0")) print("OUTPUT1 (difference):", results.as_numpy("OUTPUT1")) ``` **Sample API Request** also shows Python and gRPC calls for a Triton endpoint. ## Check the sample output The script prints two lists of 16 numbers. `OUTPUT0` holds `INPUT0 + INPUT1`: -1, 0, 1, and on up to 14. `OUTPUT1` holds `INPUT0 - INPUT1`: 1, 2, 3, and on up to 16. ## Update the model To change the model, upload the new model files and the changed `config.pbtxt` to the repository, then select **Actions** > **Restart Service** on the endpoint. The endpoint loads the new files when its workers restart. ## Operate the endpoint - **Monitoring** shows hardware use. See [Monitor an endpoint](/tir/model-endpoints/how-to/monitor-an-endpoint/). - To scale on demand, set **Active Worker** lower than **Max Workers** in **Serverless Configuration**. See [Set workers and autoscaling](/tir/model-endpoints/how-to/set-workers-and-autoscaling/). - One endpoint can serve several models from the same repository, and several models can share one GPU. To load or unload a model, restart the endpoint. - For language models, use [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) or a [vLLM server inside Triton](https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton). - See [Triton's client examples](https://github.com/triton-inference-server/client/tree/main/src/python/examples) for batching, streaming, and async requests. To serve your own models, upload your own Triton model repository in place of the sample and restart the endpoint. The same address and client code then reach your models by their folder names.