Serve a model with NVIDIA Triton

View as Markdown

Serve a model with NVIDIA Triton Inference Server. Triton is open-source serving software from NVIDIA. It answers over HTTP and gRPC, batches requests, and can serve several models from one repository. You store a Triton model repository in a Model Repository, deploy it on the NVIDIA Triton framework, and call it with the Triton client. The steps use a small sample model, so you can see the whole path before you use your own.

Before you begin

Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service. You also need permission to create model repositories. See Create an endpoint.
A terminal with wget, on your computer or in an instance.
The MinIO CLI (mc) or another tool that can upload to a bucket. See Connect to a repository with the MinIO CLI or s3cmd.
Python 3 for the client call.

Download the sample model repository

The sample is Triton’s simple model: it takes two lists of 16 integers and returns their sum and their difference. A Triton model repository has one folder for each model, with a config.pbtxt file and a numbered folder for each version.

mkdir -p model-dir/model_repository/simple/1
cd model-dir
wget -O ./model_repository/simple/1/model.graphdef https://objectstore.e2enetworks.net/iris/model_repository/simple/1/model.graphdef
wget -O ./model_repository/simple/config.pbtxt https://objectstore.e2enetworks.net/iris/model_repository/simple/config.pbtxt

To start from more of NVIDIA’s examples, clone Triton’s repository and run its fetch_models.sh script. The script collects the sample models into a model_repository folder:

git clone -b r23.10 https://github.com/triton-inference-server/server.git
./server/docs/examples/fetch_models.sh

Keep the model_repository folder under the model-dir folder.

Upload it to a Model Repository

Create a repository with Triton as the Model Type. See Create a repository. Choose a name you want to keep.
Upload the contents of the model_repository folder, so that the simple folder sits at the top of the repository. See Upload model files. With the MinIO CLI, the command looks like this. <repository-name> is the name of the alias you set up, and <bucket-name> is the repository’s bucket:
```
mc cp -r model_repository/* <repository-name>/<bucket-name>/
```
Open the repository’s Model Files tab and check that simple is listed.

Create the endpoint

Create an endpoint with these values, and leave every other field as the form fills it. For what each field does, see Deploy a model from Model Repository.

Field	Value
Framework on Choose Framework	NVIDIA Triton
Endpoint Name	`triton-simple`, or a name of your own
Download Type	Link with Model Repository, the only choice for this framework
Model Repository	The repository you uploaded to
Model Path	Empty, because the models sit at the top of the repository
Runtime Configuration	The values TIR shows
Select Computing Resource	A CPU plan, which is enough for the sample model, or a GPU card if the version you chose offers only GPU plans
Active Worker and Max Workers	1 and 1

Select Deploy Endpoint, and wait until the state is Running.

Call the endpoint

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

Check that Triton is ready:

curl -H "Authorization: Bearer $AUTH_TOKEN" "$ROOT_ENDPOINT/v2/health/ready"

A ready server answers with HTTP status 200.

Install the Triton client:

pip install "tritonclient[http]" gevent numpy

Send the sample request. The client takes the root endpoint with https:// removed:

import os
import gevent.ssl as ssl
import numpy as np
import tritonclient.http as httpclient
from tritonclient.http import InferenceServerClient

headers = {"Authorization": "Bearer " + os.environ["AUTH_TOKEN"]}
endpoint_url = os.environ["ROOT_ENDPOINT"].replace("https://", "").rstrip("/")

triton_client = InferenceServerClient(
    url=endpoint_url,
    verbose=False,
    ssl=True,
    ssl_options={},
    insecure=False,
    ssl_context_factory=ssl.create_default_context,
)

input0_data = np.expand_dims(np.arange(start=0, stop=16, dtype=np.int32), axis=0)
input1_data = np.full(shape=(1, 16), fill_value=-1, dtype=np.int32)

inputs = [
    httpclient.InferInput("INPUT0", [1, 16], "INT32"),
    httpclient.InferInput("INPUT1", [1, 16], "INT32"),
]
inputs[0].set_data_from_numpy(input0_data, binary_data=False)
inputs[1].set_data_from_numpy(input1_data, binary_data=True)

outputs = [
    httpclient.InferRequestedOutput("OUTPUT0", binary_data=True),
    httpclient.InferRequestedOutput("OUTPUT1", binary_data=False),
]

results = triton_client.infer(
    "simple",
    inputs,
    outputs=outputs,
    headers=headers,
)
print("OUTPUT0 (sum):", results.as_numpy("OUTPUT0"))
print("OUTPUT1 (difference):", results.as_numpy("OUTPUT1"))

Sample API Request also shows Python and gRPC calls for a Triton endpoint.

Check the sample output

The script prints two lists of 16 numbers. OUTPUT0 holds INPUT0 + INPUT1: -1, 0, 1, and on up to 14. OUTPUT1 holds INPUT0 - INPUT1: 1, 2, 3, and on up to 16.

Update the model

To change the model, upload the new model files and the changed config.pbtxt to the repository, then select Actions > Restart Service on the endpoint. The endpoint loads the new files when its workers restart.

Operate the endpoint

Monitoring shows hardware use. See Monitor an endpoint.
To scale on demand, set Active Worker lower than Max Workers in Serverless Configuration. See Set workers and autoscaling.
One endpoint can serve several models from the same repository, and several models can share one GPU. To load or unload a model, restart the endpoint.
For language models, use TensorRT-LLM or a vLLM server inside Triton.
See Triton’s client examples for batching, streaming, and async requests.

To serve your own models, upload your own Triton model repository in place of the sample and restart the endpoint. The same address and client code then reach your models by their folder names.