Triton Inference

Triton is an open-source, efficient inferencing serving software from Nvidia that offers best-in-class throughput on inference requests. It also enables multitude of options for client-server communication (like http, grpc, dynamic batching, async requests).

If you need to serve more than one model then the flexibility of Triton Inference and TIRs high performance GPU infrastructure is your best bet.

Utilization

Triton can be used to deploy models either on GPU or CPU. It maximizes GPU/CPU utilization with features such as dynamic batching and concurrent model execution. This means if you have a single GPU and can load more than one model if GPU memory is available.

Scalability

You can auto-scale replicas and auto-deploy models across all them with a click of a button.

Application Experience

You get http/rest and gRPC endpoints out of the box. There is also support for real-time, batch (includes in-flight batching) and streaming options for sending inference requests. Models can be updated in production without a downtime.

Quick Start: Tutorial

1. Create a directory model-dir to download the model repository.

TIR NotebookShell (Local)

Start a new notebook in Jupyter labs and run the below command:

# create a directory to download model weights
!mkdir -p model-dir

You may also use terminal to perform the steps in this tutorial by omitting the ! at the start of commands.

2. Download the sample model repository:

Wget (TIR Notebook)Wget (Shell)Github (TIR Notebook)

!cd model-dir && mkdir -p model_repository/simple/1 && wget  -O ./model_repository/simple/1/model.graphdef https://objectstore.e2enetworks.net/iris/model_repository/simple/1/model.graphdef &&  wget  -O ./model_repository/simple/config.pbtxt https://objectstore.e2enetworks.net/iris/model_repository/simple/config.pbtxt

3. Upload the sample models from local directory to a TIR Repository.

TIR NotebookLocal

• Install SDK:

  pip install e2enetworks
  

• Start a python shell or jupyter notebook and import TIR.

  from e2enetworks.cloud import tir
  

• Push models:

  Initiate model repository client.

  model_repo_client = tir.Models()
  

  Define a model.

  model_repo = model_repo_client.create(name='test-triton', model_type='triton')
  

  Upload from model-dir. A new model repository will be created.

  model_repo_client.push_model(model_path='./model-dir', prefix='', model_id=model_repo.id)
  

4. Create a model endpoint to access our model over REST/gRPC:

A model endpoint offers a way to serve your model over REST or gRPC endpoints. TIR will automatically provision a HTTP Server (handler) for your model.

TIR DashboardUsing SDK

Use the simplicity of TIR Dashboard to the endpoint.

1. Go to Model Endpoints in Inference

2. Click Create Endpoint

3. Select Nvidia Triton as framework and Click Continue

4. Select from one of the available GPU plans. You may choose a CPU plan as well for this exercise.

5. Click Next

6. Change endpoint name if desired. Click Next

7. Select the model repository we created in step 3

8. Click Finish to launch a model endpoint server

9. Wait for the endpoint to reach Ready state before moving further

5. Use Triton Client to call the model endpoint

Triton Client is extremely flexible as it can support a number of client settings. Apart from the support for variety of programming languages (C++, Java or Python), you can also utilize features async io, streaming, decoupled connections, etc.

For the sake of simplicity, we will use a python client for that synchronously calls the simple model endpoint.

The triton client repo has examples on more complex usecases like async, streaming, etc.

TIR Notebook

• Install the triton client

  !pip install tritonclient[http]
  

• Set Auth header for accessing TIR API. We will need this dict in every function call we make to triton endpoint.

  headers = {'Authorization': 'Bearer eyJhbfd3zI1......'}
  
  # On TIR notebook you may also use os environment variable to set header. Do confirm that E2E_TIR_ACCESS_TOKEN is set correctly in your notebook.
  
  # headers = {'Authorization': 'Bearer {}'.format(os.environ['E2E_TIR_ACCESS_TOKEN'])}
  

• Set endpoint URL: You can get the endpoint URL through sdk (step 4 above) or from TIR Dashboard by locating the target model’s endpoint.

  endpoint_url="infer.e2enetworks.net/project/<enter your projectid>/endpoint/is-<enter inference service id>/"
  

• Run the following sample code from a notebook cell

  import tritonclient.http as httpclient
  from tritonclient.http import InferenceServerClient
  import gevent.ssl as ssl
  import numpy as np
  
  triton_client = InferenceServerClient(
                  url=endpoint_url,
                  verbose=False,
                  ssl=True,
                  ssl_options={},
                  insecure=False,
                  ssl_context_factory=ssl.create_default_context)
  
  model_name = "simple"
  
  # Create the data for the two input tensors. Initialize the first
  # to unique integers and the second to all ones.
  input0_data = np.arange(start=0, stop=16, dtype=np.int32)
  input0_data = np.expand_dims(input0_data, axis=0)
  input1_data = np.full(shape=(1, 16), fill_value=-1, dtype=np.int32)
  
  # initiate model input variable in triton format
  inputs = []
  outputs = []
  inputs.append(httpclient.InferInput("INPUT0", [1, 16], "INT32"))
  inputs.append(httpclient.InferInput("INPUT1", [1, 16], "INT32"))
  
  # Initialize the data
  inputs[0].set_data_from_numpy(input0_data, binary_data=False)
  inputs[1].set_data_from_numpy(input1_data, binary_data=True)
  
  # initiate model output variable in triton format
  outputs.append(httpclient.InferRequestedOutput("OUTPUT0", binary_data=True))
  outputs.append(httpclient.InferRequestedOutput("OUTPUT1", binary_data=False))
  
  query_params = {"test_1": 1, "test_2": 2}
  
  results = triton_client.infer(
      model_name,
      inputs,
      outputs=outputs,
      query_params=query_params,
      headers=headers,
  )
  print(results)
  

Troubleshooting and Operations Guide

Model Updates

To deploy a new model version or model config is a two step process: * Push updated model (files and config) to Model Repository * Restart the model endpoint service

When you restart the service, TIR will stop the existing container and start a new one. The new container will download the most recent files from the repository.

Metrics and Logging

You can use the dashboard to keep track of resource metrics as well as service level metrics like QPS, P99, P50 etc. The dashboard also reports detailed logs streamed from the running replicas.

Autoscaling

You can configure autoscaling to dynamically launch new replicas when the load is high. Please note, the scaling operation will depend on availability of resources.

Multi-Model Support

Triton allows sharing of GPUs between multiple models. TIR supports multi-model configuration. However, the option to use explicit model where you only load or unload a few selected models is not supported yet. In case this feature is important to you, please feel free to raise a support ticket.

Frequently Asked Questions

• Can I use Triton Server to deploy Large Language Models (LLMs)?

  Yes. We recommnad TensorRT-LLM or vLLM server

• Can Triton server handle streaming or batching requests?

  Yes. The triton client repo has serveral examples.