Serve a PyTorch model with TorchServe

Serve a PyTorch model through the PyTorch framework. The framework runs TorchServe, which serves one or more models over REST and gRPC and takes its settings from a config.properties file. You package your model as a .mar archive, put the archive and the config file in a Model Repository, and deploy. The steps use TorchServe’s MNIST digit classifier, so you can run the whole path with a ready-made model.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service. You also need permission to create model repositories. See Create an endpoint.
• A terminal with wget, on your computer or in an instance.
• The MinIO CLI (mc) or another tool that can upload to a bucket. An instance has mc installed. See Connect to a repository with the MinIO CLI or s3cmd.
• A small PNG image of a handwritten digit, for the test call. Any image from the MNIST test set works.

Get the sample model and the config file

mkdir mnist && mkdir ./mnist/model-store && mkdir ./mnist/config
cd mnist/model-store
wget https://objectstore.e2enetworks.net/iris/mnist/model-store/mnist.mar
cd ../config
wget https://objectstore.e2enetworks.net/iris/mnist/config/config.properties
cd ..

The mnist folder now has two folders: model-store with the mnist.mar archive, and config with config.properties. The framework looks for the archive in model-store and for the settings in config.

Upload them to a Model Repository

1. Create a repository with PyTorch as the Model Type. See Create a repository.

2. Upload the contents of the mnist folder, where <repository-name> is the alias you set up with mc and <bucket-name> is the repository’s bucket. See Upload model files.

   mc cp -r * <repository-name>/<bucket-name>

3. On the repository’s Model Files tab, check that model-store and config are listed.

Create the endpoint

Create an endpoint with these values, and leave every other field as the form fills it. For what each field does, see Deploy a model from Model Repository.

Field	Value
Framework on Choose Framework	PyTorch
Endpoint Name	mnist-torchserve, or a name of your own
Download Type	Link with Model Repository, the only choice for this framework
Model Repository	The repository you uploaded to
Model Path	Empty, because model-store and config sit at the top of the repository
Runtime Configuration	The values TIR shows
Select Computing Resource	A CPU plan, which is enough for the small MNIST model, or a GPU card if the version you chose offers only GPU plans
Active Worker and Max Workers	1 and 1

Select Deploy Endpoint, and wait until the state is Running.

Call the endpoint

On the endpoint’s details page, select API Request. In Sample API Request, choose a token under Auth token, or select Generate Token to make one, and copy the Root Endpoint. Set both as variables in your terminal, where <root-endpoint> is the address you copied and <auth-token> is the token:

export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"

1. Check the model’s status:

   curl -H "Authorization: Bearer $AUTH_TOKEN" "$ROOT_ENDPOINT/v1/models/mnist"

2. Send a digit image. digit.png is the PNG you chose, and the image goes in the request as a base64 string:

   IMAGE=$(base64 < digit.png | tr -d '\n')
   curl -X POST "$ROOT_ENDPOINT/v1/models/mnist:predict" \
     -H "Authorization: Bearer $AUTH_TOKEN" \
     -H "Content-Type: application/json" \
     -d "{\"instances\": [{\"data\": \"$IMAGE\"}]}"

To send several images in one request, add one {"data": ...} entry to instances for each image. Sample API Request also shows the call as cURL and gRPC.

Check the predictions

The status call returns details of the mnist model. The predict call returns a predictions list with one entry for each image, holding the digit the model chose. If the call fails, open Worker Logs on the Events & Logs tab. Check that the archive is in model-store and that its file name matches the model name in the URL.

Package your own model

To serve your own model, create a .mar archive with the Torch Model Archiver:

torch-model-archiver --model-name mnist --version 1.0 \
  --model-file mnist.py --serialized-file mnist.pt --handler image_classifier \
  --export-path model-store

• --model-name is the name the model has in the URL, for example mnist in /v1/models/mnist:predict.
• --version is an optional version tag.
• --model-file is the file that defines the model.
• --serialized-file is the file with the trained weights.
• --handler is a handler that TorchServe provides, or your own. See TorchServe’s handlers.

Then upload the archive to model-store and your settings to config, the way you uploaded the sample in Upload them to a Model Repository. A config.properties file for the sample looks like this:

metrics_format=prometheus
number_of_netty_threads=4
job_queue_size=10
enable_envvars_config=true
install_py_dep_per_model=true

Setting metrics_format=prometheus publishes TorchServe’s metrics in Prometheus format. For TorchServe’s own examples, see its examples folder.

Update the model

To publish a new archive or a new config file, upload it to the repository, then select Actions > Restart Service on the endpoint. The endpoint downloads the repository again when its workers restart.

Watch the endpoint

• Events & Logs > Worker Logs shows TorchServe’s log lines.
• Monitoring shows hardware use. See Monitor an endpoint.

Go further

• To add your own server code around a model, build a container and deploy it on the Custom Container framework. See Deploy a custom API handler on a Custom Container.
• For models too large for one GPU, see TorchServe’s large model inference guide.