---
title: "Serve a PyTorch model with TorchServe"
description: "Package a PyTorch model as a TorchServe archive, upload it to a Model Repository, deploy it on the PyTorch framework, and send it an image to classify."
content_type: "how-to"
product: "model-endpoints"
platform: "TIR"
url: "/tir/model-endpoints/how-to/serve-a-model-with-pytorch/"
topic: ["tutorials"]
---

# Serve a PyTorch model with TorchServe

Serve a PyTorch model through the **PyTorch** framework. The framework runs TorchServe, which serves one or more models over REST and gRPC and takes its settings from a `config.properties` file. You package your model as a `.mar` archive, put the archive and the config file in a Model Repository, and deploy. The steps use TorchServe's MNIST digit classifier, so you can run the whole path with a ready-made model.

## Before you begin

- Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service. You also need permission to create model repositories. See [Create an endpoint](/tir/model-endpoints/how-to/create-an-endpoint/).
- A terminal with `wget`, on your computer or in an instance.
- The MinIO CLI (`mc`) or another tool that can upload to a bucket. An instance has `mc` installed. See [Connect to a repository with the MinIO CLI or s3cmd](/tir/model-repository/how-to/connect-to-a-repository-with-an-s3-tool/).
- A small PNG image of a handwritten digit, for the test call. Any image from the MNIST test set works.

## Get the sample model and the config file

```bash
mkdir mnist && mkdir ./mnist/model-store && mkdir ./mnist/config
cd mnist/model-store
wget https://objectstore.e2enetworks.net/iris/mnist/model-store/mnist.mar
cd ../config
wget https://objectstore.e2enetworks.net/iris/mnist/config/config.properties
cd ..
```

The `mnist` folder now has two folders: `model-store` with the `mnist.mar` archive, and `config` with `config.properties`. The framework looks for the archive in `model-store` and for the settings in `config`.

## Upload them to a Model Repository

1. Create a repository with **PyTorch** as the **Model Type**. See [Create a repository](/tir/model-repository/how-to/create-a-repository/).
2. Upload the contents of the `mnist` folder, where `<repository-name>` is the alias you set up with `mc` and `<bucket-name>` is the repository's bucket. See [Upload model files](/tir/model-repository/how-to/upload-model-files/).

   ```bash
   mc cp -r * <repository-name>/<bucket-name>
   ```

3. On the repository's **Model Files** tab, check that `model-store` and `config` are listed.

## Create the endpoint

Create an endpoint with these values, and leave every other field as the form fills it. For what each field does, see [Deploy a model from Model Repository](/tir/model-endpoints/how-to/deploy-a-model-from-model-repository/).

| Field                                 | Value                                                                                                                       |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Framework on **Choose Framework**     | **PyTorch**                                                                                                                 |
| **Endpoint Name**                     | `mnist-torchserve`, or a name of your own                                                                                   |
| **Download Type**                     | **Link with Model Repository**, the only choice for this framework                                                          |
| **Model Repository**                  | The repository you uploaded to                                                                                              |
| **Model Path**                        | Empty, because `model-store` and `config` sit at the top of the repository                                                  |
| **Runtime Configuration**             | The values TIR shows                                                                                                        |
| **Select Computing Resource**         | A **CPU** plan, which is enough for the small MNIST model, or a **GPU** card if the version you chose offers only GPU plans |
| **Active Worker** and **Max Workers** | 1 and 1                                                                                                                     |

Select **Deploy Endpoint**, and wait until the state is **Running**.

## Call the endpoint

On the endpoint's details page, select **API Request**. In **Sample API Request**, choose a token under **Auth token**, or select **Generate Token** to make one, and copy the **Root Endpoint**. Set both as variables in your terminal, where `<root-endpoint>` is the address you copied and `<auth-token>` is the token:

```bash
export ROOT_ENDPOINT="<root-endpoint>"
export AUTH_TOKEN="<auth-token>"
```

1. Check the model's status:

   ```bash
   curl -H "Authorization: Bearer $AUTH_TOKEN" "$ROOT_ENDPOINT/v1/models/mnist"
   ```

2. Send a digit image. `digit.png` is the PNG you chose, and the image goes in the request as a base64 string:

   ```bash
   IMAGE=$(base64 < digit.png | tr -d '\n')
   curl -X POST "$ROOT_ENDPOINT/v1/models/mnist:predict" \
     -H "Authorization: Bearer $AUTH_TOKEN" \
     -H "Content-Type: application/json" \
     -d "{\"instances\": [{\"data\": \"$IMAGE\"}]}"
   ```

To send several images in one request, add one `{"data": ...}` entry to `instances` for each image. **Sample API Request** also shows the call as cURL and gRPC.

## Check the predictions

The status call returns details of the `mnist` model. The predict call returns a `predictions` list with one entry for each image, holding the digit the model chose. If the call fails, open **Worker Logs** on the **Events & Logs** tab. Check that the archive is in `model-store` and that its file name matches the model name in the URL.

## Package your own model

To serve your own model, create a `.mar` archive with the Torch Model Archiver:

```bash
torch-model-archiver --model-name mnist --version 1.0 \
  --model-file mnist.py --serialized-file mnist.pt --handler image_classifier \
  --export-path model-store
```

- `--model-name` is the name the model has in the URL, for example `mnist` in `/v1/models/mnist:predict`.
- `--version` is an optional version tag.
- `--model-file` is the file that defines the model.
- `--serialized-file` is the file with the trained weights.
- `--handler` is a handler that TorchServe provides, or your own. See [TorchServe's handlers](https://github.com/pytorch/serve/tree/master/ts/torch_handler).

Then upload the archive to `model-store` and your settings to `config`, the way you uploaded the sample in [Upload them to a Model Repository](#upload-them-to-a-model-repository). A `config.properties` file for the sample looks like this:

```properties
metrics_format=prometheus
number_of_netty_threads=4
job_queue_size=10
enable_envvars_config=true
install_py_dep_per_model=true
```

Setting metrics_format=prometheus publishes TorchServe's metrics in Prometheus format. For TorchServe's own examples, see its [examples folder](https://github.com/pytorch/serve/tree/master/examples).

## Update the model

To publish a new archive or a new config file, upload it to the repository, then select **Actions** > **Restart Service** on the endpoint. The endpoint downloads the repository again when its workers restart.

## Watch the endpoint

- **Events & Logs** > **Worker Logs** shows TorchServe's log lines.
- **Monitoring** shows hardware use. See [Monitor an endpoint](/tir/model-endpoints/how-to/monitor-an-endpoint/).

## Go further

- To add your own server code around a model, build a container and deploy it on the **Custom Container** framework. See [Deploy a custom API handler on a Custom Container](/tir/model-endpoints/how-to/deploy-a-custom-api-handler-on-a-custom-container/).
- For models too large for one GPU, see TorchServe's [large model inference guide](https://pytorch.org/serve/large_model_inference.html).