Deploy a custom container

Run your own Docker image behind a Model Endpoint when your model or your serving code needs more than the other frameworks offer. The endpoint runs your image on the machine you choose and gives it an HTTPS address.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• An image that serves your model on a port, and its address. A private image needs a registry; see Container Registry for E2E Registry.
• Model files, if the container downloads them. They can come from a Model Repository or from a GitHub repository.

Choose Custom Container

1. On Choose Framework, select Custom Container. See Choose a framework.
2. In Download Type, choose what the container needs. It starts on Skip Model Download, which skips the model download at startup. Link with Model Repository downloads the model from the repository you choose when the container starts or restarts; see Deploy a model from Model Repository. Link with Github downloads the model from GitHub.

Set the container

In Container Configuration, choose where the image lives, then fill in the rest.

1. Choose Public or Private.
2. For a public image, type the Image, up to 128 characters, for example nginx:latest.
3. For a private image, choose a Registry Type. With Nvidia NGC Registry, type the image address. With E2E Registry, choose a registry namespace, an image, and a tag. Refresh the list if it is out of date, or follow the link to create a registry.
4. Set Image Pull Policy. Always is the default, and If not present pulls the image only when it is missing from the machine.
5. Set Service Type. External is reachable from the internet through the root endpoint with a token. Internal is reachable only from inside the project’s network.
6. Under Application Port, tick HTTP, gRPC, or both. On an External endpoint, HTTP uses port 8080 and gRPC uses port 9000. Your container must listen on those ports. On an Internal endpoint you can add several HTTP ports: type a port and press Enter or a comma. TIR refuses ports 80 and 8082 as internal HTTP ports. While gRPC is on, port 9000 is reserved for gRPC.
7. Set Metrics Port, from 1 to 65535, where your application serves metrics at /metrics. It starts at 8080.

Set the start command and health checks

Advanced Configuration holds the start command and two health checks.

• Set the start command in three parts. Choose a Serving Framework. vLLM, SGLang, and Triton Inference Server fill in that server’s usual command and arguments, with the port set to 8080. Bash takes your whole command as one shell line, Custom starts empty for your own command, and SGLang Multimodal is on the list too. Then check the Entrypoint Command and add Arguments as rows of Flag and Value with Add argument. Check the whole command under Command Preview, where you can also copy it.
• Turn on Readiness Probe and Liveness Probe if you want them. Each offers three protocols. HTTP takes a URL path. Execute takes a Commands list in square brackets, such as ["cat", "tmp/healthy"]. TCP takes a Port from 1 to 65535. A probe you turn on starts with Period Seconds 10, Timeout Seconds 10, Failure Threshold 3, Initial Delay Seconds 10, and Success Threshold 1, with the readiness path /health, the liveness path /metrics, and port 8080. Only a probe you turn on is applied.

(Optional) Add environment variables

Open Advanced Settings to pass values your container expects. See Set environment variables.

(Optional) Redeploy from GitHub

With Link with Github, type the GitHub Repository as https://github.com/<owner>/<repo>/tree/<branch> and choose a GitHub Integration that can reach it. The form says “GitHub access token has access to the repository.” when the check passes. See Create a GitHub integration.

Before you deploy, add the workflow file at .github/workflows/deploy.yml in your repository. The form offers the file for download. After the endpoint exists, its Integrations tab has Download yml, the endpoint’s UUID to paste into the workflow, and the two repository secrets to add. When someone merges a change, the workflow restarts the endpoint so that it pulls the new code. Download YML is also in the Actions menu.

Deploy and call it

Choose the machine and workers as in Create an endpoint, then select Deploy Endpoint. For an external endpoint, the cURL sample is a request to the root endpoint and a route that you fill in, with your own body. See Call an endpoint with a token. An internal endpoint needs no token and is called at http://is-<endpoint-id>.p-<project-id>:<port>, where the port is the first HTTP port, or 9000 for gRPC only.

Change the container later

To change the image, ports, or commands on a running endpoint, open Actions and select Update Container Configuration. You change the same container settings as at create, except the name and framework.

Check the container

The endpoint becomes Running when its workers are ready. If a worker shows ImagePullBackOff or ErrImagePull, the worker failed to fetch the image. Check the image name, tag, and registry login. CrashLoopBackOff means the container keeps starting and stopping, so read Worker Logs. See View logs and events and Troubleshooting Model Endpoints.