Deploy a model from Model Repository

Deploy a model whose files are in a Model Repository. The endpoint downloads the model from the repository when its container starts or restarts. Use this for a fine-tuned model or a LoRA adapter. The same steps are in Use a repository from a Model Endpoint.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• A repository with the model files in it, in the same project and region as the endpoint. See Upload model files.
• The folder the model sits in, if it is in a folder rather than at the top of the repository’s bucket. If you uploaded to v1 or production, give that folder.
• For a LoRA adapter on vLLM, a repository that holds the adapter’s configuration file, named adapter_config.json, and a file ending in .safetensors. You also need a Hugging Face integration for the base model. See Create a Hugging Face integration.

Start the endpoint

In Model Endpoints, select Create Endpoint, or select Deploy Model on a repository’s row in Model Repository. Then follow the steps below, and link the repository on the form.

Link the repository

1. Choose a framework that offers Link with Model Repository, such as vLLM, SGLang, NVIDIA Dynamo, Custom Container, TensorRT, PyTorch, or NVIDIA Triton. See Choose a framework.
2. In Download Type, choose Link with Model Repository. TIR downloads the model from the repository when the container starts or restarts. TensorRT, PyTorch, and NVIDIA Triton offer only this choice.
3. In Model Repository, choose one of the project’s repositories in the current region. To check what it holds, follow the link next to it. If there is none yet, create one first.
4. By default, the endpoint looks for the model at the top of the repository’s bucket, s3://<bucket-name>/. To use a folder, type it in Model Path, for example /v1 for a version. Leave Model Path empty for the top of the bucket.
5. (Optional) Select Validate to check that TIR finds model files at the path.

Choose the model type for vLLM

For a vLLM endpoint that loads from a repository, Model Type is required:

• Full Model (for standard inference) serves the model in the repository as it is.
• PEFT Model (for LoRA-based inference) serves a LoRA adapter on top of a base model. Type the Base Model Name. It must match the base model named in the repository’s adapter_config.json file. Choose an HF Token Integration so the endpoint can download the base model.

A PEFT repository must hold adapter_config.json and a file ending in .safetensors. If TIR reports “adapter_config.json or .safetensors files not present in model repo, please check file path”, check the path. If it reports “Base model name does not match with the name given in config file”, correct the Base Model Name.

Deploy

Finish the form and select Deploy Endpoint. See Create an endpoint.

Check the link

When the endpoint is Running, its Overview gives the Model Name, Model Type, and Model Path of a repository model. In requests, the model name is /mnt/models for a model from a repository and peft-model for a LoRA adapter.

If the endpoint cannot find the model files, the folder is empty or missing. Check three things: the files are on the repository’s Model Files tab, Model Path matches the folder layout, and the bucket name is right. Then read the endpoint’s logs. See Troubleshooting Model Endpoints and Troubleshooting Model Repository.

The repository and its files stay in place if you delete the endpoint later.