Use a repository from a Model Endpoint

Link a repository to a Model Endpoint, and the endpoint downloads the model when its container starts or restarts. A repository stores model files. The endpoint is the running service that answers requests. The same steps are in Deploy a model from Model Repository.

Before you begin

• Permission to create model endpoints in the project. A Member gets it through a policy that allows creating for the Inference service. Deploy Model is available with the create permission.
• A repository with the model files in it, in the same project and region as the endpoint. See Upload model files.
• The folder the model sits in, if it is in a subfolder of the repository’s bucket. If you uploaded to v1 or production, give that folder.
• For a LoRA adapter on vLLM, a repository that holds the adapter’s configuration file, named adapter_config.json, and a file ending in .safetensors. You also need a Hugging Face integration for the base model. See Create a Hugging Face integration.

Start from the repository

In Model Repository, find the repository’s row and select Deploy Model. You can also start from Model Endpoints with Create Endpoint. Then follow the steps below.

Link the repository

1. On Choose Framework, choose a framework that takes a repository: vLLM, SGLang, NVIDIA Dynamo, Custom Container, TensorRT, PyTorch, or NVIDIA Triton. Its create form opens. See Choose a framework.
2. In Download Type, choose Link with Model Repository. TIR downloads the model from the repository when the container starts or restarts. TensorRT, PyTorch, and NVIDIA Triton offer only this choice.
3. In Model Repository, choose one of the project’s repositories in the current region. To check what it holds, follow the link next to it. If there is none yet, create one first.
4. By default the endpoint looks for the model at the top of the repository’s bucket, s3://<bucket-name>/, so leave Model Path empty for that. To use a folder, type it in Model Path, for example /v1 for a version.
5. (Optional) Select Validate to check the path. The portal shows “Model artifacts found at given path” or “Model artifacts not found at given path”.

Choose the model type for vLLM

For a vLLM endpoint that loads from a repository, Model Type is required:

• Full Model (for standard inference) serves the model in the repository as it is.
• PEFT Model (for LoRA-based inference) serves a LoRA adapter on top of a base model. Type the Base Model Name. It must match the base model named in the repository’s adapter_config.json file. Choose an HF Token Integration so the endpoint can download the base model.

A PEFT repository must hold adapter_config.json and a file ending in .safetensors. If the portal shows “adapter_config.json or .safetensors files not present in model repo, please check file path”, check the path. If it shows “Base model name does not match with the name given in config file”, correct the Base Model Name.

Deploy

Finish the form and select Deploy Endpoint. See Create an endpoint.

Check the link

The endpoint’s Overview gives the Model Name, Model Type, and Model Path of a repository model. In requests, the model name is /mnt/models for a model from a repository and peft-model for a LoRA adapter.

If the endpoint cannot find the model files, check three things. The files must be on the repository’s Model Files tab. Model Path must match the folder layout. The bucket name must be right. Then read the endpoint’s logs. See Troubleshooting Model Endpoints and Troubleshooting Model Repository.

You can delete a repository only after you delete its endpoint. See Delete a repository.