Deploy a model from Hugging Face

Deploy a foundation model straight from Hugging Face, with no files to upload first. The endpoint downloads the model from Hugging Face when its workers start.

Before you begin

• Permission to create model endpoints in the project. A Member needs a policy that grants create permission for the Inference service.
• A Hugging Face access token with Read scope, saved as a Hugging Face integration. See Create a Hugging Face integration. For most frameworks, the form asks for this integration when you download from Hugging Face.
• The model ID, in the form <organization>/<model-name>, like the example the field shows as a placeholder.
• For a gated model, the licence accepted on the model’s Hugging Face page with the same account that owns the token. The download fails until you accept it.

Choose the Hugging Face download

1. On Choose Framework, select a framework that loads a model, such as vLLM, and open its create form. See Choose a framework.
2. In Download Type, choose Download from Huggingface. It is the default for vLLM and its family. Use it for a foundation model as published on Hugging Face.
3. In HuggingFace Model, type the model ID as organization/model-name. TIR checks the ID.
4. In HF Token Integration, choose your integration. If you have none yet, make one with Create Integration.
5. Wait while TIR checks that the token can read the model. If it cannot, TIR gives the reason. If the model’s owner must grant access, request it first from the link on the form.
6. Finish the rest of the form, starting with Model Cache Storage, and select Deploy Endpoint. See Create an endpoint.

Check the download

The endpoint starts in Deploying and becomes Running when the model is loaded. A large model can take 10 to 15 minutes or more. If it takes much longer, open Deployment Events and Worker Logs. See View logs and events.

A download usually fails for one of these reasons:

• A wrong or empty model ID.
• A token without Read scope.
• A gated model, before you accept its licence with the account that owns the token.
• A revoked token.

The portal can also show “Huggingface Token does not have access to gated <model> model” or “Failed to validate huggingface token”. See Troubleshooting Model Endpoints.

When the endpoint is Running, call it with a token. In requests, the model name is the Hugging Face ID.