Call an endpoint with a token

Send a request to a running endpoint from your own machine or application. An external endpoint has one HTTPS address, the root endpoint, and you call it with an auth token in the Authorization header.

Before you begin

• An endpoint in the Running state. An Idle endpoint also answers: the first request starts a worker. See Set workers and autoscaling.
• Permission to read model endpoints in the project. A Member needs a policy that grants read permission for the Inference service.
• An auth token from Personal Access Tokens, or permission to create one. See Create a Personal Access Token. To call the endpoint, a valid auth token from the project is enough, whatever your portal permissions.

Get the address and a token

1. In Model Endpoints, open the endpoint. Everything you need to connect is at the top of its Overview: the address, Auth token, and sample requests. API Request in the list gives the same samples.
2. Copy the endpoint address. It is the Root Endpoint. The address differs between Delhi and Chennai, and it ends with the project and endpoint numbers.
3. In Auth token, choose one of your unexpired tokens, and the sample requests fill it in for you.
4. To make a token, select Generate token. If you have no token yet, TIR creates one at once under a generated name. Otherwise, it asks for a name of up to 50 characters: letters, digits, hyphens, and underscores. To generate a token, you need the create permission on tokens. Manage your tokens under Personal Access Tokens in the side menu.

Send a request

The Root Endpoint is the prefix of every route that the server offers, so add the route to it. For a vLLM or SGLang endpoint, the cURL sample is a chat request:

curl -X POST "<root-endpoint>/v1/chat/completions" \
  -H "Authorization: Bearer <auth-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{"role": "user", "content": "Who won the world series in 2020?"}]
  }'

• <root-endpoint> is the address you copied.
• <auth-token> is the token you chose.
• <model-name> is the Hugging Face ID of the model. For a model from a repository it is /mnt/models, and for a LoRA adapter it is peft-model.

The chat route works for instruct or chat models. A base model needs a chat template in the request. For a plain prompt, vLLM, SGLang, and Finetuned endpoints also accept a request to <root-endpoint>/v1/completions with a prompt.

To stream the answer or use the OpenAI Python package, see Use the OpenAI client and stream responses.

Other kinds of endpoint

• A Custom Container endpoint has a cURL sample with your own route and body to fill in. Use the routes your container serves.
• Which samples you get depends on the framework: cURL, OpenAI, gRPC, Python, and Async when async is on. Serving Protocol on Sample API Request names the interface the framework follows.
• An internal endpoint, a Custom Container with Service Type Internal, needs no token. Call it at http://is-<endpoint-id>.p-<project-id>:<port> from inside the project’s network.
• With a VPC IP attached, Sample API Request also has a Private Endpoint sample at the endpoint’s private address. See Use a private network and a security group.

Connect over WebSocket

A model whose server offers a WebSocket route, such as a streaming speech-to-text model, accepts a WebSocket connection, which sends and receives data over one open line. Build the address from the root endpoint: change https to wss and add the route your server uses, such as /ws. Send the token in an Authorization: Bearer header when you connect. The endpoint accepts the token only in that header.

In Python, with the websocket-client package (pip install websocket-client):

import websocket

ROOT_ENDPOINT = "<root-endpoint>"  # starts with https://
AUTH_TOKEN = "<auth-token>"
WS_ROUTE = "/ws"  # the route your model's server uses

ws_url = ROOT_ENDPOINT.replace("https://", "wss://", 1).rstrip("/") + WS_ROUTE

ws = websocket.WebSocket()
ws.connect(ws_url, header={"Authorization": f"Bearer {AUTH_TOKEN}"})

ws.send("<your-data>")  # text, or bytes such as an audio chunk
print(ws.recv())  # the server's reply
ws.close()

In Node.js, with the ws package (npm install ws):

const WebSocket = require("ws");

const ROOT_ENDPOINT = "<root-endpoint>"; // starts with https://
const AUTH_TOKEN = "<auth-token>";
const WS_ROUTE = "/ws"; // the route your model's server uses

const wsUrl = ROOT_ENDPOINT.replace("https://", "wss://").replace(/\/$/, "") + WS_ROUTE;

const ws = new WebSocket(wsUrl, {
  headers: { Authorization: `Bearer ${AUTH_TOKEN}` },
});

ws.on("open", () => ws.send("<your-data>"));
ws.on("message", (data) => {
  console.log(data.toString());
  ws.close();
});

<root-endpoint> and <auth-token> are the address and token from the first section, and <your-data> is what your model expects, such as a chunk of audio. What the server sends back, and in what order, depends on the model.

Check the request

A reply from the model means the call worked. If the request is refused or hangs, check that the endpoint is Running and that the token is still valid. See Troubleshooting Model Endpoints.