Set workers and autoscaling

Set how many workers serve your endpoint and how that number changes with traffic. You can keep a fixed number, let the endpoint grow and shrink with load, or scale it to zero when it is idle. You set the counts when you create the endpoint, in Serverless Configuration on the form, and change them later on the endpoint’s Serverless Configuration tab.

Before you begin

• Permission to update model endpoints in the project, to change an existing endpoint. A Member needs a policy that grants update permission for the Inference service.
• A running endpoint. Start a Stopped endpoint before you change its scaling; see Stop, start, and restart an endpoint.

Choose the worker counts

Two numbers decide how many workers run. On the create form they are Active Worker, the workers that are always ready, and Max Workers, the most the endpoint can grow to. On the Serverless Configuration tab the same two are Minimum Workers and Maximum Workers.

You are charged continuously for the active workers, so set the minimum to the load you always expect. Active Worker starts at 1, and Max Workers goes up to 64, or 512 on a private cluster. The endpoint scales in one of three ways, depending on how the two numbers compare.

Keep a fixed number of workers

Set the minimum equal to the maximum, for example both to 2. Autoscaling is off, and the endpoint keeps that many workers. Choose this for steady traffic, or for a first test with both at 1.

On the Serverless Configuration tab, select Update, then confirm with Disable.

Let the endpoint grow with traffic

Set the minimum below the maximum, for example 1 and 4. Autoscaling is on, and you choose a Metric Name, the load signal that adds and removes workers:

Metric Name	It counts	Choose it for
Concurrent Request Count	The requests in the queue and those in progress	Language models, where each request holds the GPU for the whole answer. Start here for an LLM.
Request Rate per second	The requests arriving each second	Short requests that arrive at a high rate.
Custom	A metric the framework or your container reports	A model whose own state, such as its queue or cache use, says more about load than the request count.

For Custom, pick the metric under Custom Metric Name, or type its name for a Custom Container. The metrics on offer depend on the framework version:

• vLLM offers Running Requests, Waiting Requests, and GPU Cache Usage, and versions v0.9.0 and v0.9.1 also offer Swapped Requests and CPU Cache Usage.
• SGLang offers Queued Requests, Running Requests, Generation Throughput, and Used Tokens.
• NVIDIA Dynamo offers In-flight Requests, Active KV Cache Blocks, Total KV Cache Blocks, Request Bytes Received, and Request Duration.
• PyTorch offers Queue Time, Inference Requests, Queue Latency, CPU Utilization, and Memory Used.
• NVIDIA Triton offers Successful Inference Requests, Failed Inference Requests, Inference Count, Inference Execution Count, and Pending Inference Requests.

Then set three values:

• Target Value is the load one worker carries before the endpoint adds another, a whole number from 1 to 100000. With Concurrent Request Count and a target of 10, a new worker starts once a worker is handling about 10 requests at once. With Custom and Running Requests on vLLM, a target of 32 adds a worker when a worker handles about 32 requests at once.
• Idle Timeout (in seconds), from 1 to 1800, is how long an idle worker stays running for new requests before it is removed. You are charged for that time.
• Initial Cooldown Period (in seconds), from 30 to 86400, is how long the endpoint waits after a trigger before it scales down to 0 workers. It stops the worker count from changing too often.

Scale to zero when idle

To pay nothing while an hourly endpoint sits idle, set Active Worker, or Minimum Workers on the tab, to 0, and choose Concurrent Request Count or Request Rate per second as the metric. The endpoint scales down to 0 when requests stop arriving and shows Idle. The next request starts a worker and waits while the model loads, and traffic beyond that grows the endpoint up to Max Workers. Choose this for development and for endpoints that are called now and then. Keep one worker running when every request must reach a warm worker.

Use the presets and save on the tab

The Serverless Configuration tab has three choices at the top. Scale to zero when idle sets scale to zero. Keep one warm keeps one worker always on, which suits production because every caller reaches a warm worker. The third shows the current configuration and discards unsaved changes.

Change a setting, then select Update to save it, or Reset to return to the saved values. When you raise the minimum, TIR first checks free capacity and credit for the extra workers.

Check the workers

In the endpoint list, Status counts the running workers against the total, for example 1 / 2. Total GPUs is the GPUs on each worker times the number of workers set. Scaling down takes a few seconds to a few minutes, depending on the model size, the requests in progress, and how quickly the workers stop. To follow the load that drives scaling, see Monitor an endpoint.

Committed, Multi Node, and custom-metric endpoints have their own rules for the minimum; the Model Endpoints FAQ lists them. If a change is refused, see Troubleshooting Model Endpoints.