--- title: "Set workers and autoscaling" description: "Set how many workers a Model Endpoint runs, let it grow with traffic, keep a fixed count, or scale to zero when idle, on the Serverless Configuration tab." content_type: "how-to" product: "model-endpoints" platform: "TIR" url: "/tir/model-endpoints/how-to/set-workers-and-autoscaling/" topic: ["scale"] --- # Set workers and autoscaling Set how many workers serve your endpoint and how that number changes with traffic. You can keep a fixed number, let the endpoint grow and shrink with load, or scale it to zero when it is idle. You set the counts when you create the endpoint, in **Serverless Configuration** on the form, and change them later on the endpoint's **Serverless Configuration** tab. ## Before you begin - Permission to update model endpoints in the project, to change an existing endpoint. A Member needs a policy that grants update permission for the Inference service. - A running endpoint. Start a **Stopped** endpoint before you change its scaling; see [Stop, start, and restart an endpoint](/tir/model-endpoints/how-to/stop-start-and-restart-an-endpoint/). ## Choose the worker counts Two numbers decide how many workers run. On the create form they are **Active Worker**, the workers that are always ready, and **Max Workers**, the most the endpoint can grow to. On the **Serverless Configuration** tab the same two are **Minimum Workers** and **Maximum Workers**. You are charged continuously for the active workers, so set the minimum to the load you always expect. **Active Worker** starts at 1, and **Max Workers** goes up to 64, or 512 on a private cluster. The endpoint scales in one of three ways, depending on how the two numbers compare. ## Keep a fixed number of workers Set the minimum equal to the maximum, for example both to 2. Autoscaling is off, and the endpoint keeps that many workers. Choose this for steady traffic, or for a first test with both at 1. On the **Serverless Configuration** tab, select **Update**, then confirm with **Disable**. ## Let the endpoint grow with traffic Set the minimum below the maximum, for example 1 and 4. Autoscaling is on, and you choose a **Metric Name**, the load signal that adds and removes workers: | Metric Name | It counts | Choose it for | | ---------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | **Concurrent Request Count** | The requests in the queue and those in progress | Language models, where each request holds the GPU for the whole answer. Start here for an LLM. | | **Request Rate per second** | The requests arriving each second | Short requests that arrive at a high rate. | | **Custom** | A metric the framework or your container reports | A model whose own state, such as its queue or cache use, says more about load than the request count. | For **Custom**, pick the metric under **Custom Metric Name**, or type its name for a Custom Container. The metrics on offer depend on the framework version: - vLLM offers **Running Requests**, **Waiting Requests**, and **GPU Cache Usage**, and versions v0.9.0 and v0.9.1 also offer **Swapped Requests** and **CPU Cache Usage**. - SGLang offers **Queued Requests**, **Running Requests**, **Generation Throughput**, and **Used Tokens**. - NVIDIA Dynamo offers **In-flight Requests**, **Active KV Cache Blocks**, **Total KV Cache Blocks**, **Request Bytes Received**, and **Request Duration**. - PyTorch offers **Queue Time**, **Inference Requests**, **Queue Latency**, **CPU Utilization**, and **Memory Used**. - NVIDIA Triton offers **Successful Inference Requests**, **Failed Inference Requests**, **Inference Count**, **Inference Execution Count**, and **Pending Inference Requests**. Then set three values: - **Target Value** is the load one worker carries before the endpoint adds another, a whole number from 1 to 100000. With **Concurrent Request Count** and a target of 10, a new worker starts once a worker is handling about 10 requests at once. With **Custom** and **Running Requests** on vLLM, a target of 32 adds a worker when a worker handles about 32 requests at once. - **Idle Timeout (in seconds)**, from 1 to 1800, is how long an idle worker stays running for new requests before it is removed. You are charged for that time. - **Initial Cooldown Period (in seconds)**, from 30 to 86400, is how long the endpoint waits after a trigger before it scales down to 0 workers. It stops the worker count from changing too often. ## Scale to zero when idle To pay nothing while an hourly endpoint sits idle, set **Active Worker**, or **Minimum Workers** on the tab, to 0, and choose **Concurrent Request Count** or **Request Rate per second** as the metric. The endpoint scales down to 0 when requests stop arriving and shows **Idle**. The next request starts a worker and waits while the model loads, and traffic beyond that grows the endpoint up to **Max Workers**. Choose this for development and for endpoints that are called now and then. Keep one worker running when every request must reach a warm worker. ## Use the presets and save on the tab The **Serverless Configuration** tab has three choices at the top. **Scale to zero when idle** sets scale to zero. **Keep one warm** keeps one worker always on, which suits production because every caller reaches a warm worker. The third shows the current configuration and discards unsaved changes. Change a setting, then select **Update** to save it, or **Reset** to return to the saved values. When you raise the minimum, TIR first checks free capacity and credit for the extra workers. ## Check the workers In the endpoint list, **Status** counts the running workers against the total, for example 1 / 2. **Total GPUs** is the GPUs on each worker times the number of workers set. Scaling down takes a few seconds to a few minutes, depending on the model size, the requests in progress, and how quickly the workers stop. To follow the load that drives scaling, see [Monitor an endpoint](/tir/model-endpoints/how-to/monitor-an-endpoint/). Committed, Multi Node, and custom-metric endpoints have their own rules for the minimum; the [Model Endpoints FAQ](/tir/model-endpoints/faq/#workers-and-scaling) lists them. If a change is refused, see [Troubleshooting Model Endpoints](/tir/model-endpoints/troubleshooting/model-endpoints/).