Set up async invocation

Use async invocation for long-running or batch requests. TIR queues the call and processes it in the background. The call answers at once with a request ID and a response location. TIR stores the answer in a dataset you choose, so your application can carry on while the result is prepared.

Before you begin

• An existing endpoint on any plan except Multi Node. You set up async on the endpoint’s Async Invocations tab once the endpoint exists.
• Permission to update model endpoints in the project. A Member needs a policy that grants update permission for the Inference service.
• A dataset in the same region to hold the results. You can also create one in the setup panel. See TIR Object Storage.
• The routes you want to queue. They must accept only the POST and OPTIONS methods.

Set up the queues

1. Open the endpoint and select the Async Invocations tab.
2. Select Setup Queues.
3. Under Source, select Add Queue for each route. Each queue has a fixed name, QUEUE_1, QUEUE_2, and so on, and an Application Route. Type the route without a leading slash and without spaces, for example api/v1/chat/completion. Each queue needs its own route.
4. Under Target, leave Destination Type on Dataset and choose one of the project’s ready Object Storage datasets under Destination, or create a new one from the same list. A new one is an Object Storage bucket that TIR names async-invocations-<random>.
5. Choose Concurrency, the number of requests processed in parallel: 1, 2, 4, 8, 16, 32, or 64. It starts at 4.
6. Select Submit.

A small CPU machine handles the queue. You pay for it only while the queue has requests, and it scales down to zero when the queue is empty. The endpoint’s own workers are billed as usual.

Send an async request

With async on, each queue has its own samples under API Request on its row. The cURL call goes to the root endpoint, the route, and async/ after it, with your token in the Authorization: Bearer header. For OpenAI-style routes, the Python sample uses <root-endpoint>/async/v1/ as the base URL. See Call an endpoint with a token.

Check a request

Select Request Status on a queue row. It lists requests with Queue Name, Request ID, Status, and Updated On. Search by request ID with Search request by id, choose an interval of 1 hour, 24 hours, 7 days, or 1 month, and refresh the list. A request’s Status is Received, Queued, Processing, Completed, Failed, or Canceled. For a Failed request, you can copy the error message.

The saved answer is a JSON file in the dataset. Use the response location from the call to find it. The tab also lists the steps to read a response from an application with the TIR SDK.

Change or stop async

• Modify Queues changes the routes and destination.
• Disable Async Invocations turns async off after you confirm.

Your application now gets an answer to each queued call at once and picks up the result from the dataset when it is ready. If a request stays Queued or never completes, see Troubleshooting Model Endpoints.