View logs and events

Read what a worker reports, to see why an endpoint is slow to start, keeps restarting, or fails requests. The Events & Logs tab has Deployment Events, Worker Logs, and Request Logs.

Before you begin

• Permission to read model endpoints in the project. A Member needs a policy that grants read permission for the Inference service.
• An endpoint in any state except Stopped or Terminating.

Open the tab

Open the endpoint and select Events & Logs. To see only the workers in one state, such as Ready or Error, use All statuses.

Read a worker’s events

1. Select Deployment Events.
2. Under Select Worker, choose a worker.
3. Read the table. Message and Reason say what happened. Type is Normal, Warning, Error, or Info, and Age says how long ago.

Read a worker’s logs

1. Select Worker Logs and choose a worker under Select Worker.
2. Refresh the logs, or set Auto Refresh to Disable, 10 Seconds, 30 Seconds, or 1 Minute.
3. Set Filter By to Last N Lines, Last N Seconds, or None, and give Value of N. It starts at 1000 lines.
4. Select Download logs to save every line as <worker name>-logs.json.

You can also reach a worker’s events and logs from the Workers tab, with Events and Logs on its row.

Read the request logs

Select Request Logs. There is one log file per hour; choose a file to read it. Access logs can take up to an hour to appear here.

Check what you found

A worker state such as ImagePullBackOff or ErrImagePull means the worker failed to fetch the image. CrashLoopBackOff means the container keeps starting and stopping. For the other messages, see Troubleshooting Model Endpoints. To follow an endpoint’s load, see Monitor an endpoint.