How a load balancer works

A TIR Load Balancer takes the traffic sent to one address and shares it across backends you choose, after checking that they are healthy. Three things make it up: the Load Balancer itself, one or more backend groups, and the rules that connect them.

The parts and how a request flows

A Load Balancer is the address that receives requests. It gets an E2E address ending in .e2enetworks.net when you create it, and the address stays the same for the life of the Load Balancer. You can also serve your own domain; see Use a custom domain and certificate.

A backend group is a named pool of backends, with its own port, protocol, health check, balancing method, request timeout, and retry count. Each backend group belongs to a single Load Balancer.

A rule decides which backend group serves a request. A request flows like this:

1. The request arrives at the Load Balancer.
2. The Load Balancer finds the rule that matches it. A rule lists conditions on the path, the method, a header, or a query parameter, and a request must satisfy every condition. A rule with zero conditions matches every request. When more than one rule matches, the more specific match wins. Among rules that are equally specific, the lower Priority number is tried first.
3. The rule forwards the request to a backend group. A rule can split traffic between groups by weight, for example 90 and 10, which sends roughly 90% of requests to the first group and 10% to the second. This is useful for trying a new version on a small share of users.
4. The backend group’s balancing method picks one healthy backend, and the backend answers.

A Load Balancer that is Running answers every request with a 503 error until it has a rule that forwards to a backend group. To add the parts, see Create a backend group and Add a rule.

Type

You can create a web (HTTP/HTTPS) Load Balancer, reachable from the public internet. It routes by path, method, header, or query parameter and handles HTTPS for you.

Plans and capacity

You choose a plan under Capacity when you create the Load Balancer. The plan sets the request rate limit, the number of simultaneous connections, and the size of the Load Balancer. Each plan has a price per hour, and you choose from the plans of the project’s region.

High availability is on by default, and you can use it with any plan. Redundancy on the Overview tab gives the current setting. To change the plan or High availability later, see Change the plan.

What a backend can be

Each backend group holds one kind of backend:

• Virtual Machines: the project’s VMs that are ready. Each has its own port and a weight.
• Inference: the project’s running inference endpoints, each with a port.
• Auto Scaling Group: the group itself, so traffic keeps following the group as it scales up and down. An Auto Scaling Group can be a backend of a web Load Balancer only. See Connect an Auto Scaling Group.

You choose backends from these lists rather than typing in an IP address. A backend and port can appear once in a group, and the same backend and port can also be in other groups.

How requests are shared

Distribute requests by chooses the balancing method of a backend group:

• Round Robin distributes requests evenly across all backends. It is the first choice for a new group.
• Least Connection sends each request to the backend with the fewest open connections.
• Source IP Hash sends the same client IP address to the same backend.

Each backend also has a weight that starts at 1.

Health checks

The Health check switch is on by default. It probes the group’s backends and stops sending traffic to unhealthy ones. You choose how a backend is probed:

• TCP connects to the backend port.
• HTTP requests a path and checks the status code. The path starts as / and the success code as 200.
• gRPC calls the standard gRPC health service, for a gRPC group.

A probe runs every Interval (s) and waits up to Timeout (s). The interval starts at 10 seconds and the timeout at 3 seconds; keep the timeout below the interval. A backend leaves rotation after Unhealthy threshold failed probes in a row and gets traffic again after Healthy threshold successful probes in a row. Unhealthy threshold starts at 3 and Healthy threshold at 2.

Next, see Get started with TIR Load Balancer to build one, or Billing for TIR Load Balancer to see how it is charged.