Use a private network and a security group

Give an endpoint a private address on your VPC, and choose which IPs can reach it.

Before you begin

• Permission to update model endpoints in the project. A Member needs a policy that grants update permission for the Inference service.
• A VPC IP that is available in the project, in the same region as the endpoint. See Reserve an IP. The VPC must use E2E-provided ranges. TIR refuses a VPC with a custom range.
• A security group, if you want one. See Create a security group.
• A machine on the same VPC to call the endpoint from, such as a MyAccount node attached to the VPC.

Attach a VPC IP

1. Open the endpoint and select the Settings tab.
2. In Attach VPC IP, choose a VPC IP under Select VPC IP. If there is none yet, select Reserve New VPC IP to reserve one first.
3. Select Attach.
4. If the endpoint still needs a security group, TIR asks you to attach one first, then attaches the VPC IP.

The VPC IP’s state updates while it connects. You can attach one VPC IP to an endpoint at a time.

See and change the security group

Security Group has the groups attached to the endpoint. To attach more, choose up to 5 at a time in Select Security Groups and select Attach. You can pick only groups that are new to the endpoint. Adding a group keeps the endpoint’s open ports exactly as they were. The group’s inbound rules only define which IPs can reach the ports the endpoint already has open. To change a group’s rules, see Manage security group rules.

Call the private endpoint

1. Open the endpoint’s security group and add an inbound rule for TCP port 8080. Set the source to the address of the machine that calls, not to the attached VPC IP, or choose Any Network. See Manage security group rules.

2. On the endpoint, select API Request and open the Private Endpoint tab. Its cURL sample calls http://<vpc-ip>:8080 with the same route and token as the public sample.

3. Sign in to a machine on the same VPC, for example a node in MyAccount, and run the sample there. For a vLLM endpoint it looks like this:

   ssh root@<node-address>
   curl "http://<vpc-ip>:8080/v1/completions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer <auth-token>" \
     -d '{"model": "<model-name>", "prompt": "Tell me about AI", "max_tokens": 100}'

   <node-address> is the public address of the node you sign in to. <vpc-ip> is the VPC IP attached to the endpoint. <auth-token> is your auth token, and <model-name> is the model name the public sample uses.

The reply is the same as from the public address, and the request travels over the VPC instead of the internet. For the public sample and its routes, see Call an endpoint with a token.

Detach the VPC IP

Select Detach VPC in the Attach VPC IP block and tick “Are you sure you want to proceed?” Deleting the endpoint detaches an attached VPC IP too.

Check the connection

The attached IP is under Attach VPC IP, and Sample API Request has a Private Endpoint sample. If the attach fails, TIR gives the reason. See Troubleshooting Model Endpoints and Attach a reserved IP in TIR.