Skip to content

Create Endpoints

POST
/serving/inference/
curl --request POST \
  --url 'https://api.e2enetworks.com/myaccount/api/v1/gpu/serving/inference/?project_id=<PROJECT_ID>&location=<LOCATION>&apikey=<API_KEY>' \
  --header 'Authorization: Bearer <API_TOKEN>' \
  --header 'Content-Type: application/json' \
  --data '{
  "name": "string",
  "custom_endpoint_details": {
    "service_port": false,
    "metric_port": false,
    "container": {
      "container_name": "string",
      "container_type": "string",
      "private_image_details": {},
      "advance_config": {
        "image_pull_policy": "string",
        "is_readiness_probe_enabled": false,
        "is_liveness_probe_enabled": false,
        "readiness_probe": {
          "protocol": "string",
          "initial_delay_seconds": 0,
          "success_threshold": 0,
          "failure_threshold": 0,
          "port": 0,
          "period_seconds": 0,
          "timeout_seconds": 0,
          "path": "string",
          "grpc_service": "string",
          "commands": "string"
        },
        "liveness_probe": {
          "protocol": "string",
          "initial_delay_seconds": 0,
          "success_threshold": 0,
          "failure_threshold": 0,
          "port": 0,
          "period_seconds": 0,
          "timeout_seconds": 0,
          "path": "string",
          "grpc_service": "string",
          "commands": "string"
        }
      }
    },
    "resource_details": {
      "disk_size": 0,
      "mount_path": "string",
      "env_variables": [
        "string"
      ]
    },
    "public_ip": "yes"
  },
  "model_id": "string",
  "replica": 0,
  "committed_replicas": 0,
  "path": "string",
  "framework": "string",
  "is_auto_scale_enabled": false,
  "auto_scale_policy": {
    "min_replicas": 0,
    "max_replicas": 0,
    "rules": [
      {
        "metric": "string",
        "condition_type": "string",
        "value": 0,
        "watch_period": 0,
        "granularity": 0,
        "window": 0
      }
    ],
    "stability_period": 0
  },
  "detailed_info": {
    "commands": "string",
    "args": "string",
    "hugging_face_id": "string",
    "tokenizer": "string",
    "server_version": "string",
    "world_size": 0,
    "error_log": false,
    "info_log": false,
    "warning_log": false,
    "log_verbose_level": 0,
    "model_serve_type": "string"
  },
  "model_load_integration_id": 0,
  "dataset_id": "string",
  "dataset_path": "string",
  "cluster_type": "string",
  "sku_id": 0,
  "sku_item_price_id": 0
}'

Create a new model inference endpoint. The following models and frameworks are supported — select the corresponding example from the request body:

# Model / Framework framework value Model source
1 vLLM vllm HuggingFace model ID
2 SGLang sglang HuggingFace model ID
3 Nvidia Dynamo dynamo HuggingFace model ID
4 PyTorch / TorchServe pytorch Fine-tuned TIR model (model_id)
5 Triton Inference Server triton Fine-tuned TIR model (model_id)
6 Custom Container custom Any Docker image
7 LLaMA 2 Chat 7B llma HuggingFace model ID
8 CodeLlama 7B Instruct codellama HuggingFace model ID
9 Mistral 7B Instruct mistral-7b-instruct HuggingFace model ID
10 Mixtral 8x7B Instruct mixtral-8x7b-instruct HuggingFace model ID
11 Gemma 2B IT gemma-2b-it HuggingFace model ID
12 LLaMA 3 8B Instruct llama-3-8b-instruct HuggingFace model ID
13 Phi-3 Mini 128K Instruct Phi-3-mini-128k-instruct HuggingFace model ID
14 LLaMA 3.1 8B Instruct llama-3_1-8b-instruct HuggingFace model ID
18 BGE Large EN v1.5 bge-large-en-v1_5 Built-in (via args)
19 BGE Reranker Large bge-reranker-large Built-in (via args)
20 Stable Diffusion XL stable_diffusion_xl E2E registry
21 YOLOv8 yolov8 E2E registry
22 Stable Video Diffusion stable-video-diffusion-img2vid-xt E2E registry
23 NVIDIA NV-Embed v1 nvidia-nv-embed-v1 E2E registry
24 TensorRT-LLM tensorrt Fine-tuned TIR model (model_id)
25 Nemotron 3 8B Chat nemotron-3-8b-chat-4k-rlhf E2E registry
26 Nemotron 3 Nano 30B A3B nemotron-3-nano-30b-a3b HuggingFace model ID
27 Nemotron Nano 12B v2 VL nemotron-nano-12b-v2-vl HuggingFace model ID
28 Nemotron Speech Streaming EN 0.6B nemotron-speech-streaming-en-0.6b E2E registry

To launch on TIR Cluster

Replace "cluster_type", "sku_id", and "sku_item_price_id" with the values for your location:

Location sku_id sku_item_price_id
Delhi 162 755
Chennai 40 1628 

"cluster_type": "tir-cluster",
"sku_id": 162,
"sku_item_price_id": 755

To launch on Private Cluster

Replace “cluster_type”,“sku_id” and “sku_item_price_id” with the following data.


"cluster_type": "private-cluster",
"custom_sku": {
  "cpu": 1,
  "gpu": 0,
  "memory": 1
},
"private_cloud_id": < Private Cluster Id>

Parameters

Section titled “Parameters”

Query Parameters

Section titled “Query Parameters”
project_id
required
integer

Project ID

active_iam
integer

Active IAM ID (To access contact person account) Find your Active IAM ID here

location
required
string
Allowed values: Delhi Chennai

Location

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json
object
name
required

Name of the deployment

string
custom_endpoint_details
required
object
service_port
boolean
metric_port
boolean
container
object
container_name
string
container_type
string
private_image_details
object
advance_config
object
image_pull_policy
string
is_readiness_probe_enabled
boolean
is_liveness_probe_enabled
boolean
readiness_probe
object
protocol
string
initial_delay_seconds
integer
success_threshold
integer
failure_threshold
integer
port
integer
period_seconds
integer
timeout_seconds
integer
path
string
grpc_service
string
commands
string
liveness_probe
object
protocol
string
initial_delay_seconds
integer
success_threshold
integer
failure_threshold
integer
port
integer
period_seconds
integer
timeout_seconds
integer
path
string
grpc_service
string
commands
string
resource_details
object
disk_size
integer
mount_path
string
env_variables
Array<string>
public_ip
string
Allowed values: yes no
model_id
string
nullable
replica
required
integer
committed_replicas
integer
path
string
framework
required
string
is_auto_scale_enabled
required
boolean
auto_scale_policy
required
object
min_replicas
integer
max_replicas
integer
rules
Array<object>
object
metric
string
condition_type
string
value
integer
watch_period
integer
granularity
integer
window
integer
stability_period
integer
detailed_info
required
object
commands
string
args
string
hugging_face_id
string
tokenizer
string
server_version
string
world_size
integer
error_log
boolean
info_log
boolean
warning_log
boolean
log_verbose_level
integer
model_serve_type
string
model_load_integration_id
required
integer
dataset_id
string
nullable
dataset_path
string
cluster_type
string
sku_id
required
integer
sku_item_price_id
required
integer

Responses

Section titled “Responses”

201

Section titled “201”

Model deployment created successfully.

Media typeapplication/json
object
code

HTTP response status code.

integer
data

Deployment creation response data.

object
id

Unique deployment ID.

integer
model

Model details (null if not available).

string
model_type

Type of model (null if not available).

string
sku_details
object
specs
object
sku_id

SKU identifier.

integer
name

SKU name.

string
series

SKU series.

string
cpu

Number of CPUs.

integer
gpu

Number of GPUs.

integer
memory

Memory in GB.

integer
is_free

Whether the SKU is free.

boolean
is_active

Whether the SKU is active.

boolean
gpu_switch_type

GPU switch type (if applicable).

string
local_storage_in_gb

Local storage in GB.

integer
plan
object
sku_item_price_id

SKU price ID.

integer
sku_type

SKU pricing type.

string
committed_days

Committed days for pricing.

integer
unit_price

Unit price in the given currency.

number format: float
currency

Currency of the price.

string
is_active

Whether the pricing plan is active.

boolean
description

Description of the pricing plan.

string
sku_item_price_id

SKU price ID.

integer
name

Name of the deployment.

string
storage_url

URL for storage.

string
status

Deployment status.

string
created_by
object
id

Creator’s user ID.

integer
name

Name of the creator.

string
email

Email of the creator.

string
username

Username of the creator.

string
created_at

Timestamp when the deployment was created.

string format: date-time
updated_at

Timestamp when the deployment was last updated.

string format: date-time
detailed_info
object
commands

List of commands related to the deployment.

Array<string>
args

List of arguments.

Array<string>
hugging_face_id

Hugging Face model ID (nullable).

string
tokenizer

Tokenizer (nullable).

string
server_version

Server version.

string
world_size

World size for distributed systems.

integer
error_log

Whether error logs are enabled.

boolean
info_log

Whether info logs are enabled.

boolean
warning_log

Whether warning logs are enabled.

boolean
log_verbose_level

Verbosity level for logs.

integer
model_serve_type

Model serve type (nullable).

string
framework

Framework used for the model.

string
custom_endpoint_details
object
public_ip

Whether the deployment has a public IP.

string
container
object
container_name

Name of the container.

string
container_type

Type of container (public or private).

string
advance_config
object
image_pull_policy

Policy for pulling container images.

string
is_readiness_probe_enabled

Whether readiness probe is enabled.

boolean
is_liveness_probe_enabled

Whether liveness probe is enabled.

boolean
readiness_probe
object
initial_delay_seconds

Initial delay before readiness probe starts.

integer
period_seconds

Period between readiness probe checks.

integer
timeout_seconds

Timeout for readiness probe.

integer
success_threshold

Number of successful attempts before marking as healthy.

integer
failure_threshold

Number of failed attempts before marking as unhealthy.

integer
protocol

Protocol for readiness probe.

string
port

Port for readiness probe.

string
path

Path for readiness check.

string
liveness_probe

Liveness probe configuration.

object
resource_details
object
disk_size

Disk size for the deployment.

integer
mount_path

Mount path for resources.

string
env_variables

List of environment variables for the deployment.

Array<string>
replica

Number of replicas for the deployment.

integer
auto_scale_policy
object
min_replicas

Minimum replicas for auto-scaling.

integer
max_replicas

Maximum replicas for auto-scaling.

integer
rules
Array<object>
object
metric

Metric to monitor for auto-scaling.

string
condition_type

Type of condition for scaling (e.g., limit).

string
value

Value for the condition.

integer
watch_period

Period to monitor the metric.

integer
granularity

Granularity of the metric.

integer
window

Window size for scaling.

integer
custom_metric_name

Custom metric name (nullable).

string
stability_period

Stability period before scaling.

string
is_auto_scale_enabled

Whether auto-scaling is enabled.

boolean
desired_replica

Desired number of replicas.

integer
model_load_integration_id

Integration ID for model load.

integer
dataset

Dataset associated with the deployment (nullable).

string
dataset_storage_url

URL for dataset storage (nullable).

string
committed_replicas

Number of committed replicas.

integer
private_cloud_id

Private cloud ID (nullable).

string
custom_sku

Custom SKU details (nullable).

string
errors
object
key
additional properties
any
message

Success or error message.

string

Example 

{
  "code": 201,
  "data": {
    "id": 3036,
    "model": null,
    "model_type": null,
    "sku_details": {
      "specs": {
        "sku_id": 162,
        "name": "GDC.1xH200-30.375GB_SXM",
        "series": "GPU",
        "cpu": 30,
        "gpu": 1,
        "memory": 375,
        "is_free": false,
        "is_active": true,
        "gpu_switch_type": null,
        "local_storage_in_gb": 0
      },
      "plan": {
        "sku_item_price_id": 755,
        "sku_type": "hourly",
        "committed_days": 0,
        "unit_price": 470.4,
        "currency": "INR",
        "is_active": true,
        "description": "Hourly Compute Instance: inference_service"
      }
    },
    "sku_item_price_id": 755,
    "name": "tir-endpoint-1115165asd83333",
    "storage_url": null,
    "status": "waiting",
    "created_by": {
      "id": 3573,
      "name": "Nipun",
      "email": "nipun.arora@e2enetworks.com",
      "username": "nipun.arora@e2enetworks.com"
    },
    "created_at": "2024-11-15T11:32:35.527296Z",
    "updated_at": "2024-11-15T11:32:35.527371Z",
    "detailed_info": {
      "commands": [],
      "args": [],
      "hugging_face_id": "",
      "tokenizer": "",
      "server_version": "",
      "world_size": 1,
      "error_log": true,
      "info_log": true,
      "warning_log": true,
      "log_verbose_level": 1,
      "model_serve_type": ""
    },
    "framework": "stable_diffusion_xl",
    "custom_endpoint_details": {
      "public_ip": "no",
      "container": {
        "container_name": "registry.e2enetworks.net/aimle2e/stable-diffusion-xl-base-1.0:hf",
        "container_type": "public",
        "advance_config": {
          "image_pull_policy": "Always",
          "is_readiness_probe_enabled": false,
          "is_liveness_probe_enabled": false,
          "readiness_probe": {
            "initial_delay_seconds": 10,
            "period_seconds": 10,
            "timeout_seconds": 10,
            "success_threshold": 1,
            "failure_threshold": 3,
            "protocol": "http",
            "port": "8080",
            "path": "/v2/health/ready"
          }
        }
      },
      "resource_details": {
        "disk_size": 100,
        "mount_path": "",
        "env_variables": []
      }
    },
    "replica": 1,
    "auto_scale_policy": {
      "min_replicas": 1,
      "max_replicas": 1,
      "rules": [
        {
          "metric": "",
          "condition_type": "limit",
          "value": 100,
          "watch_period": 60,
          "granularity": 1,
          "window": 1,
          "custom_metric_name": ""
        }
      ],
      "stability_period": "300"
    },
    "is_auto_scale_enabled": false,
    "desired_replica": 1,
    "model_load_integration_id": 428,
    "dataset": null,
    "dataset_storage_url": null,
    "committed_replicas": 0,
    "private_cloud_id": null,
    "custom_sku": null
  },
  "message": "Created Successfully"
}

© E2E Networks Limited · CIN L72900DL2009PLC341980