Skip to main content

Model Repositories

TIR Model Repositories are designed to store model weights and configuration files. These repositories can be backed by either E2E Object Storage (EOS) or PVC storage within a Kubernetes environment.

Key Concepts​

  • Flexible Model Definition: A model in TIR is simply a directory hosted on EOS or a disk. There is no rigid structure, format, or framework required. You can organize your model by creating subfolders (e.g., v1, v2, etc.) to track different versions.
  • Collaboration: Once a model is defined within a project, all team members gain access to it, enabling reuse and collaboration. It is recommended to define a TIR model to store, share, and utilize your model weights efficiently.

Uploading Weights to TIR Models​

When you create a new model in TIR, an EOS bucket is automatically provisioned to store your model weights and configuration files (e.g., TorchServe configuration). Connection details to access the EOS bucket will be provided.

If you haven't used EOS (E2E Object Storage) before, note that it offers an S3-compatible API for uploading and downloading content. You can use any S3-compatible CLI, such as Minio’s mc or s3cmd. We recommend using the Minio CLI (mc). The TIR Model Setup section provides ready-to-use commands to configure the client and upload contents.

Example Setup Command​

mc config host add <tir-model-name> https://objectstore.e2enetworks.net <access-key> <secret-key>

Once the alias (connection profile) is set up, you can upload content using commands like:

Click to expand upload examples
# Upload contents of saved-model directory to llma-7b-23233 (EOS Bucket)
mc cp -r /user/jovyan/saved-model/* <tir-model-name>/llma-7b-23233

# Upload contents of saved-model to llma-7b-23233(bucket)/v1(folder)
mc cp -r /user/jovyan/saved-model/* <tir-model-name>/llma-7b-23233/v1

For Hugging Face models, upload the entire snapshot folder (located under .cache/huggingface/hub/<model-name>/) to the model bucket. This ensures that the weights and configurations can be easily downloaded and used by TIR notebooks or inference service pods with a simple call such as AutoModelForCausalLM.from_pretrained().

Downloading Weights from TIR Models​

Model weights are required on the device whether you are fine-tuning the model or serving inference requests through an API. You can download model files manually using commands like:

Click to expand download examples
# Download contents of the EOS bucket to a local directory
mc cp -r <tir-model-name>/llma-7b-23233 /user/jovyan/download-model/*

# Download contents from a specific version folder (e.g., 'v1')
mc cp -r <tir-model-name>/llma-7b-23233/v1 /user/jovyan/download-model/*

Typical Use Cases​

  • Fine-tuning on a local device: Download model files using the mc command.
  • Running in TIR Notebooks: Download model files directly to TIR notebooks for fine-tuning or inference testing.
  • Inference Service (Model Endpoints): Once a model is attached to an endpoint, the files are automatically downloaded to the container.

Creating Model Repositories in the TIR Dashboard​

To create a Model Repository:

  1. Navigate to Inference → Model Repository.
  2. Click CREATE REPOSITORY.
  3. Choose the model type (e.g., PyTorch, Triton, or Custom) and storage type.

Storage Types​

1. New EOS Bucket​

Selecting this option automatically provisions a new EOS bucket for model weights. After creation, configure uploads using one of these methods:

  • SDK – Install the SDK and run commands in Python or Jupyter Notebooks.
  • TIR Notebook – Directly integrate within TIR Notebooks.
  • CLI – Use Minio CLI or s3cmd for manual uploads.

2. Existing EOS Bucket​

Select from existing EOS buckets and use the same upload configuration options (SDK, TIR Notebook, CLI).

3. External EOS Bucket​

You can use an external EOS bucket by providing the bucket name, access key, and secret key. Ensure the credentials match the target bucket.

Managing Model Repositories​

After creation, models appear in the repository list. The Overview tab shows details and bucket information. The File Browser tab allows uploading and viewing files stored in the linked EOS bucket using Minio.

Deleting Model Repositories​

To delete a repository:

  1. Select the model from the list.
  2. Click the Delete icon.
  3. Confirm deletion when prompted.

Your model repositories are now configured for efficient versioning, collaboration, and deployment integration with TIR endpoints.