Skip to main content

Deploy inference endpoint for MPT-7B-CHAT

In this tutorial, we will create a model endpoint for the MPT-7B-CHAT model.

The tutorial focuses on:

  • Creating a model endpoint using a pre-built container.
  • Creating a model endpoint with custom model weights.
  • Supported parameters for text generation.

Creating a ready-to-use model endpoint​

Steps to create the endpoint​

  1. Go to the Dashboard.
  2. Navigate to Model Endpoints.
  3. Click Create New Endpoint.
  4. Choose the MPT-7B-CHAT option.
  5. Select a suitable GPU plan.
  6. Complete endpoint creation.
  7. After setup, go to the Sample API Request section to test your endpoint using curl.

Creating model endpoint with custom model weights​

To create an inference endpoint with custom model weights:

  1. Download the MPT-7B-CHAT model from Hugging Face.
  2. Upload the model to the Model Bucket (EOS).
  3. Create an inference endpoint to serve API requests.

Step 1: Define a model in the dashboard​

  1. Go to the AI Platform.
  2. Choose a project.
  3. Navigate to the Model section.
  4. Click Create Model.
  5. Enter a model name (e.g., mpt-7b-chat-1).
  6. Set Model Type as Custom.
  7. Click CREATE.
  8. Note the EOS bucket details created for your model.
  9. EOS provides an S3-compatible API. We will use MinIO CLI.
  10. Copy the Setup Host command from the Setup MinIO CLI tab.

Step 2: Start a new Instance​

  1. In the dashboard, go to Instance.
  2. Launch a Instance with the mpt-7b-chat image and a GPU plan (e.g., A10080).
  3. Open the Jupyter Labs environment.
  4. Create a new terminal and run the MinIO setup command copied earlier.
  5. Verify that the mc CLI is configured correctly.

Step 3: Download the MPT-7B-CHAT model​

Click to expand code
import transformers
import torch
from transformers import AutoTokenizer

model = transformers.AutoModelForCausalLM.from_pretrained(
    'mosaicml/mpt-7b-chat',
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained('EleutherAI/gpt-neox-20b')

pipe = transformers.pipeline(
    "text-generation",
    model='mosaicml/mpt-7b-chat',
    tokenizer=tokenizer,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
pip install transformers accelerate xformers einopsx

Run a sample inference:

Click to expand code
output = pipe(
    'Here is a recipe for vegan banana bread:\n',
    do_sample=True,
    max_new_tokens=100,
    use_cache=True
)
print(output)

Step 4: Upload the model to EOS​

After confirming the model works, upload it to EOS.

Click to expand code
# Navigate to the Hugging Face model directory
cd $HOME/.cache/huggingface/hub/models--mpt-7b-chat-1/snapshots

# Upload all contents to EOS
mc cp -r * mpt-7b-chat-1/mpt-b-chat-1-b9402a
ls $HOME/.cache/huggingface/hub

Step 5: Create the model endpoint​

Once uploaded, return to the dashboard and create a new endpoint. Select the uploaded model under Model Details.

Refer to the earlier section on Creating a ready-to-use model endpoint.

Inference​

After launching the endpoint, wait a few minutes for the model to load. Then send an API request for prediction.

Inference request example​

{
  "instances": [
    { "text": "here's the Recipe of pancakes" }
  ],
  "params": { "max_length": 100 }
}

Inference response example​

{
  "predictions": [
    [
      "here's the Recipe of pancakes with strawberries and cream..."
    ]
  ]
}

You can also pass multiple prompts:

{
  "instances": [
    { "text": "here's the tea recipe" },
    { "text": "here's the pancakes recipe" }
  ],
  "params": { "max_length": 100 }
}

Supported parameters for output control​

Length control​

  • max_length: Maximum token length.
  • max_new_tokens: Max tokens to generate (ignores input length).
  • min_length / min_new_tokens: Minimum tokens.
  • early_stopping: Boolean or string controlling beam-search stopping behavior.
  • max_time: Maximum compute time (seconds).

Generation strategy​

  • do_sample: Enable sampling.
  • num_beams / num_beam_groups: Beam search and diversity settings.
  • penalty_alpha: Balances confidence vs. diversity.
  • use_cache: Uses cached key/values to speed decoding.

Logit manipulation​

  • temperature, top_k, top_p, typical_p: Control randomness.
  • epsilon_cutoff, eta_cutoff: Token filtering thresholds.
  • diversity_penalty, repetition_penalty: Penalize repetition.
  • length_penalty: Length scaling for beams.
  • no_repeat_ngram_size: Prevent repeated n-grams.
  • bad_words_ids / force_words_ids: Restrict token generation.

Output configuration​

  • num_return_sequences: Number of outputs.
  • output_attentions, output_hidden_states, output_scores: Return model internals.
  • return_dict_in_generate: Return structured output.

Special tokens​

  • pad_token_id, bos_token_id, eos_token_id: Token control options.

Encoder-decoder specific​

  • encoder_no_repeat_ngram_size: Prevent encoder n-gram repetition.
  • decoder_start_token_id: Define decoder start token.

Troubleshooting and best practices​

  • Ensure correct MinIO credentials before upload.
  • Verify GPU availability before model deployment.
  • If inference fails, check logs in the dashboard.
  • Use lower max_new_tokens for faster testing.
  • Always test endpoints with smaller inputs first.