Fine-tune Llama 3 with LLaMA Factory

View as Markdown

Fine-tune Llama 3 8B Instruct with LoRA on a small sample dataset, then chat with the fine-tuned model. You can also save a merged copy of the model in your workspace. LLaMA Factory is an open-source tool for fine-tuning large language models, from the command line or from its web interface, LlamaBoard. Every step runs on your instance, in a JupyterLab terminal or over SSH.

Everything you keep goes in the workspace at /home/jovyan: the LLaMA Factory folder, the models downloaded from Hugging Face, and the training output. Files there survive restarts and stops, while the rest of the container is lost each time the instance restarts.

Before you begin

A GPU instance in the Running state, from a pre-built image with JupyterLab. See Create an instance.
Python 3.11 or later in the instance’s image, which LLaMA Factory needs. Each pre-built image has its own Python version; run python3 --version in a terminal to check it. If yours is older, move the instance to another image; see Update the image of an instance.
Room in the workspace for the models, which download into /home/jovyan. The workspace starts at 30 GB, and you can make it larger while the instance runs; see Increase the workspace size.
For Meta’s official Llama 3 models only: a Hugging Face account that has been granted access to them. Anyone can download the models used here.

Install LLaMA Factory

Open a JupyterLab terminal (File > New > Terminal), or connect over SSH. Run every later llamafactory-cli command in a terminal where the environment you create here is active.

Create a virtual environment in the workspace and activate it:

python3 -m venv /home/jovyan/llamafactory-env
. /home/jovyan/llamafactory-env/bin/activate

The PyTorch pre-built image pins every pip install to its own build of PyTorch, even inside a new virtual environment, so clear that pin before you install anything:

unset PIP_CONSTRAINT

Install PyTorch, then clone LLaMA Factory and install it with the packages it needs for metrics and 4-bit models:

pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
cd /home/jovyan
git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git LLaMA-Factory
cd LLaMA-Factory
pip install -e .
pip install -r requirements/metrics.txt -r requirements/bitsandbytes.txt

Check that the versions match and LLaMA Factory starts:

python -c "import torch, torchvision, torchaudio; print(torch.__version__, torchvision.__version__, torchaudio.__version__)"
llamafactory-cli version

This takes about 10 minutes, most of it the PyTorch download, and uses about four to five GB of the workspace. The environment and the LLaMA-Factory folder are both under /home/jovyan, so they, and your training output, survive restarts and stops. After a restart, open a terminal and activate the environment again:

. /home/jovyan/llamafactory-env/bin/activate

Check that the environment sees the GPU:

python -c "import torch; assert torch.cuda.is_available(), 'GPU not detected'"

The command finishes silently when a GPU is there. If it stops with GPU not detected, check that the instance runs on a GPU plan.

Choose the GPUs and where models are saved

Set these two variables in the same terminal, before you train:

export CUDA_VISIBLE_DEVICES=0   # or "0,1" for multi-GPU
export HF_HOME=/home/jovyan/.cache/huggingface

The first line picks the GPUs LLaMA Factory uses: 0 is the first GPU, and 0,1 uses the first two. The second line tells Hugging Face to save the models it downloads, and your sign-in token, in the workspace, where they stay through restarts. Set them again in each new terminal.

The unsloth models in this guide are open to everyone. To fine-tune Meta’s official Llama 3 models instead, first request access on the model’s page on Hugging Face. Once access is granted, sign in with a Hugging Face access token:

hf auth login

Paste your token when you are asked for it. It is saved in the Hugging Face folder in the workspace, so it survives restarts.

Create the training configuration

LLaMA Factory reads its settings from a YAML or JSON file. In the LLaMA-Factory folder, write the training file:

cat > train_llama3.yaml <<'EOF'
### model
model_name_or_path: unsloth/llama-3-8b-Instruct-bnb-4bit

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all

### dataset
dataset: alpaca_en_demo
template: llama3
cutoff_len: 2048

### output
output_dir: ./output/llama3-lora

### train
per_device_train_batch_size: 2
num_train_epochs: 1
learning_rate: 2.0e-5
fp16: true
EOF

The file trains a LoRA adapter, a small set of extra weights, on a 4-bit copy of Llama 3 8B Instruct, with supervised fine-tuning (stage: sft). The dataset is a small sample of English instructions and answers that comes in the LLaMA-Factory folder, and template: llama3 formats each example the way Llama 3 expects a chat. To tune the run, change the batch size, the number of epochs, and the learning rate in the last section of the file.

Train the model

With the environment active, in the LLaMA-Factory folder, run:

llamafactory-cli train train_llama3.yaml

Run it from the LLaMA-Factory folder, because LLaMA Factory reads the sample dataset from the folder’s data subfolder. The first run downloads the model, then the terminal prints the training progress. When it finishes, the trained adapter is in output/llama3-lora in the LLaMA-Factory folder.

The training runs only as long as the command does: closing the terminal, restarting the instance, or using more memory than the plan has all end it. See The instance restarts on its own.

Chat with the fine-tuned model

The chat reads a separate file, because it needs to load the adapter and takes different settings from training. Write it in the LLaMA-Factory folder:

cat > chat_llama3.yaml <<'EOF'
model_name_or_path: unsloth/llama-3-8b-Instruct-bnb-4bit
adapter_name_or_path: ./output/llama3-lora
template: llama3
infer_backend: huggingface
EOF

The chat waits for you to type, so run it in the foreground:

llamafactory-cli chat chat_llama3.yaml

Type a message at the prompt, and the fine-tuned model answers. Type exit to end the chat. To pick the GPUs for this one command only, put the GPU setting in front of it:

CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat chat_llama3.yaml

Merge the adapter into the model (optional)

Merging writes a standalone copy of the model with your fine-tuning built in, so you can load it on its own. The merge uses the full-precision model unsloth/llama-3-8b-Instruct as its base, because LLaMA Factory merges an adapter only into an unquantized model, and the copy used for training is 4-bit. Write the merge settings in the LLaMA-Factory folder:

cat > merge_llama3.yaml <<'EOF'
### model
model_name_or_path: unsloth/llama-3-8b-Instruct
adapter_name_or_path: ./output/llama3-lora
template: llama3

### export
export_dir: ./output/llama3-merged
export_size: 5
export_device: cpu
export_legacy_format: false
EOF

Then run the merge:

llamafactory-cli export merge_llama3.yaml

The merge downloads the full-precision model and, on the CPU, builds the merged model in the instance’s memory. It needs a plan with enough memory: LLaMA Factory needs around 18 GB of RAM to merge an 8B model, and processes that use more memory than the plan has are ended. The merged model is saved in output/llama3-merged in the LLaMA-Factory folder, split into files of up to 5 GB.

Fine-tune from the LlamaBoard web UI (optional)

LlamaBoard, LLaMA Factory’s web interface, lets you choose the model, dataset, and training settings in a form instead of a YAML file. It is a Gradio app. Start it in the LLaMA-Factory folder, bound to the Gradio add-on’s port:

GRADIO_SERVER_NAME=0.0.0.0 GRADIO_SERVER_PORT=7860 llamafactory-cli webui

The command keeps running while LlamaBoard is up; press Ctrl+C to stop it. Turn on the instance’s Gradio add-on, then open LlamaBoard at its address; see Use Gradio on an instance.

Where your fine-tuned model is

Your LoRA adapter is in output/llama3-lora in the LLaMA-Factory folder in /home/jovyan, and the merged model, if you made one, is next to it in llama3-merged. Both stay in the workspace through restarts and stops until you delete the instance. To download them to your computer, see Copy files to and from an instance. If a download fills the disk or the instance restarts during training, see Troubleshooting TIR Instances.

For more training methods and examples, see the LLaMA Factory repository and its CLI examples.