--- title: "Fine-tune Llama 3 with LLaMA Factory" description: "Fine-tune Llama 3 8B with LoRA on a TIR Instance using LLaMA Factory, chat with the result, merge it into the model, or train from the LlamaBoard web UI." content_type: "how-to" product: "tir-instance" platform: "TIR" url: "/tir/tir-instance/how-to/fine-tune-llama-3-with-llama-factory/" topic: ["guides"] --- # Fine-tune Llama 3 with LLaMA Factory Fine-tune Llama 3 8B Instruct with LoRA on a small sample dataset, then chat with the fine-tuned model. You can also save a merged copy of the model in your workspace. LLaMA Factory is an open-source tool for fine-tuning large language models, from the command line or from its web interface, LlamaBoard. Every step runs on your instance, in a JupyterLab terminal or over SSH. Everything you keep goes in the workspace at `/home/jovyan`: the LLaMA Factory folder, the models downloaded from Hugging Face, and the training output. Files there survive restarts and stops, while the rest of the container is lost each time the instance restarts. ## Before you begin - A GPU instance in the **Running** state, from a pre-built image with JupyterLab. See [Create an instance](/tir/tir-instance/how-to/create-an-instance/). - Python 3.11 or later in the instance's image, which LLaMA Factory needs. Each pre-built image has its own Python version; run `python3 --version` in a terminal to check it. If yours is older, move the instance to another image; see [Update the image of an instance](/tir/tir-instance/how-to/update-the-image-of-an-instance/). - Room in the workspace for the models, which download into `/home/jovyan`. The workspace starts at 30 GB, and you can make it larger while the instance runs; see [Increase the workspace size](/tir/tir-instance/how-to/increase-the-workspace-size/). - For Meta's official Llama 3 models only: a Hugging Face account that has been granted access to them. Anyone can download the models used here. ## Install LLaMA Factory Open a JupyterLab terminal (**File** > **New** > **Terminal**), or connect over SSH. Run every later `llamafactory-cli` command in a terminal where the environment you create here is active. Create a virtual environment in the workspace and activate it: ```bash python3 -m venv /home/jovyan/llamafactory-env . /home/jovyan/llamafactory-env/bin/activate ``` The PyTorch pre-built image pins every `pip install` to its own build of PyTorch, even inside a new virtual environment, so clear that pin before you install anything: ```bash unset PIP_CONSTRAINT ``` Install PyTorch, then clone LLaMA Factory and install it with the packages it needs for metrics and 4-bit models: ```bash pip install --upgrade pip pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126 cd /home/jovyan git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git LLaMA-Factory cd LLaMA-Factory pip install -e . pip install -r requirements/metrics.txt -r requirements/bitsandbytes.txt ``` Check that the versions match and LLaMA Factory starts: ```bash python -c "import torch, torchvision, torchaudio; print(torch.__version__, torchvision.__version__, torchaudio.__version__)" llamafactory-cli version ``` This takes about 10 minutes, most of it the PyTorch download, and uses about four to five GB of the workspace. The environment and the LLaMA-Factory folder are both under `/home/jovyan`, so they, and your training output, survive restarts and stops. After a restart, open a terminal and activate the environment again: ```bash . /home/jovyan/llamafactory-env/bin/activate ``` Check that the environment sees the GPU: ```bash python -c "import torch; assert torch.cuda.is_available(), 'GPU not detected'" ``` The command finishes silently when a GPU is there. If it stops with `GPU not detected`, check that the instance runs on a GPU plan. ## Choose the GPUs and where models are saved Set these two variables in the same terminal, before you train: ```bash export CUDA_VISIBLE_DEVICES=0 # or "0,1" for multi-GPU export HF_HOME=/home/jovyan/.cache/huggingface ``` The first line picks the GPUs LLaMA Factory uses: `0` is the first GPU, and `0,1` uses the first two. The second line tells Hugging Face to save the models it downloads, and your sign-in token, in the workspace, where they stay through restarts. Set them again in each new terminal. ## Sign in to Hugging Face (optional) The `unsloth` models in this guide are open to everyone. To fine-tune Meta's official Llama 3 models instead, first request access on the model's page on Hugging Face. Once access is granted, sign in with a Hugging Face access token: ```bash hf auth login ``` Paste your token when you are asked for it. It is saved in the Hugging Face folder in the workspace, so it survives restarts. ## Create the training configuration LLaMA Factory reads its settings from a YAML or JSON file. In the LLaMA-Factory folder, write the training file: ```bash cat > train_llama3.yaml <<'EOF' ### model model_name_or_path: unsloth/llama-3-8b-Instruct-bnb-4bit ### method stage: sft do_train: true finetuning_type: lora lora_target: all ### dataset dataset: alpaca_en_demo template: llama3 cutoff_len: 2048 ### output output_dir: ./output/llama3-lora ### train per_device_train_batch_size: 2 num_train_epochs: 1 learning_rate: 2.0e-5 fp16: true EOF ``` The file trains a LoRA adapter, a small set of extra weights, on a 4-bit copy of Llama 3 8B Instruct, with supervised fine-tuning (`stage: sft`). The dataset is a small sample of English instructions and answers that comes in the LLaMA-Factory folder, and `template: llama3` formats each example the way Llama 3 expects a chat. To tune the run, change the batch size, the number of epochs, and the learning rate in the last section of the file. ## Train the model With the environment active, in the LLaMA-Factory folder, run: ```bash llamafactory-cli train train_llama3.yaml ``` Run it from the LLaMA-Factory folder, because LLaMA Factory reads the sample dataset from the folder's `data` subfolder. The first run downloads the model, then the terminal prints the training progress. When it finishes, the trained adapter is in `output/llama3-lora` in the LLaMA-Factory folder. The training runs only as long as the command does: closing the terminal, restarting the instance, or using more memory than the plan has all end it. See [The instance restarts on its own](/tir/tir-instance/troubleshooting/tir-instance/#the-instance-restarts-on-its-own). ## Chat with the fine-tuned model The chat reads a separate file, because it needs to load the adapter and takes different settings from training. Write it in the LLaMA-Factory folder: ```bash cat > chat_llama3.yaml <<'EOF' model_name_or_path: unsloth/llama-3-8b-Instruct-bnb-4bit adapter_name_or_path: ./output/llama3-lora template: llama3 infer_backend: huggingface EOF ``` The chat waits for you to type, so run it in the foreground: ```bash llamafactory-cli chat chat_llama3.yaml ``` Type a message at the prompt, and the fine-tuned model answers. Type `exit` to end the chat. To pick the GPUs for this one command only, put the GPU setting in front of it: ```bash CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat chat_llama3.yaml ``` ## Merge the adapter into the model (optional) Merging writes a standalone copy of the model with your fine-tuning built in, so you can load it on its own. The merge uses the full-precision model `unsloth/llama-3-8b-Instruct` as its base, because LLaMA Factory merges an adapter only into an unquantized model, and the copy used for training is 4-bit. Write the merge settings in the LLaMA-Factory folder: ```bash cat > merge_llama3.yaml <<'EOF' ### model model_name_or_path: unsloth/llama-3-8b-Instruct adapter_name_or_path: ./output/llama3-lora template: llama3 ### export export_dir: ./output/llama3-merged export_size: 5 export_device: cpu export_legacy_format: false EOF ``` Then run the merge: ```bash llamafactory-cli export merge_llama3.yaml ``` The merge downloads the full-precision model and, on the CPU, builds the merged model in the instance's memory. It needs a plan with enough memory: LLaMA Factory needs around 18 GB of RAM to merge an 8B model, and processes that use more memory than the plan has are ended. The merged model is saved in `output/llama3-merged` in the LLaMA-Factory folder, split into files of up to 5 GB. ## Fine-tune from the LlamaBoard web UI (optional) LlamaBoard, LLaMA Factory's web interface, lets you choose the model, dataset, and training settings in a form instead of a YAML file. It is a Gradio app. Start it in the LLaMA-Factory folder, bound to the Gradio add-on's port: ```bash GRADIO_SERVER_NAME=0.0.0.0 GRADIO_SERVER_PORT=7860 llamafactory-cli webui ``` The command keeps running while LlamaBoard is up; press Ctrl+C to stop it. Turn on the instance's Gradio add-on, then open LlamaBoard at its address; see [Use Gradio on an instance](/tir/tir-instance/how-to/use-gradio-on-an-instance/). ## Where your fine-tuned model is Your LoRA adapter is in `output/llama3-lora` in the LLaMA-Factory folder in `/home/jovyan`, and the merged model, if you made one, is next to it in `llama3-merged`. Both stay in the workspace through restarts and stops until you delete the instance. To download them to your computer, see [Copy files to and from an instance](/tir/tir-instance/how-to/copy-files-to-and-from-an-instance/). If a download fills the disk or the instance restarts during training, see [Troubleshooting TIR Instances](/tir/tir-instance/troubleshooting/tir-instance/). For more training methods and examples, see the [LLaMA Factory repository](https://github.com/hiyouga/LlamaFactory) and its [CLI examples](https://github.com/hiyouga/LlamaFactory/blob/main/examples/README.md).