{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [], "gpuType": "T4" }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "accelerator": "GPU" }, "cells": [ { "cell_type": "markdown", "id": "cell-0", "metadata": {}, "source": [ "# Supervised Fine-tuning (SFT) with TRL\n", "\n", "Fine-tuning requires a GPU. If you don't have one locally, you can run this notebook for free on [Google Colab](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/sft_with_trl.ipynb) using a free NVIDIA T4 GPU instance.\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/sft_with_trl.ipynb)\n", "\n", "### What's in this notebook?\n", "\n", "In this notebook you will learn how to perform Supervised Fine-tuning (SFT) using the Hugging Face TRL (Transformer Reinforcement Learning) library.\n", "We will use the [LFM2.5-1.2B-Instruct](https://docs.liquid.ai/docs/models/lfm25-1.2b-instruct) model and fine-tune it on the SmolTalk dataset. You'll learn both standard SFT and parameter-efficient fine-tuning with LoRA for constrained hardware.\n", "\n", "We will cover\n", "- Environment setup\n", "- Data preparation\n", "- Model training with TRL's SFTTrainer\n", "- LoRA for parameter-efficient fine-tuning\n", "- Local inference with your new model\n", "- Model saving and exporting it into the format you need for **deployment**.\n", "\n", "### Deployment options\n", "\n", "LFM2.5 models are small and efficient, enabling deployment across a wide range of platforms:\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Deployment TargetUse Case
๐Ÿ“ฑ AndroidMobile apps on Android devices
๐Ÿ“ฑ iOSMobile apps on iPhone/iPad
๐ŸŽ Apple Silicon MacLocal inference on Mac with MLX
๐Ÿฆ™ llama.cppLocal deployments on any hardware
๐Ÿฆ™ OllamaLocal inference with easy setup
๐Ÿ–ฅ๏ธ LM StudioDesktop app for local inference
โšก vLLMCloud deployments with high throughput
โ˜๏ธ ModalServerless cloud deployment
๐Ÿ—๏ธ BasetenProduction ML infrastructure
๐Ÿš€ FalFast inference API
" ] }, { "cell_type": "markdown", "id": "cell-0b", "metadata": {}, "source": [ "### Need help building with our models and tools?\n", "Join the Liquid AI Discord Community and ask.\n", "\n", "\"Join\n", "\n", "And now, let the fine tune begin!" ] }, { "cell_type": "markdown", "source": [ "# ๐Ÿ“ฆ Installation & Setup\n", "\n", "First, let's install all the required packages:\n" ], "metadata": { "id": "x0RPLu2h9ome" } }, { "cell_type": "code", "source": [ "!pip install transformers==4.54.0 trl>=0.18.2 peft>=0.15.2" ], "metadata": { "id": "3FIcp_wo9nsR" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "Let's now verify the packages are installed correctly" ], "metadata": { "id": "41UEf1uxCd6m" } }, { "cell_type": "code", "source": [ "import torch\n", "import transformers\n", "import trl\n", "import os\n", "os.environ[\"WANDB_DISABLED\"] = \"true\"\n", "\n", "print(f\"๐Ÿ“ฆ PyTorch version: {torch.__version__}\")\n", "print(f\"๐Ÿค— Transformers version: {transformers.__version__}\")\n", "print(f\"๐Ÿ“Š TRL version: {trl.__version__}\")" ], "metadata": { "id": "bSJgYtHT_Os4" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "# Loading the model from Transformers ๐Ÿค—\n", "\n" ], "metadata": { "id": "v_uXLzxQ_rnK" } }, { "cell_type": "code", "source": [ "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "from IPython.display import display, HTML, Markdown\n", "import torch\n", "\n", "# Select a model to fine-tune from the list\n", "lfm_models = [\n", " \"LiquidAI/LFM2.5-1.2B-Instruct\",\n", " \"LiquidAI/LFM2.5-1.2B-JP\",\n", " \"LiquidAI/LFM2-8B-A1B\",\n", " \"LiquidAI/LFM2-2.6B-Exp\",\n", " \"LiquidAI/LFM2-2.6B\",\n", " \"LiquidAI/LFM2-700M\",\n", " \"LiquidAI/LFM2-350M\",\n", "]\n", "\n", "# Model to fine-tune\n", "model_id = \"LiquidAI/LFM2.5-1.2B-Instruct\"\n", "\n", "print(\"๐Ÿ“š Loading tokenizer...\")\n", "tokenizer = AutoTokenizer.from_pretrained(model_id)\n", "\n", "print(\"๐Ÿง  Loading model...\")\n", "model = AutoModelForCausalLM.from_pretrained(\n", " model_id,\n", " device_map=\"auto\",\n", " torch_dtype=\"bfloat16\",\n", "# attn_implementation=\"flash_attention_2\" <- uncomment on compatible GPU\n", ")\n", "\n", "print(\"โœ… Local model loaded successfully!\")\n", "print(f\"๐Ÿ”ข Parameters: {model.num_parameters():,}\")\n", "print(f\"๐Ÿ“– Vocab size: {len(tokenizer)}\")\n", "print(f\"๐Ÿ’พ Model size: ~{model.num_parameters() * 2 / 1e9:.1f} GB (bfloat16)\")" ], "metadata": { "id": "iA3erKM4-HhS" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "# ๐ŸŽฏ Part 1: Supervised Fine-Tuning (SFT)\n", "\n", "SFT teaches the model to follow instructions by training on input-output pairs (instruction vs response). This is the foundation for creating instruction-following models." ], "metadata": { "id": "6ABA6Yrm_lql" } }, { "cell_type": "markdown", "source": [ "## Load an SFT Dataset\n", "\n", "We will use [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), limiting ourselves to the first 5k samples for brevity. Feel free to change the limit by changing the slicing index in the parameter `split`." ], "metadata": { "id": "KufdgeypHtst" } }, { "cell_type": "code", "source": [ "from datasets import load_dataset\n", "\n", "print(\"๐Ÿ“ฅ Loading SFT dataset...\")\n", "train_dataset_sft = load_dataset(\"HuggingFaceTB/smoltalk\", \"all\", split=\"train[:5000]\")\n", "eval_dataset_sft = load_dataset(\"HuggingFaceTB/smoltalk\", \"all\", split=\"test[:500]\")\n", "\n", "print(\"โœ… SFT Dataset loaded:\")\n", "print(f\" ๐Ÿ“š Train samples: {len(train_dataset_sft)}\")\n", "print(f\" ๐Ÿงช Eval samples: {len(eval_dataset_sft)}\")\n", "print(f\"\\n๐Ÿ“ Single Sample: {train_dataset_sft[0]['messages']}\")" ], "metadata": { "id": "XCe8O06-_Cps" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Launch Training\n", "\n", "We are now ready to launch an SFT run with `SFTTrainer`, feel free to modify `SFTConfig` to play around with different configurations.\n", "\n" ], "metadata": { "id": "n5pI5JWpIlFQ" } }, { "cell_type": "code", "source": [ "from trl import SFTConfig, SFTTrainer\n", "\n", "sft_config = SFTConfig(\n", " output_dir=\"./lfm2-sft\",\n", " num_train_epochs=1,\n", " per_device_train_batch_size=1,\n", " learning_rate=5e-5,\n", " lr_scheduler_type=\"linear\",\n", " warmup_steps=100,\n", " warmup_ratio=0.2,\n", " logging_steps=10,\n", " save_strategy=\"epoch\",\n", " eval_strategy=\"epoch\",\n", " load_best_model_at_end=True,\n", " report_to=None,\n", " bf16=False # <- not all colab GPUs support bf16\n", ")\n", "\n", "print(\"๐Ÿ—๏ธ Creating SFT trainer...\")\n", "sft_trainer = SFTTrainer(\n", " model=model,\n", " args=sft_config,\n", " train_dataset=train_dataset_sft,\n", " eval_dataset=eval_dataset_sft,\n", " processing_class=tokenizer,\n", ")\n", "\n", "print(\"\\n๐Ÿš€ Starting SFT training...\")\n", "sft_trainer.train()\n", "\n", "print(\"๐ŸŽ‰ SFT training completed!\")\n", "\n", "sft_trainer.save_model()\n", "print(f\"๐Ÿ’พ SFT model saved to: {sft_config.output_dir}\")" ], "metadata": { "id": "ixD8Po-eAbPp" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "# ๐ŸŽ›๏ธ Part 2: LoRA + SFT (Parameter-Efficient Fine-tuning)\n", "\n", "LoRA (Low-Rank Adaptation) allows efficient fine-tuning by only training a small number of additional parameters. Perfect for limited compute resources!\n" ], "metadata": { "id": "08Y3TxKrBRXo" } }, { "cell_type": "markdown", "source": [ "## Wrap the model with PEFT\n", "\n", "We specify target modules that will be finetuned while the rest of the models weights remains frozen. Feel free to modify the `r` (rank) value:\n", "- higher -> better approximation of full-finetuning\n", "- lower -> needs even less compute resources" ], "metadata": { "id": "-MfWfc-Pvl9q" } }, { "cell_type": "code", "source": [ "from peft import LoraConfig, get_peft_model, TaskType\n", "\n", "GLU_MODULES = [\"w1\", \"w2\", \"w3\"]\n", "MHA_MODULES = [\"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\"]\n", "CONV_MODULES = [\"in_proj\", \"out_proj\"]\n", "\n", "lora_config = LoraConfig(\n", " task_type=TaskType.CAUSAL_LM,\n", " inference_mode=False,\n", " r=8, # <- lower values = fewer parameters\n", " lora_alpha=16,\n", " lora_dropout=0.1,\n", " target_modules=GLU_MODULES + MHA_MODULES + CONV_MODULES,\n", " bias=\"none\",\n", " modules_to_save=None,\n", ")\n", "\n", "lora_model = get_peft_model(model, lora_config)\n", "lora_model.print_trainable_parameters()\n", "\n", "print(\"โœ… LoRA configuration applied!\")\n", "print(f\"๐ŸŽ›๏ธ LoRA rank: {lora_config.r}\")\n", "print(f\"๐Ÿ“Š LoRA alpha: {lora_config.lora_alpha}\")\n", "print(f\"๐ŸŽฏ Target modules: {lora_config.target_modules}\")" ], "metadata": { "id": "puYp_gTpBSsf" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Launch Training\n", "\n", "Now ready to launch the SFT training, but this time with the LoRA-wrapped model" ], "metadata": { "id": "L1Hem_DOwHgY" } }, { "cell_type": "code", "source": [ "from trl import SFTConfig, SFTTrainer\n", "\n", "lora_sft_config = SFTConfig(\n", " output_dir=\"./lfm2-sft-lora\",\n", " num_train_epochs=1,\n", " per_device_train_batch_size=1,\n", " learning_rate=5e-5,\n", " lr_scheduler_type=\"linear\",\n", " warmup_steps=100,\n", " warmup_ratio=0.2,\n", " logging_steps=10,\n", " save_strategy=\"epoch\",\n", " eval_strategy=\"epoch\",\n", " load_best_model_at_end=True,\n", " report_to=None,\n", ")\n", "\n", "print(\"๐Ÿ—๏ธ Creating LoRA SFT trainer...\")\n", "lora_sft_trainer = SFTTrainer(\n", " model=lora_model,\n", " args=lora_sft_config,\n", " train_dataset=train_dataset_sft,\n", " eval_dataset=eval_dataset_sft,\n", " processing_class=tokenizer,\n", ")\n", "\n", "print(\"\\n๐Ÿš€ Starting LoRA + SFT training...\")\n", "lora_sft_trainer.train()\n", "\n", "print(\"๐ŸŽ‰ LoRA + SFT training completed!\")\n", "\n", "lora_sft_trainer.save_model()\n", "print(f\"๐Ÿ’พ LoRA model saved to: {lora_sft_config.output_dir}\")" ], "metadata": { "id": "u-VYQysHBY8-" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Save merged model\n", "\n", "Merge the extra weights learned with LoRA back into the model to obtain a \"normal\" model checkpoint." ], "metadata": { "id": "xI1-N-_Ev0cC" } }, { "cell_type": "code", "source": [ "print(\"\\n๐Ÿ”„ Merging LoRA weights...\")\n", "merged_model = lora_model.merge_and_unload()\n", "merged_model.save_pretrained(\"./lfm2-lora-merged\")\n", "tokenizer.save_pretrained(\"./lfm2-lora-merged\")\n", "print(\"๐Ÿ’พ Merged model saved to: ./lfm2-lora-merged\")" ], "metadata": { "id": "_rizEFUsvwce" }, "execution_count": null, "outputs": [] } ] }