{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "0JTbPe1ljoVK" }, "source": [ "# How to Fine-Tune a Decision Model with Unsloth (SFT + LoRA)\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/sft_for_decision_models_with_unsloth.ipynb)\n", "\n", "This tutorial shows you how to fine-tune a decision model, a model that makes decisions instead of generating text, on your own labels.\n", "\n", "We fine-tune our [`d1-3B`](https://huggingface.co/LiquidAI/d1-3B) decision model with Supervised Fine-tuning (SFT) and LoRA using Unsloth, with intent classification for customer support routing on Banking77 as the example.\n", "\n", "Follow along if it's your first time using Unsloth, or take single code snippets for your own workflow\n", "\n", "## 🎯 What You'll Find:\n", "- **LoRA + SFT** - Fine-tune the d1-3B decision model on your own decisions using LoRA (from PEFT)\n", "- **Evaluation** - Compare the decision model's accuracy before and after fine-tuning\n", "\n", "## 📋 Prerequisites:\n", "- **GPU Runtime**: Select a GPU in `Runtime` → `Change runtime type`. A free T4 GPU is enough (about 30 minutes of training)\n", "- **Hugging Face Account**: Only needed to push your model to the Hub" ] }, { "cell_type": "markdown", "metadata": { "id": "lQXUKSU0joVM" }, "source": [ "### Installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "sLCnGJx0joVM" }, "outputs": [], "source": [ "%%capture\n", "import os, re\n", "if \"COLAB_\" not in \"\".join(os.environ.keys()):\n", " !pip install unsloth\n", "else:\n", " # Do this only in Colab notebooks! Otherwise use pip install unsloth\n", " !pip install --no-deps bitsandbytes accelerate peft trl triton cut_cross_entropy unsloth_zoo\n", " !pip install sentencepiece protobuf \"datasets==4.3.0\" \"huggingface_hub>=0.34.0\" hf_transfer\n", " !pip install --no-deps unsloth\n", "!pip install transformers==5.17.0 \"torchao>0.16.0\"\n", "!pip install --no-deps trl==1.15.0" ] }, { "cell_type": "markdown", "metadata": { "id": "1uMS3OIUjoVN" }, "source": [ "### Loading d1-3B\n", "\n", "d1-3B ships its own code for the System One API (`system_one` and `system_one_batch`), so we load it with `trust_remote_code = True`.\n", "\n", "We also download the model repository locally. It contains `prompt.py`, the module d1-3B uses to turn a state and a question into a prompt. We will reuse it to build our training data, so that the model sees the exact same prompt during training and inference.\n", "\n", "Unsloth needs to be imported before Transformers, so we import it first." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 420, "referenced_widgets": [ "03c04a594e5f48d49d9a2da60fbb080b", "64da96b66a654ec993b2cd2036383320", "881194efc9294ca29c482bdb6e8759cd", "d37825edb50849628ec75c48e0bec20a", "b5c68d407c614b5ab66bae41c4d0deae", "2d958fc154f64c33aff29ac306ec3a67", "bf4df828ec474be4a6f892f018dfe08d", "e550889515a54245bad906821bf72774", "136ef49d1f694152a1ccf64c58296091", "e7d85285753344e4a87c2af0d1f8bf37", "08f548fe604f4bd08aa2175fda881bef", "d8fa90cf944d463aa0d383a64a4051de", "97e4fd0df391480f9193805c89aaad5a", "727f8bee49e542a9b5a49b67791e8f50", "20bd68fcde9f4885b6f4570edf7b563c", "c20789384919492a81abb08cef8ca124", "a2d73628fb7f408ca78cad2f7be655b7", "4bbfa23344774d8f840a2b7bda70c00a", "72644a7487b6483c8f21a2fe9c5cf38e", "52a3519bd5334981b5f63105bd816b4c", "4fe3e3cfc8b74c2aac280a61120024b1", "d8e87dedcc094bf1b68f8f820e29475d", "31b857126bbe4824ba66a36da80c4867", "6dbd4d6ce1af4a9487ba07941af59edf", "87c0aa1507b34194b22d30bab1ff7d8a", "51f0434b83464ee5a17129f46d2e5a6a", "b43d2e7d47b1448fa8aa2b792c281bed", "9b58df5e85304ac285c9046a66c8ed0c", "055aa35f3653479ca7fbe41af0a6e867", "46edbb8501fc4de9add3bd99f9c14e1c", "d007eee2aa5045609b20365f47e13d6c", "320c229b770e4f7b84c51531a831f724", "162c8584731143038cc1d18b5183ded8", "8770410cab0d481295f1a46cf89dbb33", "6f1f490026604509b03bf1305c465d73", "4d6ff46cd9774d06a6313e3bd96f5236", "0e0846faccf14438b64d86d4b99385cd", "c53607e679504ef885f4b0c9e2d18afb", "abc7d17068f94bb699f72d737f130a37", "44acbbe1f94346bdb27c0a0d33e44bcb", "24492f465753424bb5e66c362327fb7f", "828a3876d8514e8d9ae759ff7500d5af", "1ca8986a4a7e442d89574547f9a82d65", "1de969d465774b0c848f1446326f738d" ] }, "id": "ou1kGQXKjoVN", "outputId": "bf971cda-db60-4c1c-d7df-63810f096831" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n", "🦥 Unsloth Zoo will now patch everything to make training faster!\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "03c04a594e5f48d49d9a2da60fbb080b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading bytes: | 0.00B " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d8fa90cf944d463aa0d383a64a4051de", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Reconstructing (incomplete total...): | | 0.00B / 0.00B " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "31b857126bbe4824ba66a36da80c4867", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Fetching 15 files: 0%| | 0/15 [00:00<|im_start|>user\n", "I was charged twice this month, please refund one of them.\n", "\n", "\n", "QUESTION:\n", "Which team should handle this?\n", "\n", "Options:\n", "A Charges, refunds, invoices\n", "B App or site faults\n", "C Suspected unauthorised use\n", "\n", "Reply with the option code only.<|im_end|>\n", "<|im_start|>assistant\n", "\n" ] } ], "source": [ "from prompt import aliases, as_question, render\n", "\n", "rendered_prompt = render(\n", " tokenizer = tokenizer,\n", " state = state,\n", " q = as_question(questions[\"team\"]),\n", " bos = tokenizer.bos_token,\n", ")\n", "\n", "print(rendered_prompt)" ] }, { "cell_type": "markdown", "metadata": { "id": "DyJoqaWxjoVO" }, "source": [ "The model then runs one forward pass and reads the next-token probabilities at the end of the prompt, the **answer slot**. It only looks at the tokens of the option codes and normalizes them into the probabilities you saw above.\n", "\n", "This means that fine-tuning a decision model is regular SFT where **the completion is a single token: the code of the correct option**, followed by the end-of-turn token `<|im_end|>`." ] }, { "cell_type": "markdown", "metadata": { "id": "_kJqSpYvtIhW" }, "source": [ "Let's free up the GPU memory before we load the model for training:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "caQmy8WyjoVP" }, "outputs": [], "source": [ "import gc\n", "\n", "del model\n", "gc.collect()\n", "torch.cuda.empty_cache()" ] }, { "cell_type": "markdown", "metadata": { "id": "MmogV3mIjoVP" }, "source": [ "### Unsloth\n", "\n", "For training, we load the same weights with Unsloth's `FastVisionModel`. d1-3B's checkpoint uses the standard LFM2-VL module names, so we don't need its custom code here.\n", "\n", "**Why not `FastDecisionModel`?** Unsloth also has `FastDecisionModel` and `DecisionTrainer` for training decision models. Given d1-3B, they would add a new, randomly initialized decision head on top of the model and save the result in Unsloth's own decision-model format. You would lose d1-3B's trained readout and its System One API. Instead, we fine-tune d1-3B's own readout: the model keeps answering at the answer slot, and the fine-tuned model works with `system_one` exactly like d1-3B." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 185, "referenced_widgets": [ "4f0e38cfbf0743b0bc6fdb9010e739a2", "dd4287fc4acc4259ba89ccc2abc058a4", "7393b78b5e024bb389d07074c7b45090", "7b0399fa02904c94a88ac00411e0df1c", "9d091125f1db411bb64b00b6707b771a", "d4cf42c29da34e17a71fedafe1164180", "869b27576e2949d4ab157816cc021edf", "596aad4eb3a64a02aa90a798038cde34", "db4cd5acaf3a4d02a3fc71dfb93b1d22", "73a6bcb5cae044fdb7a96b05635e309e", "248249a9886742eb8abc25f16305f496" ] }, "id": "tVXVqJo6joVP", "outputId": "92eafab9-a282-430c-da75-11d32cc307e0" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "==((====))== Unsloth 2026.10.6: Fast Lfm2 patching. Transformers: 5.17.0.\n", " \\\\ /| NVIDIA A100-SXM4-40GB. Num GPUs = 1. Max memory: 39.494 GB. Platform: Linux.\n", "O^O/ \\_/ \\ Torch: 2.11.0+cu130. CUDA: 8.0. CUDA Toolkit: 13.0. Triton: 3.6.0\n", "\\ / Bfloat16 = TRUE. FA [Xformers = None. FA2 = False]\n", " \"-____-\" Free license: http://github.com/unslothai/unsloth\n", "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n", "Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4f0e38cfbf0743b0bc6fdb9010e739a2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading weights: 0%| | 0/707 [00:00 0 ! Suggested 8, 16, 32, 64, 128\n", " # Only the language model's layers, not the vision encoder's\n", " target_modules = r\".*language_model.*\\.(q_proj|k_proj|v_proj|out_proj|in_proj|w1|w2|w3)\",\n", " lora_alpha = 32,\n", " lora_dropout = 0, # Supports any, but = 0 is optimized\n", " bias = \"none\", # Supports any, but = \"none\" is optimized\n", " # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n", " use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n", " random_state = 3407,\n", " use_rslora = False, # We support rank stabilized LoRA\n", " loftq_config = None, # And LoftQ\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "8L3XIBRwjoVO" }, "source": [ "\n", "### Data Prep\n", "We will use [Banking77](https://huggingface.co/datasets/mteb/banking77), a dataset of ~13k online banking customer messages, each labeled with one of 77 fine-grained intents. We train on 3,000 messages and evaluate on 100 to keep things fast. Feel free to increase these numbers with `select(range(...))`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 411, "referenced_widgets": [ "5e0fc0cabf6b48f892686b466dd28a9f", "219d37b2b9a944bbafc34d49414da9c8", "8248b8a139ee45cfb1d0468adc826813", "31246f51ee884712b6ec2d05ba4d0692", "49e2c0d030ba4106b8bdf835156e5bb1", "21406961a98a409388491689f0f6402e", "c33143511cec400fbbac9e9a4c1345e3", "f2cd2c9403e74ecd83c70dc9356d2da4", "de3b624e3fc4428d8baf804bac2e9838", "4db4b9d04eba448286b13e4f9ac41c08", "f27466d27206480bbe02ae63efc2bfcb", "926ee0b9633d47eabbe06a6a7b36a6c6", "bdb8f789589f4e7fab7927dc8caf196b", "f5306cd2f115448f8f15aa83af49a6dc", "2b6706b950d2427a96910d4044acf7c0", "2f0cb5486fde4862962db33cad7f5157", "b839c1e9c76a4dc09577e7ffcfa5b0de", "015fa3f225684ad8931748a47dda1a7c", "8f41f45c7f7a41d8b119ac403e86a54e", "14d7840ee68d4b91ac07dfa0c46b92f7", "42819650e6a341a1a19664a9d4e1e9d0", "abc9b2fe278d49bfaebeb4c3194ee79a", "d7f58e68d0314236892978ddf67b4279", "8a397a8800bb44ea92253b380a683d31", "32f1f8fa11c645529beb1be1120c86e8", "43c5aa604ad149baa07d59dde6b0cfe0", "d9d47cea45214d499c67f29cd2f0a620", "5d3dfcefbdd14894a89e43e69d347080", "a3eb202a97db4635844b64fc132d6614", "b737b367d53f4e048eae1183bbebf5fe", "05ffc50e7865445d98fe47fab17ed2e2", "817c2ef412574caab4ae5d0d784b138d", "f1071fa6a67a4bca9af453cda477a166", "04fe5b3f62b244c6b74d44bed0badd6d", "133c23117b6949dabb718876ce407c3d", "1c1973d03a7847da9ade13ed6c700614", "bdda8cd794c24ec1bc6cc84c11a3cfcd", "83916cfb75ce469d916eb0c50e31578d", "ba4c2c5e5a6e403a94690ac8db231d2c", "7c66d73b0733421ab07f31af9365971d", "83271b372521447980b89fbcab3c4978", "ee31a08faa904be2bc0c4b5527bcaeb3", "094d4b97fcf24b899cffcfaf9ca74a3a", "451f150a82b34c46a47c1eb5c6ec1357", "582961f2461a41cd9133ea9ef9ecb48a", "ec6613fbf23d46b7a7522ec25d9299fa", "cf8207124959402d8b047d7fa986acdf", "5a3fe471a00a4c3383f988df40c1d3ba", "192fa535aa3f4960b6e3854b042aa7b9", "91a842df1f924f45a26609c99e90694b", "67147426145242b3b07c595017e144cd", "28ded8ca27a043ef97588ac0c8f50200", "c4d16ce606f946f3af7d9beb788b1389", "81da6fba428443f68057b3b3090431c8", "7d819b920e3744bcb2a95e52a865d1a2", "2c24a04406d94f2fbe6f5fb54fc41f09", "c30c37b62da94927a68c84150264fe06", "5508036e6f694b289b11f4ab07446707", "7a0cc72996e44e0fadad74a61134fc46", "d63667bc8f144149a47c8eeb9b00e6b5", "94c29cf1498b4bbfb91bbb4cb7c815ec", "7862e0c9dd61464a914a642d3ea25eb5", "b3958f44857a48da8a6dc80729d77d6d", "abc03bedba1f45ec8e836c7c00a9127a", "51b9efd480f7490cbca586f30c3c7e63", "678ebc8d485943cda1466af3bb80cf47", "e51c2badab734c029b5e3ed81eafab5a", "c57be694881e49d6a13b3440659ad8f3", "8b8b5f4a88c649df92f4cf69a45e241e", "1a58e13f230a47a2921b615b9dfad5c3", "d061a9cd4ad942a8b3877cfd6c4970f5", "ea3b1f669df042db9f42461a879cf961", "979251096bc24bed85570e2783ae8b0e", "9148c9f3c9fe4c60bdd8ee8e1315e37b", "337ca56ac70c4e62bfc7ceccb18f7cea", "fe534abc4591409db0f1b195afb2b793", "77ebb87f945a4ffe8e554b89a3353733" ] }, "id": "k0eG3VIVjoVO", "outputId": "43809320-c67d-4807-ebd9-5271be82cad2" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5e0fc0cabf6b48f892686b466dd28a9f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "README.md: 0%| | 0.00/16.6k [00:00= threshold\n", "\n", "def case_ids(texts):\n", " \"\"\"Give messages that reword each other the same case ID, transitively.\"\"\"\n", " word_sets = [words(text) for text in texts]\n", " parent = list(range(len(texts)))\n", "\n", " def find(i):\n", " while parent[i] != i:\n", " i = parent[i]\n", " return i\n", "\n", " for i in range(len(texts)):\n", " for j in range(i + 1, len(texts)):\n", " if same_case(word_sets[i], word_sets[j]):\n", " parent[find(j)] = find(i)\n", " return [find(i) for i in range(len(texts))]\n", "\n", "# 1. Drop training messages that reword a test message\n", "test_words = [words(text) for text in test_dataset[\"text\"]]\n", "leaked = {i for i, text in enumerate(train_dataset[\"text\"]) if any(same_case(words(text), w) for w in test_words)}\n", "train_dataset = train_dataset.select([i for i in range(len(train_dataset)) if i not in leaked])\n", "\n", "# 2. Hold out 10% of the training messages to track the loss, taking whole cases only\n", "cases = case_ids(train_dataset[\"text\"])\n", "case_sizes = Counter(cases)\n", "shuffled_cases = sorted(case_sizes)\n", "random.Random(42).shuffle(shuffled_cases)\n", "eval_cases, n_eval = set(), 0\n", "for case in shuffled_cases:\n", " if n_eval >= 0.1 * len(train_dataset):\n", " break\n", " eval_cases.add(case)\n", " n_eval += case_sizes[case]\n", "\n", "sft_train = train_dataset.select([i for i, case in enumerate(cases) if case not in eval_cases])\n", "sft_eval = train_dataset.select([i for i, case in enumerate(cases) if case in eval_cases])\n", "\n", "print(f\"Dropped {len(leaked)} training messages that reword a test message\")\n", "print(f\"{len(cases) - len(case_sizes)} training messages reword another training message\")\n", "print(f\"No case on both sides: {len(sft_train)} train and {len(sft_eval)} eval messages\")" ] }, { "cell_type": "markdown", "metadata": { "id": "Wcy3fWS6joVO" }, "source": [ "We turn the classification task into a single `choice` question. Each intent becomes one option, described by its label name.\n", "\n", "**Important**: The option codes are assigned by position: the first option gets the first code, the second option the second, and so on. If we always trained with the options in the same order, the model could learn which code means which intent without reading the option list. Adding or reordering an intent later would then shift the codes and break the fine-tuned model. We avoid that below by giving every training example its own random subset of the intents, in a random order." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LdC70OtgjoVP", "outputId": "077d82d0-04d9-4ed3-f831-ae4fab11b15f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of options: 77\n", "First option codes: {'Refund_not_showing_up': '00', 'activate_my_card': '01', 'age_limit': '02', 'apple_pay_or_google_pay': '03', 'atm_support': '04'}\n" ] } ], "source": [ "labels = sorted(set(dataset[\"train\"][\"label_text\"]))\n", "\n", "intent_question = {\n", " \"type\": \"choice\",\n", " \"instructions\": \"Which intent does this banking customer message express?\",\n", " \"criteria\": {label: label.replace(\"_\", \" \") for label in labels},\n", "}\n", "\n", "# The code d1-3B assigns to each option in this order, e.g. \"card_arrival\" -> \"12\"\n", "codes = {label: code for label, (code, _) in zip(labels, aliases(tokenizer, labels))}\n", "\n", "print(f\"Number of options: {len(labels)}\")\n", "print(f\"First option codes: {dict(list(codes.items())[:5])}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "MqkMRsisjoVP" }, "source": [ "With up to 26 options, d1-3B uses the letters `A` to `Z` as codes. With more options, as here, it switches to the two-digit codes `00` to `76`, each of which is a single token. Going from 26 to 27 options therefore changes every code at once, which is another reason to train on shuffled option lists. Our subsets keep at least 27 options, so the codes stay two-digit, as they are when we ask with all 77 intents." ] }, { "cell_type": "markdown", "metadata": { "id": "XVoQ9gy5joVP" }, "source": [ "We now render every message into d1-3B's prompt with its own option list: the correct intent plus a random number of the other intents (27 to 77 options in total), in a random order. We append the code the correct intent gets in that list, followed by `<|im_end|>`, and save it to `text`.\n", "\n", "Because the list changes from example to example, a code says nothing about which intent it stands for: the model has to read the option list to find the right one, whichever intents it contains." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 81, "referenced_widgets": [ "83de817c54234a6faaaac51169be6fe5", "487ff3fdbeb840bab4008b6552c0ebb0", "2a4114b5b6834493bb9da0ab5c303332", "8545f7ae1f72487eab546cff06e95a54", "dbab4dd0d06540a88fdb5fd7faf6e72a", "058517e61a4044b8b9cead170e6c9c3c", "7fb4dc14e9ad44d0b86ff68b2adc6af2", "55870c16024f4c30a2fcbe4088bc2056", "98257cad0a41426fbbc066aba306c154", "cfdb3917b1ea4b0191004e46e93de2c6", "e6578f03423c46fa8bfb20ce20a43d12", "dea82f33ff5143f4be9fd7343e67b2c2", "8902583f4ef54b9cbab82aebb6c33221", "b1750648857f4c5285a7288df8ed6426", "0b08b0a1c3664b119588592ed1d31961", "ed7814fbf3ea49f984ecaa9dfac91294", "5a1a8884ca1f477e9d5b85e93e9dd9bf", "e7b8a3dc10534a0bac367ccd60c0c687", "99a859cdbdff490fab94748107eb0330", "a3e90ecd18364f3995bd820e27cef426", "9e49d36c3fa341e58c61987609e88b83", "2a131c7f4bbe4597af6236f6d0b46c67" ] }, "id": "MJ5FJV2AjoVP", "outputId": "af72d343-b608-482c-c4a0-1a65eb4540eb" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "83de817c54234a6faaaac51169be6fe5", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map: 0%| | 0/2694 [00:00<|im_start|>user\n", "Whats the process of tranfering money into an account?\n", "\n", "\n", "QUESTION:\n", "Which intent does this banking customer message express?\n", "\n", "Options:\n", "00 reverted card payment?\n", "01 card acceptance\n", "02 cash withdrawal charge\n", "03 card swallowed\n", "04 declined cash withdrawal\n", "05 get physical card\n", "06 wrong exchange rate for cash withdrawal\n", "07 request refund\n", "08 wrong amount of cash received\n", "09 automatic top up\n", "10 transaction charged twice\n", "11 apple pay or google pay\n", "12 top up by bank transfer charge\n", "13 edit personal details\n", "14 card payment not recognised\n", "15 card linking\n", "16 verify my identity\n", "17 supported cards and currencies\n", "18 extra charge on statement\n", "19 compromised card\n", "20 age limit\n", "21 terminate account\n", "22 top up by card charge\n", "23 card delivery estimate\n", "24 visa or mastercard\n", "25 direct debit payment not recognised\n", "26 atm support\n", "27 order physical card\n", "28 cash withdrawal not recognised\n", "29 exchange charge\n", "30 balance not updated after bank transfer\n", "31 why verify identity\n", "32 lost or stolen phone\n", "33 disposable card limits\n", "34 get disposable virtual card\n", "35 declined card payment\n", "36 transfer into account\n", "37 verify source of funds\n", "38 unable to verify identity\n", "39 transfer timing\n", "40 cancel transfer\n", "41 card payment wrong exchange rate\n", "42 country support\n", "43 getting spare card\n", "44 card arrival\n", "45 Refund not showing up\n", "46 passcode forgotten\n", "47 pending cash withdrawal\n", "48 top up by cash or cheque\n", "49 transfer not received by recipient\n", "50 card not working\n", "51 pin blocked\n", "\n", "Reply with the option code only.<|im_end|>\n", "<|im_start|>assistant\n", "36<|im_end|>\n" ] } ], "source": [ "print(train_dataset[0][\"text\"])" ] }, { "cell_type": "markdown", "metadata": { "id": "79d7cafc0225" }, "source": [ "Let's look at two training messages with the same intent. Each prompt has its own option list, so `card arrival` gets a different code in each, and the model has to read the list to answer." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "22428a6e7ab3", "outputId": "3f40fe7b-8509-4cb8-d509-e6dab81a1f3a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Message: Where is the card I ordered 2 weeks ago?\n", "Number of options: 76\n", "First options: ['00 get disposable virtual card', '01 edit personal details', '02 getting virtual card']\n", "Correct option: 08 card arrival\n", "Completion: '08<|im_end|>'\n", "\n", "Message: Can I track when my card will be delivered?\n", "Number of options: 69\n", "First options: ['00 activate my card', '01 getting spare card', '02 getting virtual card']\n", "Correct option: 55 card arrival\n", "Completion: '55<|im_end|>'\n", "\n" ] } ], "source": [ "# The first two training messages whose intent is card_arrival\n", "for i in [i for i, label in enumerate(sft_train[\"label_text\"]) if label == \"card_arrival\"][:2]:\n", " text = train_dataset[i][\"text\"]\n", " options = re.findall(r\"^\\d\\d .+$\", text, re.M)\n", " completion = text.rsplit(\"<|im_start|>assistant\\n\", 1)[1]\n", " print(f\"Message: {sft_train[i]['text']}\")\n", " print(f\"Number of options: {len(options)}\")\n", " print(f\"First options: {options[:3]}\")\n", " print(f\"Correct option: {next(option for option in options if option.endswith(' card arrival'))}\")\n", " print(f\"Completion: {completion!r}\\n\")" ] }, { "cell_type": "markdown", "metadata": { "id": "Ne4fYTMOjoVQ" }, "source": [ "\n", "### Train the model\n", "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We train for one full epoch. Our data is text-only, so we pass the text tokenizer instead of the processor." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 152, "referenced_widgets": [ "2f021d2ee38b4ff7aa9497b42b987cec", "d20c7b7a39e348e4997f9d7f633ed9ae", "caf532887dac4ef297cf455155cbc974", "c31096b3f8364370812c4936d8da3645", "cc4458e1323047b58e4df7e7b3725b69", "9d38f8c546a14a25bb0f98eb490c9ea8", "127418e30b734769ac718ceafbeb8d23", "7b53cb65c2564a47ae6fe75e2f43929c", "f22b0b86cf88418a883198bba41b1a92", "69604e546a6d4fd19f0621f77dc7eb3d", "2b05d4bbc1d1493f9a17866267251f35", "2b0e54d088524246a1704302a9df2e27", "f8dbad2f0bd44602add9b2b7a2de32db", "8d9b93ce4fae4a0b8624ea086b647538", "8a175672cef449a7aac21c811fdedfd4", "1d172594283749399955e35119e1be12", "2169a7cd0a5347ab8d3ce48bf3939899", "a2345183a13d49b584de2ba1f135d8d9", "946c50ede1e34d11aec080efeae5fa02", "222e9b0f6d924e74bb1364e65878ad76", "626257e9422d4638a514fc2ba9751b91", "519a5c4a15ca40c6945258f86381eea8" ] }, "id": "l5LwJpBVjoVQ", "outputId": "a6a5604b-049a-4908-c38a-973f2fdb762a" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:unsloth.hybrid_packing:Unsloth: hybrid linear-attention packing disabled (padded path): causal_conv1d_fn has no accelerated kernel.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Unsloth: We found double BOS tokens - we shall remove one automatically.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2f021d2ee38b4ff7aa9497b42b987cec", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Unsloth: Tokenizing [\"text\"] (num_proc=6): 0%| | 0/2694 [00:00user\\n\",\n", " response_part = \"<|im_start|>assistant\\n\",\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "Fz2HQr0jjoVQ" }, "source": [ "Let's verify masking the instruction part is done! Let's print the first row again." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "tDVfIn90joVQ", "outputId": "9e78d351-ef71-4a66-f070-2f65cb9a413e" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<|startoftext|><|im_start|>user\n", "Whats the process of tranfering money into an account?\n", "\n", "\n", "QUESTION:\n", "Which intent does this banking customer message express?\n", "\n", "Options:\n", "00 reverted card payment?\n", "01 card acceptance\n", "02 cash withdrawal charge\n", "03 card swallowed\n", "04 declined cash withdrawal\n", "05 get physical card\n", "06 wrong exchange rate for cash withdrawal\n", "07 request refund\n", "08 wrong amount of cash received\n", "09 automatic top up\n", "10 transaction charged twice\n", "11 apple pay or google pay\n", "12 top up by bank transfer charge\n", "13 edit personal details\n", "14 card payment not recognised\n", "15 card linking\n", "16 verify my identity\n", "17 supported cards and currencies\n", "18 extra charge on statement\n", "19 compromised card\n", "20 age limit\n", "21 terminate account\n", "22 top up by card charge\n", "23 card delivery estimate\n", "24 visa or mastercard\n", "25 direct debit payment not recognised\n", "26 atm support\n", "27 order physical card\n", "28 cash withdrawal not recognised\n", "29 exchange charge\n", "30 balance not updated after bank transfer\n", "31 why verify identity\n", "32 lost or stolen phone\n", "33 disposable card limits\n", "34 get disposable virtual card\n", "35 declined card payment\n", "36 transfer into account\n", "37 verify source of funds\n", "38 unable to verify identity\n", "39 transfer timing\n", "40 cancel transfer\n", "41 card payment wrong exchange rate\n", "42 country support\n", "43 getting spare card\n", "44 card arrival\n", "45 Refund not showing up\n", "46 passcode forgotten\n", "47 pending cash withdrawal\n", "48 top up by cash or cheque\n", "49 transfer not received by recipient\n", "50 card not working\n", "51 pin blocked\n", "\n", "Reply with the option code only.<|im_end|>\n", "<|im_start|>assistant\n", "36<|im_end|>\n" ] } ], "source": [ "print(tokenizer.decode(trainer.train_dataset[0][\"input_ids\"]))" ] }, { "cell_type": "markdown", "metadata": { "id": "SkiQdCUljoVQ" }, "source": [ "Now let's print the masked out example - you should see only the option code and `<|im_end|>`:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "CHvc32vTjoVR", "outputId": "e6fdc53d-b789-4247-db1b-85de1644c235" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "36<|im_end|>\n" ] } ], "source": [ "print(tokenizer.decode([tokenizer.pad_token_id if x == -100 else x for x in trainer.train_dataset[0][\"labels\"]]).replace(tokenizer.pad_token, \" \").strip())" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Wd4QlXMAjoVR", "outputId": "2c7496e3-6494-4d75-d897-44211189220d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GPU = NVIDIA A100-SXM4-40GB. Max memory = 39.494 GB.\n", "5.934 GB of memory reserved.\n" ] } ], "source": [ "# @title Show current memory stats\n", "gpu_stats = torch.cuda.get_device_properties(0)\n", "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n", "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n", "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n", "print(f\"{start_gpu_memory} GB of memory reserved.\")" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 438, "referenced_widgets": [ "eea432a7cb524c21bc9251ced6900360", "f2cf6920c40144e68376c1c18c79c9e7", "8e251a8e7b7b4c37b6b8de98ef0c687e", "fa2cd615965144dea5398a956c392a7e", "56b3d1ac5a6c438f9c55e2063d488b63", "664b66cfc4384e92923d224f1391b339", "dd31cd8881e249fcbbd6919456455060", "2ead898cdf6942e89d30bad575cfb026", "21dfa59475f844b793b22bcf5c7cd640", "f0045694bc4541b39710720e0a1d645c", "553d1894f01e4d1cbecbc5645aa79545" ] }, "id": "iVURO5u4joVR", "outputId": "d2255558-f4c1-4cb9-b5a2-ace5a77e546c" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1\n", " \\\\ /| Num examples = 2,694 | Num Epochs = 1 | Total steps = 169\n", "O^O/ \\_/ \\ Batch size per device = 4 | Gradient accumulation steps = 4\n", "\\ / Data Parallel GPUs = 1 | Total batch size (4 x 4 x 1) = 16\n", " \"-____-\" Trainable parameters = 24,461,312 of 3,147,945,200 (0.78% trained)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Unsloth: Will smartly offload gradients to save VRAM!\n", "Unsloth: Double buffering enabled (parallel H2D + compute) for backward pass.\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [169/169 03:45, Epoch 1/1]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining LossValidation Loss
500.1786540.208320
1000.2058570.175606
1500.1511480.161431
1690.1717420.160396

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "eea432a7cb524c21bc9251ced6900360", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Filter: 0%| | 0/300 [00:00\n", "### Saving, loading finetuned models\n", "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n", "\n", "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit, scroll down!" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WDqHVQhcjoVR", "outputId": "f4f3993d-9589-464d-8daf-189c5f6aa536" }, "outputs": [ { "data": { "text/plain": [ "('lora_model/tokenizer_config.json',\n", " 'lora_model/chat_template.jinja',\n", " 'lora_model/tokenizer.json')" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.save_pretrained(\"lora_model\") # Local saving\n", "tokenizer.save_pretrained(\"lora_model\")\n", "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n", "# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving" ] }, { "cell_type": "markdown", "metadata": { "id": "r7d-1W0ajoVS" }, "source": [ "### Saving to float16\n", "\n", "To use the System One API with your fine-tuned model, merge the LoRA adapters into the model and save it in `float16`. The merged checkpoint contains only the weights and the tokenizer, so we also copy d1-3B's own code and processor configuration into the same folder." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Y7jpvYzkjoVS", "outputId": "1b9346cb-2a71-4db4-c5d0-b8d45323ffa7" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Unsloth: Restored added_tokens_decoder metadata in d1-3B-banking77-merged/tokenizer_config.json.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Found HuggingFace hub cache directory: /root/.cache/huggingface/hub\n", "Checking cache directory for required files...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Unsloth: Copying 1 files from cache to `d1-3B-banking77-merged`: 100%|██████████| 1/1 [00:14<00:00, 14.14s/it]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully copied all 1 files from cache to `d1-3B-banking77-merged`\n", "Checking cache directory for required files...\n", "Cache check failed: tokenizer.model not found in local cache.\n", "Not all required files found in cache. Will proceed with downloading.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Unsloth: Preparing safetensor model files: 100%|██████████| 1/1 [00:00<00:00, 1787.85it/s]\n", "Unsloth: Merging weights into 16bit: 100%|██████████| 1/1 [00:20<00:00, 20.35s/it]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Unsloth: Merge process complete. Saved to `/content/d1-3B-banking77-merged`\n" ] } ], "source": [ "import shutil\n", "\n", "output_dir = \"d1-3B-banking77-merged\"\n", "\n", "# Merge to 16bit\n", "model.save_pretrained_merged(output_dir, tokenizer, save_method = \"merged_16bit\",)\n", "\n", "# d1-3B's code for the System One API, its processor configuration, and its license\n", "for filename in [\"api.py\", \"hybrid.py\", \"lfm2_vl.py\", \"modeling_d1.py\", \"prompt.py\", \"runner.py\", \"processor_config.json\", \"LICENSE\"]:\n", " shutil.copy(os.path.join(local_dir, filename), output_dir)\n", "\n", "# To push the merged model and d1-3B's code to the Hugging Face Hub, uncomment:\n", "# from huggingface_hub import create_repo, upload_folder\n", "# create_repo(\"your-model-path\", exist_ok = True)\n", "# upload_folder(folder_path = output_dir, repo_id = \"your-model-path\")" ] }, { "cell_type": "markdown", "metadata": { "id": "fPf-5OSYtIha" }, "source": [ "\n", "### Evaluation\n", "Let's see how fine-tuning changed d1-3B's decisions. We first free up the GPU memory and measure how well the original d1-3B performs on the 100 test messages. `system_one_batch` packs many messages into each forward pass." ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 98, "referenced_widgets": [ "f4235b3563fe4dcb8b5ab7db3e65ff0d", "72c5dd4a0e1f43659e0b2cac662fe606", "1a0661fa98104750b6010cf220823cf0", "5b6ff73e5f44421dabd7f29729876093", "52e1b76f6f8f472d8f05033b08e0093c", "6e82a820b6124574969a6a158d7a3e7e", "31e3d2f4944e4f2c967807fefdf9b85e", "30d9f42c98de42cfa0bb98ede2e10f84", "f38dc6a99b744cca8d5a63695a605fc8", "85d0111b76d6441ea69d7eddf05095a7", "6317ec04678c43efb92b580c8a023597", "ab08f3341a224e32b4370c6f3189cddc", "748d31f01556481eb71f166fda41a867", "c4e5ce4cfb62498c88e54058f10383c2", "fbe3c36026034b83909832fdb9cb2655", "a37ab2f878e148ef9141a38dbb7a7e70", "3e2a8a0487bc475fae7675f742199ed2", "04e2d43745cc433a87e259e066c45ad3", "dfa8679b6e0e4296991aa2168c144265", "022d919330db4f1ca02abee7487488ee", "1037993ebdb44abd9c44968792d95922", "0948fa2a66fd466a8ad99c4fe1738769" ] }, "id": "CIXCJ72rtIha", "outputId": "86c67f07-c922-4db8-fb8c-7690a3510fbd" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f4235b3563fe4dcb8b5ab7db3e65ff0d", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading weights: 0%| | 0/707 [00:00