{ "cells": [ { "cell_type": "markdown", "id": "cell-0", "metadata": {}, "source": [ "# Continued Pre-training (CPT) with Unsloth for Text Completion\n", "\n", "Fine-tuning requires a GPU. If you don't have one locally, you can run this notebook for free on [Google Colab](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/cpt_text_completion_with_unsloth.ipynb) using a free NVIDIA T4 GPU instance.\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/cpt_text_completion_with_unsloth.ipynb)\n", "\n", "### What's in this notebook?\n", "\n", "In this notebook you will learn how to use Continued Pre-training (CPT) with Unsloth to adapt a language model for text completion and generation tasks.\n", "We will use the [LFM2.5-1.2B-Base](https://docs.liquid.ai/docs/models/lfm25-1.2b-base) model and perform continued pre-training on the Tiny Stories dataset. This approach teaches the model domain-specific knowledge, writing styles, and creative text generation patterns. It's ideal for tasks like novel writing, creative content generation, or adapting models to specific narrative styles.\n", "\n", "We will cover\n", "- Environment setup\n", "- Data preparation for text completion\n", "- Continued pre-training on raw text data\n", "- Local inference with your new model\n", "- Model saving and exporting it into the format you need for **deployment**.\n", "\n", "### Deployment options\n", "\n", "LFM2.5 models are small and efficient, enabling deployment across a wide range of platforms:\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Deployment TargetUse Case
📱 AndroidMobile apps on Android devices
📱 iOSMobile apps on iPhone/iPad
🍎 Apple Silicon MacLocal inference on Mac with MLX
🦙 llama.cppLocal deployments on any hardware
🦙 OllamaLocal inference with easy setup
🖥️ LM StudioDesktop app for local inference
⚡ vLLMCloud deployments with high throughput
☁️ ModalServerless cloud deployment
🏗️ BasetenProduction ML infrastructure
🚀 FalFast inference API
" ] }, { "cell_type": "markdown", "id": "cell-0b", "metadata": {}, "source": [ "### Need help building with our models and tools?\n", "Join the Liquid AI Discord Community and ask.\n", "\n", "\"Join\n", "\n", "And now, let the fine tune begin!" ] }, { "cell_type": "markdown", "metadata": { "id": "upIjjh5y_2_t" }, "source": [ "### Installation" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "EYUjxF2m_2_u" }, "outputs": [], "source": [ "%%capture\n", "import os, re\n", "if \"COLAB_\" not in \"\".join(os.environ.keys()):\n", " !pip install unsloth\n", "else:\n", " # Do this only in Colab notebooks! Otherwise use pip install unsloth\n", " import torch; v = re.match(r\"[0-9]{1,}\\.[0-9]{1,}\", str(torch.__version__)).group(0)\n", " xformers = \"xformers==\" + (\"0.0.33.post1\" if v==\"2.9\" else \"0.0.32.post2\" if v==\"2.8\" else \"0.0.29.post3\")\n", " !pip install --no-deps bitsandbytes accelerate {xformers} peft trl triton cut_cross_entropy unsloth_zoo\n", " !pip install sentencepiece protobuf \"datasets==4.3.0\" \"huggingface_hub>=0.34.0\" hf_transfer\n", " !pip install --no-deps unsloth\n", "!pip install transformers==4.57.3\n", "!pip install --no-deps trl==0.22.2" ] }, { "cell_type": "markdown", "metadata": { "id": "ymcV0ytQ_2_v" }, "source": [ "### Unsloth" ] }, { "cell_type": "markdown", "metadata": { "id": "Gr9K8phY_2_v" }, "source": [ "#### Text Completion / Raw Text Training\n", "This is a community notebook collaboration with [Mithex].\n", "\n", "We train on `Tiny Stories` (link [here](https://huggingface.co/datasets/roneneldan/TinyStories)) which is a collection of small stories. For example:\n", "```\n", "Once upon a time, there was a little car named Beep. Beep loved to go fast and play in the sun.\n", "Beep was a healthy car because he always had good fuel....\n", "```\n", "Instead of `Alpaca`'s Question Answer format, one only needs 1 column - the `\"text\"` column. This means you can finetune on any dataset and let your model act as a text completion model, like for novel writing.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "cLgiEH7o_2_w", "outputId": "58957361-2118-4d0d-8aac-aa88a480b58a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "env: UNSLOTH_RETURN_LOGITS=1 # Run this to disable CCE since it is not supported for CPT\n" ] } ], "source": [ "%env UNSLOTH_RETURN_LOGITS=1 # Run this to disable CCE since it is not supported for CPT" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "QmUBVEnvCDJv", "outputId": "8898696e-fef9-4d87-8fd4-5ece4f8ddde9" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "==((====))== Unsloth 2026.1.2: Fast Lfm2 patching. Transformers: 4.57.3.\n", " \\\\ /| Tesla T4. Num GPUs = 1. Max memory: 14.741 GB. Platform: Linux.\n", "O^O/ \\_/ \\ Torch: 2.9.0+cu126. CUDA: 7.5. CUDA Toolkit: 12.6. Triton: 3.5.0\n", "\\ / Bfloat16 = FALSE. FA [Xformers = 0.0.33.post1. FA2 = False]\n", " \"-____-\" Free license: http://github.com/unslothai/unsloth\n", "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n" ] } ], "source": [ "from unsloth import FastLanguageModel\n", "import torch\n", "max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!\n", "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n", "load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False.\n", "\n", "model, tokenizer = FastLanguageModel.from_pretrained(\n", " model_name = \"LiquidAI/LFM2.5-1.2B-Base\", # \"unsloth/mistral-7b\" for 16bit loading\n", " max_seq_length = max_seq_length,\n", " dtype = dtype,\n", " load_in_4bit = load_in_4bit,\n", " # token = \"hf_...\", # use one if using gated models like meta-llama/Llama-2-7b-hf\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "SXd9bTZd1aaL" }, "source": [ "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!\n", "\n", "We also add `embed_tokens` and `lm_head` to allow the model to learn out of distribution data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6bZsfBuZDeCL", "outputId": "65cdc185-2193-44bf-cc8b-48e0f589b427" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py:916: UserWarning: Model with `tie_word_embeddings=True` and the tied_target_modules=['model.embed_tokens', 'lm_head'] are part of the adapter. This can lead to complications, for example when merging the adapter or converting your model to formats other than safetensors. See for example https://github.com/huggingface/peft/issues/2018.\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Unsloth: Making `model.base_model.model.model.embed_tokens` require gradients\n" ] } ], "source": [ "model = FastLanguageModel.get_peft_model(\n", " model,\n", " r = 128, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n", " target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\", \"in_proj\",\n", " \"w1\", \"w2\", \"w3\",\n", " \"embed_tokens\", \"lm_head\",], # Add for continual pretraining\n", " lora_alpha = 32,\n", " lora_dropout = 0, # Supports any, but = 0 is optimized\n", " bias = \"none\", # Supports any, but = \"none\" is optimized\n", " # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n", " use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n", " random_state = 3407,\n", " use_rslora = True, # We support rank stabilized LoRA\n", " loftq_config = None, # And LoftQ\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "vITh0KVJ10qX" }, "source": [ "\n", "### Data Prep\n", "We now use the Tiny Stories dataset from https://huggingface.co/datasets/roneneldan/TinyStories. We only sample the first 5000 rows to speed training up. We must add `EOS_TOKEN` or `tokenizer.eos_token` or else the model's generation will go on forever." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 113, "referenced_widgets": [ "2599e99d841e4c388a59ec181d439232", "78cb442877844375a51779af4cf1460c", "050b3a0caeec42c79396fe6be35290b8", "db5ad287892240e3980dd45ef1236bb6", "42664e7ca03441aa9c03264f48cb6756", "a977a5f2468e460d838a16ffd107ecb7", "da2b95c86a6b4728ab073138d610351e", "93cba09fc5d5455da9a85e1c8b0b3374", "8f1f5b6d59a145f3ad4254fe03b20908", "ab009c83815b440a8bb71dac8294a6a9", "02b9709fb27246ef869971ef0f617039", "2ca47f0ca4614362b1f471d5a2a4644a", "139c21ea0d274536b0158e20fa9a2d6d", "c985d6d564144e98b09ce2b23aa22927", "6b9ec0c0278e4b00aac9d56c34db6b16", "f55cc83663d942a9ae5fa3b6433e4e14", "07a23075a3e74f70859651967cb3e404", "670e817a82804ea5aaac59c263f5d854", "fb8ae99ab8474df5a6ef8f7a795a643a", "8c2dedd677c644fc98697e000747a53f", "f5dea3934c7342299e9116a7d22eacdb", "6b86379f37ae4a529249b9e5856d1d90", "c670fd3bcf374bafad4483ffb9807f25", "574cf89e5160439dbd09acefe47a7174", "1a0a365a18d34e1da1350fcb38b42b7b", "158491ff42984fe2a0cd4c513e527952", "a507dca393fd49399c86604ac0978c55", "a1364ccf1d8f4fc2865905139bee2ffc", "348cf5addf5c435ab19cae749319330f", "bebc7fb198c441659b31978c9184c9c1", "7d1c29005bd04c1da263590a197d6734", "4392013ce91f4e3087ea7ea95ce2e490", "a5a92e30dd2c4c4ab6246c9fc49ab539" ] }, "id": "yXt8Na97yRe7", "outputId": "4d962183-8478-4d8b-fe78-946fd0249abb" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2599e99d841e4c388a59ec181d439232", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Generating train split: 0%| | 0/2119719 [00:00\n", "=========================\n", "Once upon a time, there was a little car named Beep. Beep loved to go fast and play in the sun. Beep was a healthy car because he always had good fuel. Good fuel made Beep happy and strong.\n", "\n", "One day, Beep was driving in the park when he saw a big tree. The tree had many leaves that were falling. Beep liked how the leaves fall and wanted to play with them. Beep drove under the tree and watched the leaves fall on him. He laughed and beeped his horn.\n", "\n", "Beep played with the falling leaves all day. When it was time to go home, Beep knew he needed more fuel. He went to the fuel place and got more healthy fuel. Now, Beep was ready to go fast and play again the next day. And Beep lived happily ever after.<|im_end|>\n", "=========================\n", "One day, a little fish named Fin was swimming near the shore. He saw a big crab and wanted to be friends. \"Hi, I am Fin. Do you want to play?\" asked the little fish. The crab looked at Fin and said, \"No, I don't want to play. I am cold and I don't feel fine.\"\n", "\n", "Fin felt sad but wanted to help the crab feel better. He swam away and thought of a plan. He remembered that the sun could make things warm. So, Fin swam to the top of the water and called to the sun, \"Please, sun, help my new friend feel fine and not freeze!\"\n", "\n", "The sun heard Fin's call and shone its warm light on the shore. The crab started to feel better and not so cold. He saw Fin and said, \"Thank you, little fish, for making me feel fine. I don't feel like I will freeze now. Let's play together!\" And so, Fin and the crab played and became good friends.<|im_end|>\n", "=========================\n", "Once upon a time, in a land full of trees, there was a little cherry tree. The cherry tree was very sad because it did not have any friends. All the other trees were big and strong, but the cherry tree was small and weak. The cherry tree was envious of the big trees.\n", "\n", "One day, the cherry tree felt a tickle in its branches. It was a little spring wind. The wind told the cherry tree not to be sad. The wind said, \"You are special because you have sweet cherries that everyone loves.\" The cherry tree started to feel a little better.\n", "\n", "As time went on, the cherry tree grew more and more cherries. All the animals in the land came to eat the cherries and play under the cherry tree. The cherry tree was happy because it had many friends now. The cherry tree learned that being different can be a good thing. And they all lived happily ever after.<|im_end|>\n", "=========================\n", "Once upon a time, there was a little girl named Lily. Lily liked to pretend she was a popular princess. She lived in a big castle with her best friends, a cat and a dog.\n", "\n", "One day, while playing in the castle, Lily found a big cobweb. The cobweb was in the way of her fun game. She wanted to get rid of it, but she was scared of the spider that lived there.\n", "\n", "Lily asked her friends, the cat and the dog, to help her. They all worked together to clean the cobweb. The spider was sad, but it found a new home outside. Lily, the cat, and the dog were happy they could play without the cobweb in the way. And they all lived happily ever after.<|im_end|>\n" ] } ], "source": [ "for row in dataset[:5][\"text\"]:\n", " print(\"=========================\")\n", " print(row)" ] }, { "cell_type": "markdown", "metadata": { "id": "idAEIeSQ3xdS" }, "source": [ "\n", "### Continued Pretraining\n", "Now let's use Unsloth's `UnslothTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 20 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`.\n", "\n", "Also set `embedding_learning_rate` to be a learning rate at least 2x or 10x smaller than `learning_rate` to make continual pretraining work!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 66, "referenced_widgets": [ "f1c635fb4c2e495ba43ca1fc4834e98c", "a18fe95c9bb148d7a6eadff0d0399d4b", "bcc4cabc4380450484c24a74d07c5256", "ba2b704495d542dbbfa16c1afcdb3788", "d93765a312c24f2c82db9f88f0eecd52", "3c70c4c9d3c54870a56f1db632a5b8c8", "0229c6f2f9c6465a91dcfa22ed18a5e2", "0f3051298dcd447abef1e8af0fce9bb2", "c1eab29b29c64a34b880a8680e2b69cf", "663591dcaaa94e83a3052daf8023da2d", "99e2b4f628b24651ae8fa1e8e2109877" ] }, "id": "95_Nn-89DhsL", "outputId": "686f6bd0-cac4-4849-af1b-0584b9db65a8" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f1c635fb4c2e495ba43ca1fc4834e98c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Unsloth: Tokenizing [\"text\"] (num_proc=4): 0%| | 0/2500 [00:00\n", " \n", " \n", " [157/157 11:37, Epoch 1/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
11.948500
22.100600
32.038700
41.841400
51.919800
61.880100
71.991700
81.673600
91.707500
101.764900
111.608300
121.629300
131.687400
141.552600
151.771100
161.592900
171.447700
181.538600
191.576500
201.633300
211.442300
221.644200
231.482100
241.540400
251.696300
261.591800
271.541900
281.584800
291.527000
301.463100
311.499900
321.552300
331.547400
341.579900
351.737500
361.535600
371.566700
381.529500
391.500500
401.559000
411.496500
421.368400
431.491800
441.503800
451.518500
461.589500
471.590900
481.508600
491.435400
501.392900
511.468800
521.438200
531.459800
541.502600
551.609300
561.468000
571.445300
581.321300
591.471600
601.484800
611.506500
621.493900
631.487400
641.499600
651.492300
661.394600
671.388500
681.516000
691.525500
701.500400
711.373900
721.372500
731.419600
741.537400
751.483700
761.351800
771.416500
781.502200
791.458800
801.525600
811.430200
821.464300
831.403100
841.477700
851.392200
861.507900
871.421500
881.382900
891.457400
901.584200
911.627100
921.569000
931.456600
941.488700
951.382300
961.406900
971.483800
981.411100
991.499400
1001.516900
1011.460300
1021.402500
1031.516100
1041.361700
1051.456400
1061.502800
1071.360700
1081.490900
1091.489200
1101.540500
1111.478900
1121.419700
1131.443300
1141.411400
1151.404500
1161.507100
1171.446600
1181.306800
1191.355300
1201.458700
1211.515100
1221.399100
1231.459800
1241.394000
1251.461300
1261.345500
1271.408000
1281.493800
1291.360900
1301.476200
1311.436100
1321.431200
1331.332400
1341.439800
1351.411300
1361.378000
1371.474600
1381.314700
1391.557500
1401.349400
1411.393400
1421.383300
1431.402000
1441.552400
1451.450500
1461.506800
1471.563000
1481.365200
1491.439300
1501.459100
1511.512900
1521.487400
1531.395200
1541.440100
1551.535400
1561.382100
1571.539200

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.12/dist-packages/peft/utils/save_and_load.py:279: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n", " warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n" ] } ], "source": [ "trainer_stats = trainer.train()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "colab": { "base_uri": "https://localhost:8080/" }, "id": "pCqnaKmlO1U9", "outputId": "d66af6b2-ec71-4f03-9c66-0e98c13ca897" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "758.4243 seconds used for training.\n", "12.64 minutes used for training.\n", "Peak reserved memory = 2.1 GB.\n", "Peak reserved memory for training = 0.905 GB.\n", "Peak reserved memory % of max memory = 14.246 %.\n", "Peak reserved memory for training % of max memory = 6.139 %.\n" ] } ], "source": [ "# @title Show final memory and time stats\n", "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n", "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n", "used_percentage = round(used_memory / max_memory * 100, 3)\n", "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n", "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n", "print(\n", " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n", ")\n", "print(f\"Peak reserved memory = {used_memory} GB.\")\n", "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n", "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n", "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "ekOmTR1hSNcr" }, "source": [ "\n", "### Inference\n", "Let's run the model!\n", "\n", "We first will try to see if the model follows the style and understands to write a story that is within the distribution of \"Tiny Stories\". Ie a story fit for a bed time story most likely.\n", "\n", "We select \"Once upon a time, in a galaxy, far far away,\" since it normally is associated with Star Wars." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "RHm11vaQRt1U", "outputId": "9d1d5397-b4ce-4b44-b740-0d9d04676e99" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<|startoftext|>Once upon a time, in a galaxy, far faraway, there was a little girl named Lily. She \n", "loved to explore and discover new things. One day, she found a shiny rock in the forest. She was so \n", "excited to touch it.\n", "\n", "Lily asked her mom, \"What is this?\" Her mom smiled and said, \"It's a rock. It's very \n", "special. It's like a treasure.\"\n", "\n", "Lily was so happy to have found a special rock. She decided to keep it \n", "with her forever. She loved to explore and discover new things, and she always remembered the special \n", "rock she found in the forest.<|im_end|>" ] } ], "source": [ "from transformers import TextIteratorStreamer\n", "from threading import Thread\n", "text_streamer = TextIteratorStreamer(tokenizer)\n", "import textwrap\n", "max_print_width = 100\n", "\n", "# Before running inference, call `FastLanguageModel.for_inference` first\n", "\n", "FastLanguageModel.for_inference(model)\n", "\n", "inputs = tokenizer(\n", "[\n", " \"Once upon a time, in a galaxy, far far away,\"\n", "]*1, return_tensors = \"pt\").to(\"cuda\")\n", "\n", "generation_kwargs = dict(\n", " inputs,\n", " streamer = text_streamer,\n", " max_new_tokens = 256,\n", " use_cache = True,\n", ")\n", "thread = Thread(target = model.generate, kwargs = generation_kwargs)\n", "thread.start()\n", "\n", "length = 0\n", "for j, new_text in enumerate(text_streamer):\n", " if j == 0:\n", " wrapped_text = textwrap.wrap(new_text, width = max_print_width)\n", " length = len(wrapped_text[-1])\n", " wrapped_text = \"\\n\".join(wrapped_text)\n", " print(wrapped_text, end = \"\")\n", " else:\n", " length += len(new_text)\n", " if length >= max_print_width:\n", " length = 0\n", " print()\n", " print(new_text, end = \"\")\n", " pass\n", "pass" ] }, { "cell_type": "markdown", "metadata": { "id": "fqO5CvHl_2_1" }, "source": [ "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n", "\n", "Some other links:\n", "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n", "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n", "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n", "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n", "\n", "

\n", " \n", " \n", " \n", "\n", " Join Discord if you need help + ⭐️ Star us on Github ⭐️\n", "\n", " This notebook and all Unsloth notebooks are licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme)\n", "
\n" ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "T4", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }