{ "cells": [ { "cell_type": "markdown", "id": "cell-0", "metadata": {}, "source": [ "# Continued Pre-training (CPT) with Unsloth for Translation\n", "\n", "Fine-tuning requires a GPU. If you don't have one locally, you can run this notebook for free on [Google Colab](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/cpt_translation_with_unsloth.ipynb) using a free NVIDIA T4 GPU instance.\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/cpt_translation_with_unsloth.ipynb)\n", "\n", "### What's in this notebook?\n", "\n", "In this notebook you will learn how to use Continued Pre-training (CPT) with Unsloth to adapt a language model for translation tasks.\n", "We will use the [LFM2.5-1.2B-Base](https://docs.liquid.ai/docs/models/lfm25-1.2b-base) model and perform continued pre-training on Korean Wikipedia data, followed by instruction fine-tuning on Korean translation examples. This approach is ideal for adapting models to specific languages or translation domains.\n", "\n", "We will cover\n", "- Environment setup\n", "- Data preparation for translation\n", "- Continued pre-training on domain-specific data\n", "- Instruction fine-tuning for translation tasks\n", "- Local inference with your new model\n", "- Model saving and exporting it into the format you need for **deployment**.\n", "\n", "### Deployment options\n", "\n", "LFM2.5 models are small and efficient, enabling deployment across a wide range of platforms:\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Deployment TargetUse Case
πŸ“± AndroidMobile apps on Android devices
πŸ“± iOSMobile apps on iPhone/iPad
🍎 Apple Silicon MacLocal inference on Mac with MLX
πŸ¦™ llama.cppLocal deployments on any hardware
πŸ¦™ OllamaLocal inference with easy setup
πŸ–₯️ LM StudioDesktop app for local inference
⚑ vLLMCloud deployments with high throughput
☁️ ModalServerless cloud deployment
πŸ—οΈ BasetenProduction ML infrastructure
πŸš€ FalFast inference API
" ] }, { "cell_type": "markdown", "id": "cell-0b", "metadata": {}, "source": [ "### Need help building with our models and tools?\n", "Join the Liquid AI Discord Community and ask.\n", "\n", "\"Join\n", "\n", "And now, let the fine tune begin!" ] }, { "cell_type": "markdown", "metadata": { "id": "bRlUoyrp_3sE" }, "source": [ "### Installation" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "IJ9jTPoO_3sE" }, "outputs": [], "source": [ "%%capture\n", "import os, re\n", "if \"COLAB_\" not in \"\".join(os.environ.keys()):\n", " !pip install unsloth\n", "else:\n", " # Do this only in Colab notebooks! Otherwise use pip install unsloth\n", " import torch; v = re.match(r\"[0-9]{1,}\\.[0-9]{1,}\", str(torch.__version__)).group(0)\n", " xformers = \"xformers==\" + (\"0.0.33.post1\" if v==\"2.9\" else \"0.0.32.post2\" if v==\"2.8\" else \"0.0.29.post3\")\n", " !pip install --no-deps bitsandbytes accelerate {xformers} peft trl triton cut_cross_entropy unsloth_zoo\n", " !pip install sentencepiece protobuf \"datasets==4.3.0\" \"huggingface_hub>=0.34.0\" hf_transfer\n", " !pip install --no-deps unsloth\n", "!pip install transformers==4.57.3\n", "!pip install --no-deps trl==0.22.2" ] }, { "cell_type": "markdown", "metadata": { "id": "5yQ-92KP_3sF" }, "source": [ "### Unsloth" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "lvisH6xv_3sG", "outputId": "2849c6e1-d963-498d-aebc-4430c9bd3592" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "env: UNSLOTH_RETURN_LOGITS=1 # Run this to disable CCE since it is not supported for CPT\n" ] } ], "source": [ "%env UNSLOTH_RETURN_LOGITS=1 # Run this to disable CCE since it is not supported for CPT" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 365, "referenced_widgets": [ "1e9838e93bc64d379d033db4ac7470af", "2b3410f4178d49ef99235b1bd311f2d2", "0fe3527f3dca4d5692a3aea95be00002", "478ab8860b87499cac266fa3d8e2eec4", "a01dc10c20444b6b8d2df5d06ffbdc97", "8fe7917c02fd44c2bd6dbcbff4a7b9d4", "6125017f831e4d0a9963f5cd67bd94ae", "6c083fa23cf545dca7da83f27f89e630", "78378870887444f5a0a6ca85b8bc121b", "b7900cea9abb4876b1673e90947cf3e0", "c42006a113c14e30be647f5ac90a96d6", "c6df60f6e23e4deabaff94db0d8da9a7", "4aa1a4dc18b2439ebb4b422bdbfe24af", "fefbcc48c750472bbaa22b6a15bd95a3", "fca6ef25fe7c463a8df2287ed20c4747", "e32b00d3aa87460184fabf679b66f028", "630637f7b2794500a73ccf6066e34d69", "dc37abf624524453822c81b2a7299de0", "ee7d7a5e3214472f8b88b6c6ac60caed", "e723fe45c07f48b1993c295bdb1fbd39", "e73c451a9b1c497aaaeadd75daa46033", "e3daa8480cb948e48d4b3be8fcebc7e6", "25f6ec1e5d6846649c995b78a4415bd6", "cdf1c29ab64f427ca52405b567a232a7", "3eacec872a9f4f7ea0340d3f4384de19", "1230bc1efb9b4eb4a9e87e3750522767", "0d11451ccb11433d8226454bef80124a", "99d3f5f9076c4403a7a818470246b781", "ca07426d2b184bdd9cedf632f3360d5d", "99aea5e1d8f545d9bd8aa590a294cf72", "dd3b1237daf9409199e32e2bf5edfa0e", "b2ac43f92608404587f97291b8d46605", "7c5a3e375dca4d4aa4569770f9466444", "d5a3584e1a20488ea05da8c987a8dbc5", "591fd740d9ee4342a609e95aee551228", "3093b5bbc1bf463f87c1a8c10753f873", "f81bef48fb114a4bb8747bdff9ff8e97", "9425098593b3498194880685b66ba961", "80bb8f6df4234cdfb495e1e7a9a20f3d", "290bc6f9017b46cbbf33050037c7ca08", "747eb593f06a498c9e3418cde3b73740", "4ab6ff264a2b47d7aade4c6af575d04a", "8bdbe2c5554f4530bb3547145e50f87a", "38fc904747a74710a64b0a8eba963f16", "31e2f1e45309415aa1f65ef24078d272", "e6149c236a8a428f91f9d8ec861f91d1", "4df019ea3dd947a8ae900ee77b412b23", "4032b1215c86453296ee6906878fd948", "64194788328046019214a5ffa399825a", "81a367e4d7094825914166d739632e8c", "70807ae9bab24a90a8f00e9daf4080f1", "e92ac587c8024897bc22911d4b752747", "ce5f13d1954d46e9af7a23b3f6358ff5", "bff9140a1d5a45afacef0720062d132d", "fea6e68d12714c5b84cea87c3a5702d7", "a60ae73e1700412d856d24c5bbf7ff67", "b4cf337b53fc443686aca22196e94d61", "3ca8fb2d038c42129943188ee901040d", "fe9deaf629c4418a8de41bc04ddf03d9", "d8533ab7c97443c4b7644a00be833f6a", "360425030db643b6b0abb1dd5a933212", "543288c7075a46f68f7842843014729f", "ce9c3b4e68f649239468aefb799240fd", "4ebafb1c908b4407ab435f80479f2a3f", "d036c20bd50f4549b90e4b5c75661189", "f68b9da3d12d47f88fb94351411afdc0" ] }, "id": "QmUBVEnvCDJv", "outputId": "47b00b51-ccf0-4472-f1c4-cfe6f8dfeaa4" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "πŸ¦₯ Unsloth: Will patch your computer to enable 2x faster free finetuning.\n", "πŸ¦₯ Unsloth Zoo will now patch everything to make training faster!\n", "==((====))== Unsloth 2026.1.2: Fast Lfm2 patching. Transformers: 4.57.3.\n", " \\\\ /| NVIDIA L4. Num GPUs = 1. Max memory: 22.161 GB. Platform: Linux.\n", "O^O/ \\_/ \\ Torch: 2.9.0+cu126. CUDA: 8.9. CUDA Toolkit: 12.6. Triton: 3.5.0\n", "\\ / Bfloat16 = TRUE. FA [Xformers = 0.0.33.post1. FA2 = False]\n", " \"-____-\" Free license: http://github.com/unslothai/unsloth\n", "Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n", "Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1e9838e93bc64d379d033db4ac7470af", "version_major": 2, "version_minor": 0 }, "text/plain": [ "model.safetensors: 0%| | 0.00/2.34G [00:00 0 ! Suggested 8, 16, 32, 64, 128\n", " target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\", \"in_proj\",\n", " \"w1\", \"w2\", \"w3\",\n", " \"embed_tokens\", \"lm_head\",], # Add for continual pretraining\n", " lora_alpha = 32,\n", " lora_dropout = 0, # Supports any, but = 0 is optimized\n", " bias = \"none\", # Supports any, but = \"none\" is optimized\n", " # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n", " use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n", " random_state = 3407,\n", " use_rslora = True, # We support rank stabilized LoRA\n", " loftq_config = None, # And LoftQ\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "vITh0KVJ10qX" }, "source": [ "\n", "### Data Prep\n", "We now use the Korean subset of the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia) to first continually pretrain the model. You can use **any language** you like! Go to [Wikipedia's List of Languages](https://en.wikipedia.org/wiki/List_of_Wikipedias) to find your own language!\n", "\n", "**[NOTE]** To train only on completions (ignoring the user's input) read TRL's docs [here](https://huggingface.co/docs/trl/sft_trainer#train-on-completions-only).\n", "\n", "**[NOTE]** Remember to add the **EOS_TOKEN** to the tokenized output!! Otherwise you'll get infinite generations!\n", "\n", "If you want to use the `llama-3` template for ShareGPT datasets, try our conversational [notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Conversational.ipynb)\n", "\n", "For text completions like novel writing, try this [notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb).\n", "\n", "**[NOTE]** Use https://translate.google.com to translate from English to Korean!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LjY75GoYUCB8" }, "outputs": [], "source": [ "# Wikipedia provides a title and an article text.\n", "# Use https://translate.google.com!\n", "_wikipedia_prompt = \"\"\"Wikipedia Article\n", "### Title: {}\n", "\n", "### Article:\n", "{}\"\"\"\n", "# becomes:\n", "wikipedia_prompt = \"\"\"μœ„ν‚€ν”Όλ””μ•„ 기사\n", "### 제λͺ©: {}\n", "\n", "### 기사:\n", "{}\"\"\"\n", "\n", "EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN\n", "def formatting_prompts_func(examples):\n", " titles = examples[\"title\"]\n", " texts = examples[\"text\"]\n", " outputs = []\n", " for title, text in zip(titles, texts):\n", " # Must add EOS_TOKEN, otherwise your generation will go on forever!\n", " text = wikipedia_prompt.format(title, text) + EOS_TOKEN\n", " outputs.append(text)\n", " return { \"text\" : outputs, }\n", "pass" ] }, { "cell_type": "markdown", "metadata": { "id": "rw0tuebw4Ppe" }, "source": [ "We only use 1% of the dataset to speed things up! Use more for longer runs!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 209, "referenced_widgets": [ "f5b44d7628844194998d17f439a476ae", "c33f45386cb7422a8381eb06257e63ec", "22a6352b39294acfa236bfca33571487", "a3a9a21062b94af4b0d8c0a126e40369", "eba4108943444257ac4b67570bf047f4", "1930d3069ad14a278cd560cb01a041d4", "4912737e9232414db7ef5849e23a1394", "041a8eced75447fead96020e41af922b", "2ffcd689e2d446c6bd1df4cb6f961233", "d4222317566e471ba7a62f7ccba2596c", "61a98e57d9a8437aa8edfa4eba74a292", "10f8bb92c9744c1fa37108d2e53957e1", "be2eec5442a747dcb83e977edd86b041", "9839d96f3b9e4512a9bdf34f5ea39c24", "458b31289c27479e84199fde3f69b897", "b6d3e5d3686b4c93a5ac89169134b1ea", "1514149c62864a2982ce7abfaf787007", "0443ae929b344ee39956683567b7e1bc", "66ff2b8c29aa4ed3858d1b857afe62f0", "4375bc7686374a90aa2cd4f3521fdd0a", "e18a6ee78cf347bebc89c1a420ef41fb", "5332b07c1d38442f935fcdc57ef34cc4", "67d9b8410f954e9ab3eabedb8a37cbc1", "4f30a0e4491148cea0fcdc86a21bb413", "5d8cffb6c95c4b3ab51ebe696cf1bb4e", "e323bdfbb8ca4b79ba0c71bf6f162088", "1453de4e4cb7420cbb7d05a77f9b228c", "6bf957a8e542432a957a263857079ed6", "c89ec2fe239542cf9b13c8026a656e21", "d88366d41b3049d98fcaf401ca4851a6", "55f4c28aca454dc5bc72c0ece4e150f6", "92a5d1516a8345f2ad8f0d9aa632efb3", "feee83c3474949aea23ad351785318de", "9d1cf038ff83480cba57a34c779a923a", "0f7dec72dc844270ae02af80198d8cb1", "177ff3a7542647ccaa7224229dba32b9", "b011c215f3c149199dade06d21f24283", "f582a4a7ab1e49ec9cf7a367762bdc07", "55cee5f842da437c8bbdd5bd88338d6d", "397b04da588a4b1f90008164e7dc8d61", "9c6ffe60f9614613ac92b595225a464b", "70dfafe774cd44b89619a092a177ddb7", "ce80a0d62951437d8559ddf7166e8c44", "f0950c9d68eb484e816429f2d59ee782", "805a17b0d37b4702b99c19e275c091d3", "9f5b2d673d2541d59e68ba19d130b21c", "5b13ab52b84245c9a5f2c59a64663054", "48424fffbe4b4f93b6709142e971d43f", "c7312c23f9fc4cfa94128a04b408f092", "17b0280177fa48bbadb40a0b934eff28", "13dce43730a6413fa5fb3f0a35ffef6d", "dae461c0f0124b90b1d6f0e6f8ddde8b", "9b7210bdee304d3ea153322dfb47fdcd", "32d5006f3b6a43378973f286ef0dea41", "16f71874f4114ee4ab51aea4229af5d9", "848938d420a1454e91e25ddf08a6da29", "6dc0a3c7de7f4f9aada821bb6da4312c", "c47feedc8145408ba2c32a0f90da141b", "5358e774e4de4a3a99aeb38ea2f70639", "748c5974e93a4aec99ed4148c6342189", "c39437fe2ad04a71864936aa51e39e74", "ae3153cb80d4432bbe3b5114a17828b3", "31d8a1ccf7f848b5bbc652cb9068f431", "4b96c1019afd46449cfb11ecc0b44b12", "04ab4d3f01bf4aadb0bfb1bf79782956", "97e494c8734c49f5823240f1013fb21f" ] }, "id": "EsKrpkza3VB3", "outputId": "0e4bd26d-f036-4189-9caa-0e448618229a" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f5b44d7628844194998d17f439a476ae", "version_major": 2, "version_minor": 0 }, "text/plain": [ "README.md: 0.00B [00:00, ?B/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "10f8bb92c9744c1fa37108d2e53957e1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "20231101.ko/train-00000-of-00003.parquet: 0%| | 0.00/400M [00:00\n", "### Continued Pretraining\n", "Now let's use Unsloth's `UnslothTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 20 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`.\n", "\n", "Also set `embedding_learning_rate` to be a learning rate at least 2x or 10x smaller than `learning_rate` to make continual pretraining work!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 66, "referenced_widgets": [ "2c7d67c976654d5aba5ce2572b88fb49", "35fab26ac532431891df67a33f2fbe87", "35b4a7320d124f899df6fd52a9dbc7b4", "aa2716d36e254d3c9fd0b2ea5992a67a", "995bdd86d03243f4afb392b09d21b480", "4fbd255ab0c447dba8107e051e59adcb", "88ea19a2b9574bd592188fdcf562c842", "5fb2f4ece1e84f67adee65ca97009cd2", "cfbd4ab591744939b046f8ff06b0d858", "8f59572010d5452f90da2ab94c4c2f6d", "59e2286c7b1f4f33a4e257cbc295bac7" ] }, "id": "95_Nn-89DhsL", "outputId": "9673580b-4c1b-4f2e-ee4a-9b9bf2e48112" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2c7d67c976654d5aba5ce2572b88fb49", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Unsloth: Tokenizing [\"text\"] (num_proc=16): 0%| | 0/6478 [00:00\n", " \n", " \n", " [120/120 04:24, Epoch 0/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
12.204100
21.917100
32.075200
41.892900
52.122100
62.040400
71.662600
81.794300
91.842000
101.900900
111.580900
121.880000
131.626500
141.779500
151.831500
161.751900
171.732000
181.599300
191.661500
201.886500
211.524300
221.635700
231.810400
241.848200
251.813500
261.709500
271.761600
281.427200
291.798000
301.862400
311.655300
321.434500
331.819100
341.528400
351.745900
361.718900
371.842900
381.525800
391.917400
401.876700
411.869100
421.877200
431.762600
441.318200
451.580500
461.934600
471.865300
481.860000
491.902900
501.488700
511.805500
521.696800
531.806300
541.810700
551.887900
561.895300
571.495600
581.696500
591.908500
601.488400
611.594100
622.020000
631.512200
641.759700
651.668500
661.796000
671.812900
681.606400
692.003000
701.744600
711.623500
722.004400
731.898900
741.629500
751.756600
761.897300
771.776500
781.794000
791.913500
801.813300
811.990800
821.550800
831.600400
841.710200
851.687600
861.487900
871.694200
881.681300
891.636100
901.651300
911.599500
922.039100
931.700700
941.595400
951.803100
961.763100
972.054900
981.769400
991.730800
1001.728000
1011.551600
1022.047700
1031.739400
1041.587400
1051.699400
1061.753300
1071.607900
1081.621000
1091.658300
1101.780900
1111.531200
1121.856400
1131.739500
1141.693600
1151.576500
1161.738700
1171.593500
1181.719200
1191.768000
1201.714000

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.12/dist-packages/peft/utils/save_and_load.py:279: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n", " warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n" ] } ], "source": [ "trainer_stats = trainer.train()" ] }, { "cell_type": "markdown", "metadata": { "id": "R9dRBJZulavZ" }, "source": [ "### Instruction Finetuning\n", "\n", "We now use the [Alpaca in GPT4 Dataset](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean) but translated in Korean!\n", "\n", "Go to [vicgalle/alpaca-gpt4](https://huggingface.co/datasets/vicgalle/alpaca-gpt4) for the original GPT4 dataset for Alpaca or [MultilingualSIFT project](https://github.com/FreedomIntelligence/MultilingualSIFT) for other translations of the Alpaca dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 148, "referenced_widgets": [ "7500cd3fa2d94c4c9716871e55bcba47", "0d642cf4768f4502aa20deb268fddc91", "10cae01e7ebf42f3be74eb3134c5ad85", "11bb05e49f32426cb5894748ae0ad29f", "2719225c8dc34dca8ab99bc3e7c1806e", "a62cb7b45f7f4b29a0a985cc229f1c5b", "9289fe1169b94b258c4277aec164993d", "120c06068732444f9da6c0378f5010d3", "f0a952e6c8ca4da29fcdc67750e21ad4", "801074afc4d84b81a9fd2a4a8ecb9a5d", "05cd421d5e4746d2a6072ba65ebc2015", "25cbe6b8060148ee93ad3b95f8416211", "6169416a06b9464fae20e4c3e3425334", "e927bdfcedb7493dba81c01fad134ead", "bd597e8c08bb482292c4ea2c3f7dd4b5", "3c61eaa8d975477ea88b54ce84743bea", "e455c7b6f019456b880c627e7ef89e95", "bde7a4ca1216440587f46d8a11426fb7", "d3b8ba12821848faad8900f5ca37640c", "0021f7a8cbb949918cc8575b5cfbf4a0", "3eeb37932f314347b8a10567c5754111", "05e327a56e11466c921298b94ce5743a", "97ce6ebdb1e34829acf837f9a6d3221a", "ad18ae3a29a3420a836b39f8a0cb32e4", "84ce9b8487e34f4d835e337eeaeda4ed", "cecd13882474452e948fbc0aebdde72e", "8038a74c4c774b5fa0be882efafcb5ef", "dd218d77959c442794d736d238d3ca2f", "385d2e0325744086bd6f932ebee66f2e", "224e551aeb5e4234ad59df76c93a67e7", "625bd000f8094f04964054ee577ffcaa", "9612eac692ae494fbbe995361384cdac", "077ccd2208374dd4b9229270299fd563" ] }, "id": "1oNjUwxOyG8C", "outputId": "b92bedc5-cf8d-4be0-8f7a-fb0c533013ab" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7500cd3fa2d94c4c9716871e55bcba47", "version_major": 2, "version_minor": 0 }, "text/plain": [ "README.md: 0%| | 0.00/124 [00:00\n", " \n", " \n", " [120/120 03:21, Epoch 0/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
11.960100
21.916300
31.732600
41.592300
51.505100
61.324400
71.380100
81.478100
91.302900
101.327200
111.174000
121.323000
131.115200
141.341500
151.231500
161.223500
171.229800
181.279700
191.291400
201.158600
211.189800
221.202100
231.075500
241.153000
251.246700
261.095900
271.167800
281.235900
291.246900
301.001400
311.288600
321.111600
331.261200
341.193700
351.110400
361.192600
371.220400
381.277400
391.126100
401.205600
411.148900
421.144600
431.163200
441.305500
451.218100
461.024800
471.186500
481.210500
491.189400
501.102300
511.130700
520.952200
531.134600
541.204000
551.158900
561.324500
571.208600
581.189000
591.114100
601.131300
611.238700
621.178100
631.155200
641.088200
651.347600
661.097300
671.145100
681.058100
691.250400
701.258900
711.127400
721.161100
731.088800
741.147600
751.229800
761.093800
771.114500
781.188100
791.110300
801.132600
811.358300
821.215700
831.105100
841.052100
851.343200
861.124700
871.340600
881.115500
891.048400
901.184300
911.201300
921.095200
931.058900
941.347200
951.024100
961.127000
971.293200
981.106900
991.041000
1001.286100
1011.065200
1021.133100
1031.272900
1041.213900
1050.994700
1061.186600
1071.189600
1081.234500
1091.165800
1101.124100
1111.241200
1121.092300
1131.201000
1141.161800
1151.299800
1161.137100
1171.119700
1181.006000
1191.280800
1201.066700

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.12/dist-packages/peft/utils/save_and_load.py:279: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n", " warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n" ] } ], "source": [ "trainer_stats = trainer.train()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "colab": { "base_uri": "https://localhost:8080/" }, "id": "pCqnaKmlO1U9", "outputId": "05d48d2a-7f23-466f-f7ad-52c57c5fe5c1" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "226.8895 seconds used for training.\n", "3.78 minutes used for training.\n", "Peak reserved memory = 4.361 GB.\n", "Peak reserved memory for training = 1.763 GB.\n", "Peak reserved memory % of max memory = 19.679 %.\n", "Peak reserved memory for training % of max memory = 7.955 %.\n" ] } ], "source": [ "# @title Show final memory and time stats\n", "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n", "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n", "used_percentage = round(used_memory / max_memory * 100, 3)\n", "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n", "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n", "print(\n", " f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n", ")\n", "print(f\"Peak reserved memory = {used_memory} GB.\")\n", "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n", "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n", "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "ekOmTR1hSNcr" }, "source": [ "\n", "### Inference\n", "Let's run the model! You can change the instruction and input - leave the output blank!\n", "\n", "Remember to use https://translate.google.com/!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "B_GRpqmUiFRj", "outputId": "ce57b1e6-6aae-4878-97d8-8aed766387ea" }, "outputs": [ { "data": { "text/plain": [ "['<|startoftext|>λ‹€μŒμ€ μž‘μ—…μ„ μ„€λͺ…ν•˜λŠ” λͺ…λ Ήμž…λ‹ˆλ‹€. μš”μ²­μ„ μ μ ˆν•˜κ²Œ μ™„λ£Œν•˜λŠ” 응닡을 μž‘μ„±ν•˜μ„Έμš”.\\n\\n### 지침:\\nν”Όλ³΄λ‚˜μΉ˜ μˆ˜μ—΄μ„ κ³„μ†ν•˜μ„Έμš”: 1, 1, 2, 3, 5, 8,\\n\\n### 응닡:\\nν”Όλ³΄λ‚˜μΉ˜ μˆ˜μ—΄μ€ 각 μˆ«μžκ°€ μ•žμ˜ 두 숫자의 합인 μˆ˜μ—΄μž…λ‹ˆλ‹€. 이 μˆ˜μ—΄μ€ 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, ']" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# alpaca_prompt = Copied from above\n", "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n", "inputs = tokenizer(\n", "[\n", " alpaca_prompt.format(\n", " # \"Continue the fibonacci sequence: 1, 1, 2, 3, 5, 8,\", # instruction\n", " \"ν”Όλ³΄λ‚˜μΉ˜ μˆ˜μ—΄μ„ κ³„μ†ν•˜μ„Έμš”: 1, 1, 2, 3, 5, 8,\", # instruction\n", " \"\", # output - leave this blank for generation!\n", " )\n", "], return_tensors = \"pt\").to(\"cuda\")\n", "\n", "outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)\n", "tokenizer.batch_decode(outputs)" ] }, { "cell_type": "markdown", "metadata": { "id": "CrSvZObor0lY" }, "source": [ " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "e2pEuRb1r2Vg", "outputId": "e8962db0-d3b8-4177-9946-ff36a0fd0130" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<|startoftext|>λ‹€μŒμ€ μž‘μ—…μ„ μ„€λͺ…ν•˜λŠ” λͺ…λ Ήμž…λ‹ˆλ‹€. μš”μ²­μ„ μ μ ˆν•˜κ²Œ μ™„λ£Œν•˜λŠ” 응닡을 μž‘μ„±ν•˜μ„Έμš”.\n", "\n", "### 지침:\n", "ν•œκ΅­μŒμ•…μ€ μ–΄λ–€κ°€μš”?\n", "\n", "### 응닡:\n", "ν•œκ΅­μŒμ•…μ€ 전톡 μŒμ•…κ³Ό ν˜„λŒ€ μŒμ•…μ„ ν¬ν•¨ν•œ λ‹€μ–‘ν•œ μž₯λ₯΄λ₯Ό ν¬ν•¨ν•©λ‹ˆλ‹€. 전톡 μŒμ•…μ€ μ’…μ’… 악기와 λ…Έλž˜λ₯Ό μ‚¬μš©ν•˜μ—¬ ν•œκ΅­μ˜ 역사와 λ¬Έν™”λ₯Ό λ°˜μ˜ν•©λ‹ˆλ‹€. ν˜„λŒ€ μŒμ•…μ€ 전톡 μŒμ•…μ˜ 영ν–₯을 λ°›μ•„ λ°œμ „ν•΄μ™”μœΌλ©°, μ „μž μŒμ•…, 둝, 팝 λ“± λ‹€μ–‘ν•œ μž₯λ₯΄κ°€ μžˆμŠ΅λ‹ˆλ‹€. ν•œκ΅­ μŒμ•…μ€ μ „ μ„Έκ³„μ μœΌλ‘œ 인기λ₯Ό μ–»κ³  있으며, λ§Žμ€ μ•„ν‹°μŠ€νŠΈλ“€μ΄ ꡭ제적인 성곡을 κ±°λ‘μ—ˆμŠ΅λ‹ˆλ‹€.<|im_end|>\n" ] } ], "source": [ "# alpaca_prompt = Copied from above\n", "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n", "inputs = tokenizer(\n", "[\n", " alpaca_prompt.format(\n", " # \"What is Korean music like?\"\n", " \"ν•œκ΅­μŒμ•…μ€ μ–΄λ–€κ°€μš”?\", # instruction\n", " \"\", # output - leave this blank for generation!\n", " )\n", "], return_tensors = \"pt\").to(\"cuda\")\n", "\n", "from transformers import TextStreamer\n", "text_streamer = TextStreamer(tokenizer)\n", "_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128)" ] }, { "cell_type": "markdown", "metadata": { "id": "acKgU7NMauCp" }, "source": [ "By using https://translate.google.com/ we get\n", "```\n", "Korean music is classified into many types of music genres.\n", "\n", "This genre is classified into different music genres such as pop songs,\n", "\n", "rock songs, classical songs and pop songs, music groups consisting of drums, fans, instruments and singers\n", "```" ] }, { "cell_type": "markdown", "metadata": { "id": "uMuVrWbjAzhc" }, "source": [ "\n", "### Saving, loading finetuned models\n", "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n", "\n", "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "upcOlWe7A1vc", "outputId": "6ce0bfd6-9fd7-4993-8733-d1f749ac3af3" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.12/dist-packages/peft/utils/save_and_load.py:279: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n", " warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n" ] }, { "data": { "text/plain": [ "('lora_model/tokenizer_config.json',\n", " 'lora_model/special_tokens_map.json',\n", " 'lora_model/chat_template.jinja',\n", " 'lora_model/tokenizer.json')" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.save_pretrained(\"lora_model\") # Local saving\n", "tokenizer.save_pretrained(\"lora_model\")\n", "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n", "# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving" ] }, { "cell_type": "markdown", "metadata": { "id": "AEEcJ4qfC7Lp" }, "source": [ "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "MKX_XKs_BNZR", "outputId": "cccb4070-83d5-42d0-c23e-cfb9dd57b73c" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<|startoftext|>λ‹€μŒμ€ μž‘μ—…μ„ μ„€λͺ…ν•˜λŠ” λͺ…λ Ήμž…λ‹ˆλ‹€. μš”μ²­μ„ μ μ ˆν•˜κ²Œ μ™„λ£Œν•˜λŠ” 응닡을 μž‘μ„±ν•˜μ„Έμš”.\n", "\n", "### 지침:\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "### 응닡:\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\n", "\n", "지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…\n" ] } ], "source": [ "if False:\n", " from unsloth import FastLanguageModel\n", "\n", " model, tokenizer = FastLanguageModel.from_pretrained(\n", " model_name=\"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n", " max_seq_length=max_seq_length,\n", " dtype=dtype,\n", " load_in_4bit=load_in_4bit,\n", " )\n", " FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n", "\n", "# alpaca_prompt = You MUST copy from above!\n", "\n", "inputs = tokenizer(\n", " [\n", " alpaca_prompt.format(\n", " # \"Describe the planet Earth extensively.\", # instruction\n", " \"지ꡬλ₯Ό κ΄‘λ²”μœ„ν•˜κ²Œ μ„€λͺ…ν•˜μ„Έμš”.\",\n", " \"\", # output - leave this blank for generation!\n", " ),\n", " ],\n", " return_tensors=\"pt\",\n", ").to(\"cuda\")\n", "\n", "\n", "from transformers import TextStreamer\n", "\n", "text_streamer = TextStreamer(tokenizer)\n", "_ = model.generate(\n", " **inputs, streamer=text_streamer, max_new_tokens=128, repetition_penalty=0.1\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "twNf4NXhmLqj" }, "source": [ "By using https://translate.google.com/ we get\n", "```\n", "Earth refers to all things including natural disasters such as local derailment\n", "\n", "and local depletion that occur in one space along with the suppression of water, gases, and living things.\n", "\n", "Most of the Earth's water comes from oceans, atmospheric water, underground water layers, and rivers and rivers.\n", "```\n", "\n", "Yikes the language model is a bit whacky! Change the temperature and using sampling will definitely make the output much better!" ] }, { "cell_type": "markdown", "metadata": { "id": "QQMjaNrjsU5_" }, "source": [ "You can also use Hugging Face's `AutoModelForPeftCausalLM`. Only use this if you do not have `unsloth` installed. It can be hopelessly slow, since `4bit` model downloading is not supported, and Unsloth's **inference is 2x faster**." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "yFfaXG0WsQuE" }, "outputs": [], "source": [ "if False:\n", " # I highly do NOT suggest - use Unsloth if possible\n", " from peft import AutoPeftModelForCausalLM\n", " from transformers import AutoTokenizer\n", "\n", " model = AutoPeftModelForCausalLM.from_pretrained(\n", " \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n", " load_in_4bit=load_in_4bit,\n", " )\n", " tokenizer = AutoTokenizer.from_pretrained(\"lora_model\")" ] }, { "cell_type": "markdown", "metadata": { "id": "f422JgM9sdVT" }, "source": [ "### Saving to float16 for VLLM\n", "\n", "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "iHjt_SMYsd3P" }, "outputs": [], "source": [ "# Merge to 16bit\n", "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\n", "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_16bit\", token = \"\")\n", "\n", "# Merge to 4bit\n", "if False: model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_4bit\",)\n", "if False: model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_4bit\", token = \"\")\n", "\n", "# Just LoRA adapters\n", "if False:\n", " model.save_pretrained(\"model\")\n", " tokenizer.save_pretrained(\"model\")\n", "if False:\n", " model.push_to_hub(\"hf/model\", token = \"\")\n", " tokenizer.push_to_hub(\"hf/model\", token = \"\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "TCv4vXHd61i7" }, "source": [ "### GGUF / llama.cpp Conversion\n", "To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n", "\n", "Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n", "* `q8_0` - Fast conversion. High resource use, but generally acceptable.\n", "* `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n", "* `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "FqfebeAdT073" }, "outputs": [], "source": [ "# Save to 8bit Q8_0\n", "if False: model.save_pretrained_gguf(\"model\", tokenizer,)\n", "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, token = \"\")\n", "\n", "# Save to 16bit GGUF\n", "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"f16\")\n", "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"f16\", token = \"\")\n", "\n", "# Save to q4_k_m GGUF\n", "if False: model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"q4_k_m\")\n", "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q4_k_m\", token = \"\")\n", "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q5_k_m\", token = \"\")" ] }, { "cell_type": "markdown", "metadata": { "id": "gCB7r0i2_3sS" }, "source": [ "Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in llama.cpp.\n", "\n", "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n", "\n", "Some other links:\n", "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n", "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n", "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n", "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n", "\n", "

\n", " \n", " \n", " \n", "\n", " Join Discord if you need help + ⭐️ Star us on Github ⭐️\n", "\n", " This notebook and all Unsloth notebooks are licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme).\n", "
\n" ] } ], "metadata": { "accelerator": "GPU", "colab": { "cell_execution_strategy": "setup", "gpuType": "L4", "machine_shape": "hm", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }