"
],
"text/html": [
"\n",
" \n",
" \n",
"
\n",
" [30/30 00:55, Epoch 0/1]\n",
"
\n",
" \n",
" \n",
" | Step | \n",
" Training Loss | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" 2.338012 | \n",
"
\n",
" \n",
" | 2 | \n",
" 2.091266 | \n",
"
\n",
" \n",
" | 3 | \n",
" 2.403877 | \n",
"
\n",
" \n",
" | 4 | \n",
" 1.822120 | \n",
"
\n",
" \n",
" | 5 | \n",
" 1.560136 | \n",
"
\n",
" \n",
" | 6 | \n",
" 1.678832 | \n",
"
\n",
" \n",
" | 7 | \n",
" 1.247484 | \n",
"
\n",
" \n",
" | 8 | \n",
" 1.267451 | \n",
"
\n",
" \n",
" | 9 | \n",
" 1.158446 | \n",
"
\n",
" \n",
" | 10 | \n",
" 1.315294 | \n",
"
\n",
" \n",
" | 11 | \n",
" 1.023160 | \n",
"
\n",
" \n",
" | 12 | \n",
" 0.804686 | \n",
"
\n",
" \n",
" | 13 | \n",
" 0.962911 | \n",
"
\n",
" \n",
" | 14 | \n",
" 0.972136 | \n",
"
\n",
" \n",
" | 15 | \n",
" 1.094977 | \n",
"
\n",
" \n",
" | 16 | \n",
" 0.860323 | \n",
"
\n",
" \n",
" | 17 | \n",
" 0.944462 | \n",
"
\n",
" \n",
" | 18 | \n",
" 0.724142 | \n",
"
\n",
" \n",
" | 19 | \n",
" 0.901048 | \n",
"
\n",
" \n",
" | 20 | \n",
" 1.111872 | \n",
"
\n",
" \n",
" | 21 | \n",
" 0.740135 | \n",
"
\n",
" \n",
" | 22 | \n",
" 0.646140 | \n",
"
\n",
" \n",
" | 23 | \n",
" 0.584106 | \n",
"
\n",
" \n",
" | 24 | \n",
" 0.566740 | \n",
"
\n",
" \n",
" | 25 | \n",
" 0.683506 | \n",
"
\n",
" \n",
" | 26 | \n",
" 0.673225 | \n",
"
\n",
" \n",
" | 27 | \n",
" 0.593300 | \n",
"
\n",
" \n",
" | 28 | \n",
" 0.660456 | \n",
"
\n",
" \n",
" | 29 | \n",
" 0.569731 | \n",
"
\n",
" \n",
" | 30 | \n",
" 0.636652 | \n",
"
\n",
" \n",
"
"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stderr",
"text": [
"Unsloth: Restored added_tokens_decoder metadata in outputs/checkpoint-30/tokenizer_config.json.\n"
]
}
],
"source": [
"trainer_stats = trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"cellView": "form",
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "pCqnaKmlO1U9",
"outputId": "fea34061-cab9-4657-85e2-ec21513574e3"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"84.5821 seconds used for training.\n",
"1.41 minutes used for training.\n",
"Peak reserved memory = 6.549 GB.\n",
"Peak reserved memory for training = 0.543 GB.\n",
"Peak reserved memory % of max memory = 29.722 %.\n",
"Peak reserved memory for training % of max memory = 2.464 %.\n"
]
}
],
"source": [
"# @title Show final memory and time stats\n",
"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
"used_percentage = round(used_memory / max_memory * 100, 3)\n",
"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
"print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
"print(\n",
" f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
")\n",
"print(f\"Peak reserved memory = {used_memory} GB.\")\n",
"print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
"print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
"print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ekOmTR1hSNcr"
},
"source": [
"\n",
"### Inference\n",
"Let's run the model! You can change the instruction and input - leave the output blank!\n",
"\n",
"We use `min_p = 0.1` and `temperature = 1.5`. Read this [Tweet](https://x.com/menhguin/status/1826132708508213629) for more information on why.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kR3gIAX-SM2q",
"outputId": "18cf317a-4781-4b54-d84c-aa51dda2df6f"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"$$H ^ { \\prime } = \\beta N \\int d \\lambda \\biggl \\lbrace \\frac { 1 } { 2 \\beta ^ { 2 } N ^ { 2 } } \\partial _ { \\lambda } \\zeta ^ { \\dag } \\partial _ { \\lambda } \\zeta + V ( \\lambda ) \\zeta ^ { \\dag } \\zeta \\ \\biggr \\rbrace ~ .$$<|im_end|>\n"
]
}
],
"source": [
"FastVisionModel.for_inference(model) # Enable for inference!\n",
"\n",
"image = dataset[2][\"image\"]\n",
"instruction = \"Write the LaTeX representation for this image.\"\n",
"\n",
"messages = [\n",
" {\"role\": \"user\", \"content\": [\n",
" {\"type\": \"image\"},\n",
" {\"type\": \"text\", \"text\": instruction}\n",
" ]}\n",
"]\n",
"input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True)\n",
"inputs = tokenizer(\n",
" image,\n",
" input_text,\n",
" add_special_tokens = False,\n",
" return_tensors = \"pt\",\n",
").to(\"cuda\")\n",
"\n",
"from transformers import TextStreamer\n",
"text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
"_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,\n",
" use_cache = True, temperature = 1.5, min_p = 0.1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uMuVrWbjAzhc"
},
"source": [
"\n",
"### Saving, loading finetuned models\n",
"To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
"\n",
"**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "upcOlWe7A1vc",
"outputId": "5d3f3866-a20f-4ccb-c62f-13c40a803b3c"
},
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"Unsloth: Restored added_tokens_decoder metadata in lora_model/tokenizer_config.json.\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['lora_model/processor_config.json']"
]
},
"metadata": {},
"execution_count": 19
}
],
"source": [
"model.save_pretrained(\"lora_model\") # Local saving\n",
"tokenizer.save_pretrained(\"lora_model\")\n",
"# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n",
"# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AEEcJ4qfC7Lp"
},
"source": [
"Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "MKX_XKs_BNZR",
"outputId": "fe0155a8-9c65-45f6-b933-807e88716dca"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\\frac{N}{M}\\in {{\\bf Z}},\\frac{M}{P}\\in {{\\bf Z}},\\frac{P}{Q}\\in {{\\bf Z}}<|im_end|>\n"
]
}
],
"source": [
"if False:\n",
" from unsloth import FastVisionModel\n",
" model, tokenizer = FastVisionModel.from_pretrained(\n",
" model_name = \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
" load_in_4bit = True, # Set to False for 16bit LoRA\n",
" )\n",
" FastVisionModel.for_inference(model) # Enable for inference!\n",
"\n",
"image = dataset[0][\"image\"]\n",
"instruction = \"Write the LaTeX representation for this image.\"\n",
"\n",
"messages = [\n",
" {\"role\": \"user\", \"content\": [\n",
" {\"type\": \"image\"},\n",
" {\"type\": \"text\", \"text\": instruction}\n",
" ]}\n",
"]\n",
"input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True)\n",
"inputs = tokenizer(\n",
" image,\n",
" input_text,\n",
" add_special_tokens = False,\n",
" return_tensors = \"pt\",\n",
").to(\"cuda\")\n",
"\n",
"from transformers import TextStreamer\n",
"text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
"_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128,\n",
" use_cache = True, temperature = 1.5, min_p = 0.1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f422JgM9sdVT"
},
"source": [
"### Saving to float16 for VLLM\n",
"\n",
"We also support saving to `float16` directly. Select `merged_16bit` for float16. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "iHjt_SMYsd3P"
},
"outputs": [],
"source": [
"# Select ONLY 1 to save! (Both not needed!)\n",
"\n",
"# Save locally to 16bit\n",
"if False: model.save_pretrained_merged(\"unsloth_finetune\", tokenizer,)\n",
"\n",
"# To export and save to your Hugging Face account\n",
"if False: model.push_to_hub_merged(\"YOUR_USERNAME/unsloth_finetune\", tokenizer, token = \"PUT_HERE\")"
]
},
{
"cell_type": "markdown",
"source": [
"### GGUF / llama.cpp Conversion\n",
"To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n",
"\n",
"Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n",
"* `q8_0` - Fast conversion. High resource use, but generally acceptable.\n",
"* `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n",
"* `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K.\n",
"\n",
"[**NEW**] To finetune and auto export to Ollama, try our [Ollama notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)"
],
"metadata": {
"id": "qjuPsiqcOYaA"
}
},
{
"cell_type": "code",
"source": [
"# Save to 8bit Q8_0\n",
"if False: model.save_pretrained_gguf(\"unsloth_finetune\", tokenizer,)\n",
"# Remember to go to https://huggingface.co/settings/tokens for a token!\n",
"# And change hf to your username!\n",
"if False: model.push_to_hub_gguf(\"hf/unsloth_finetune\", tokenizer, token = \"\")\n",
"\n",
"# Save to 16bit GGUF\n",
"if False: model.save_pretrained_gguf(\"unsloth_finetune\", tokenizer, quantization_method = \"f16\")\n",
"if False: model.push_to_hub_gguf(\"hf/unsloth_finetune\", tokenizer, quantization_method = \"f16\", token = \"\")\n",
"\n",
"# Save to q4_k_m GGUF\n",
"if False: model.save_pretrained_gguf(\"unsloth_finetune\", tokenizer, quantization_method = \"q4_k_m\")\n",
"if False: model.push_to_hub_gguf(\"hf/unsloth_finetune\", tokenizer, quantization_method = \"q4_k_m\", token = \"\")\n",
"\n",
"# Save to multiple GGUF options - much faster if you want multiple!\n",
"if False:\n",
" model.push_to_hub_gguf(\n",
" \"hf/unsloth_finetune\", # Change hf to your username!\n",
" tokenizer,\n",
" quantization_method = [\"q4_k_m\", \"q8_0\", \"q5_k_m\",],\n",
" token = \"\",\n",
" )"
],
"metadata": {
"id": "At1T2hJnOdGM"
},
"execution_count": 22,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "5nYIaMAqzzeh"
},
"source": [
"Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in llama.cpp.\n",
"\n",
"And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
"\n",
"Some other links:\n",
"1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
"2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
"3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
"6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n",
"\n",
"
\n",
"
![](\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\")
\n",
"
![](\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\")
\n",
"
![](\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\")
\n",
"\n",
" Join Discord if you need help + ⭐️
Star us on Github ⭐️\n",
"\n",
" This notebook and all Unsloth notebooks are licensed [LGPL-3.0](https://github.com/unslothai/notebooks?tab=LGPL-3.0-1-ov-file#readme).\n",
"
![](\"https://github.com/unslothai/unsloth/raw/main/images/Discord)
\n",
"