{ "cells": [ { "cell_type": "markdown", "id": "a3915efd", "metadata": { "id": "a3915efd" }, "source": [ "# Fine-Tuning LFM3-VL-2B to Read Receipts into JSON with TRL (SFT)\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Liquid4All/cookbook/blob/main/finetuning/notebooks/lfm3_vl_sft_with_trl.ipynb) [![GitHub](https://img.shields.io/badge/GitHub-Liquid4All%2Fcookbook-181717?logo=github)](https://github.com/Liquid4All/cookbook) [![Discord](https://img.shields.io/discord/1385439864920739850?color=7289da&label=Discord&logo=discord&logoColor=white)](https://discord.com/invite/liquid-ai)\n", "\n", "This notebook fine-tunes [LFM3-VL-2B](https://huggingface.co/LiquidAI/LFM3-VL-2B) to read photos of receipts into clean JSON. We use Supervised Fine-Tuning (SFT) with [TRL](https://github.com/huggingface/trl) and LoRA, and measure field-level accuracy before and after training on receipts the model never saw.\n", "\n", "Extracting structured data from images is a typical on-device task: the photo stays on the phone, and a small model that reliably fills your schema is faster and cheaper than a large general one.\n", "\n", "## ๐ŸŽฏ What you'll find\n", "\n", "- **The task**: a receipt photo goes in, a JSON object that follows a fixed schema comes out\n", "- **Reasoning**: the model first reads the receipt line by line in its reasoning, then writes the JSON\n", "- **LoRA + SFT**: train the language model on the reasoning and the answer, with the vision encoder frozen\n", "- **Evaluation**: valid JSON rate, field accuracy, and exact match, on the held-out test receipts" ] }, { "cell_type": "markdown", "id": "48592b83", "metadata": { "id": "48592b83" }, "source": [ "# ๐Ÿ“ฆ Installation & Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "6c78549c", "metadata": { "tags": [ "install" ], "colab": { "base_uri": "https://localhost:8080/" }, "id": "6c78549c", "outputId": "59802065-9ace-49e2-b356-539494e23ece" }, "outputs": [], "source": [ "!pip install -qqq \"torch==2.14.1\" \"torchvision==0.29.1\" \"torchao==0.18.0\" \"trl==1.16.0\" \"transformers==5.20.2\" \"peft==0.21.2\" \"accelerate==1.15.0\" \"datasets==5.1.0\" \"huggingface-hub==2.3.0\" \"tokenizers==0.23.3\" \"safetensors==0.8.0\" --progress-bar off" ] }, { "cell_type": "code", "execution_count": null, "id": "d7937ec5", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "d7937ec5", "outputId": "6805658d-f77b-4159-d16c-009ef88d5d5c" }, "outputs": [], "source": [ "import json\n", "import os\n", "import torch\n", "import transformers\n", "import trl\n", "import peft\n", "\n", "os.environ[\"WANDB_DISABLED\"] = \"true\"\n", "\n", "print(f\"๐Ÿ“ฆ PyTorch version: {torch.__version__}\")\n", "print(f\"๐Ÿค— Transformers version: {transformers.__version__}\")\n", "print(f\"๐Ÿ“Š TRL version: {trl.__version__}\")\n", "print(f\"๐ŸŽ›๏ธ PEFT version: {peft.__version__}\")\n", "\n", "# LFM3-VL-2B is released in bfloat16. GPUs without bfloat16 support, like the T4, load it in float16 instead: the conversion is exact for almost every weight, and the LoRA adapters train in float32 either way.\n", "use_bf16 = torch.cuda.is_bf16_supported(including_emulation=False)\n", "dtype = torch.bfloat16 if use_bf16 else torch.float16\n", "print(f\"๐Ÿ”ข Precision: {dtype}\")" ] }, { "cell_type": "markdown", "id": "c07f05cc", "metadata": { "id": "c07f05cc" }, "source": [ "# Load the model from Transformers ๐Ÿค—\n", "\n", "LFM3-VL-2B ships its modeling code with the checkpoint, so we load it with `trust_remote_code=True`.\n", "\n", "The processor turns each picture into vision tokens, about one token per 32ร—32 pixels. We limit pictures to 1280ร—1280 pixels (about 1,600 tokens) with the processor's `size` (`longest_edge` and `shortest_edge` are pixel counts): the text stays readable, and training and generation stay fast. We pad on the left, so that batched generation continues each prompt right where it ends." ] }, { "cell_type": "code", "execution_count": null, "id": "65cdae09", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000, "referenced_widgets": [ "12db38f4d2a845e2985696b0a5a8d1ed", "f37b22bc29dd41f99c503fded762818d", "159c93e54bc946bc8d1e9771dfa21e3e", "324bc613b3fd472c8ebf98e6e52e8c66", "0d94ad20675046ec87897806f72cd05d", "5c925ba8233646b9af6d3076ba216bac", "374aedf1064e46bfa148cb957c4c32db", "266e89b6ae3f444cb0ef9f770de2d561", "d6a66503ee1c45b09a9a4e2d6398f066", "71f45989bc9445ddb6151f87156857bf", "5601509956a246509bed0c201d761acf", "01c3fdaa013a454184455ae91b845e84", "40d969c66a6f47e890ef8e6db80c564a", "6eae9d80609d4813a1b4253a489e5f24", "61c7d3aaab654bbfa43ee3840ebbb9ae", "4af83883a05e43eaa3cb42837b1550e7", "ed8c3d9c3d0f4ffdb8be5330144bbef9", "a405b49eb5c44d85b165e3a8df27cd68", "697854aaba304ce491251a5b7ea2fb92", "fcbafa56b4df45f2b46e044b5855a3ec", "759a58c59dd24af89e8bdda8f12333f7", "ece21cac12e54473a1bfa800477c0815", "f4d4b4a2cc0042d4b9d97e3b1ee78924", "024343e457cf4c5bbd06e8da57663a3b", "1aacb5117b204a8896c2e65e5bb38e0b", "f4a4a718b30e48e4a75166a4aef0cb4e", "890730a1382742408f18785da738988a", "debeec0a53474666ba49bb1e515856f5", "f984865f7d944e49bb00ea4d42dabdf9", "5ad35573052c4ee3bc5697e4c680481c", "22a985ee68ba43eca0f0cb2d1b9d25a9", "bb58ed83a4d141ce887b822fc9096698", "7167c43e01fc4e51b3cd252eea66673c", "2cd7e0042c8040f59e8d2e9f62374066", "2c9abeeba59a46d1b7a2c6a19acd4f17", "25f16c3487454055b19a4ca44e774e6d", "af154cf08cdf4ca280b09bd11b2c234f", "10b75b49bc054eb6a19fe17845759e16", "9cbe5b67a7864a31acbcd0caa1d96a01", "5d075cf2170f4935bad06a22362a9a9c", "d39138ae07984e6fb03ec4345ee340b4", "4493f1efa81b427d8c86ddfa50eeb207", "3263a17fb9f840a691020c0248be3d0e", "d7131aaa45724c358a75e46beb7b92be", "12ef7d6ce2aa40589660d3fe6ccf0777", "281757ba202d4541a08f5d2f49d85d9d", "34423b6070594aac83b0fd4805c54632", "bdf768ac75c547c9905c4c23d84ee407", "050cea65fae34a7fb624b9d93233cba7", "991c2649bc034e088498abc22037c6c3", "406bdef8e7d64380853272893ac6b862", "40f78af93ca547a4a352888f414d2466", "b2fa2eec3bf344ae861faf0c21a3de55", "a82926b585274839a148ab96451b0a72", "b438fe2fdc824138b0e91a5d450954c3", "295a2e66884746a087966196bc57b498", "3c43d1be48824fdbbaab9c734282c40d", "4546c27379f045599b079c618cdf5f8e", "222b072ba3904ee4be285ebab088f2af", "151e15b95e8f47e9a76e10554af72963", "9103ae8a02304113bb530033e40b29fc", "d41ab0a8506845ef979d38663c453f29", "14d0d4f86f354e5ea68471dfb5263fcb", "9aa9562e7cd143b083c8e546a778c60e", "31f8f2551f6d461882deb53613f564ff", "3651d676a3db4646ac2c55430701d6b4", "484b7e6813cf450c8692aad850dc9883", "668f37c61a48405b9cdbe80f0ac06bf0", "7546307d67654197a58e9ff1455f6b52", "89adf0424ea24ebb911d0ba5022ffa58", "17e2669b75f7455ea9aca3f38e8046e6", "d4c453fcec2a43fd912914e19a6438f8", "0adf4169375b4873b0e53bea0e139f2d", "8e61b336ca624621a4b69930c42fee0b", "4bd5d1893faa42348435af1069d38b2b", "d9746f5f42484600a8dd8eff09c8a45e", "6b3fc6b89eda49d7aa5b699128d286af", "f2105f3f598540398defdb6725b634c0", "6d739d04389d466dbf0831f30b8168db", "e1f8ce8bf9314676898f8ac050c6b545", "7c11b6f2c2b44e1888e9ce1368dba6b3", "c866f6a610524411826532e0d290b7d0", "51d922d92cce4065903748b25e77bb05", "54424601eaf647e6a12350ad744001f8", "7d267cb2da8a4c7e8fb073c88a920c71", "1675f647a9ae44a6b24514e4f4201b68", "84a4a6da6d164cda898ffd6c49bf475f", "c993ac99648e4e488687e04b239fa3e6", "49fc3a32da164ce9a7edd69385d5197c", "356aac9af0a04afeafd67859a32d8fad", "8f633e97c4864aa2b21cc61fe921ccf6", "8cdec23684c6478a9a1e4f6fe6d89b3d", "d0c1fd13dddb4d3cbac5e61812e03f6a", "7269510f5dc4457ea8c2f1c94b426967", "04ffafecf61f4efabda631506d24ef74", "03e76fa1df0c4e798cd495269845024d", "8853d8c3a0354e999053407b6b252f1a", "805665421cee44988ba26da47983a853", "6bb111074813413daa9d43616ecb5e55", "a76f9b6909f84a22b9f7385faa7f61dd", "70f5757568294dc484c497f447bd23d6", "21117630dca849ef92f62c17fac4dfd3", "c0b3b761f9554663a36c4cb9bcd2efcc", "025266b80a1744659d5b22850101b797", "ad089df8b7b0488faee1631e2093f186", "d8d2e8d69a34439c9f034bb34fee9282", "68c6e78048494be5ad77756eb5f7581d", "ea9f0787a95e4d7f857a145b47a2719d", "cda8dda9028d4331942d460f013c5370", "0563ddfb460940cca2b795dd53c355d1", "5b36fd78443746e6b7eb29ffa02210be", "5481c2c3b4cf4e1589d8db8b706f231e", "b7dd199f3a714eb69df91bd3294fde9d", "476029436b1346779177d1d0e536a9c6", "8b337a607c51461aabeb7fbbe4e663a8", "fde17863387743dd9df36442a3b93148", "9834f4592b694b92ab0dab7b5746a41f", "5a3912e9f2604e6eb8526926108838df", "5c9399790fce45f48f02fc65d55c671c", "ac8201a8efa84db7970f6099aa9b4e6f", "007e82073f384c5cadbd1cdf27da7c1e", "25d7f9d25a88462fa8227413c7c065aa", "50051b547b054c589e96ca7b66e6e41d", "1a68ad30e70e417d86617a94ce1a818a", "66c2e247164a4148bdec6d8ffa210afb", "334a5f04fc3142fe810b40ef12f5190b", "db3783ea6f254f31a950aa25aa4c9a5f", "1a2b465d8f0b4618b21619437739caa8", "acc9e468373a46b4b9e7fdf58d87991e", "95e4cd35f71b4d0494f3df142d4f5c65", "b3598821c77449859b193bb54b9f2204", "114e563fafb24684957aa2f1fab0425e", "c1960b4327584bb28196e8a85edc8c8b", "395f4a25d6ff4550845c3f98f52a8cef", "835820a57959445dab2a2f283fe329e7", "a1b53605ed02486bb83fcedd8526b329", "ca727f568f9b411191bdc23a4074b2ac", "c454827669d2407d9d0a986f0a7106fa", "b9bdbe023f654421a4b75706361dd4db", "ce6991115f8e45cf91d9eaf858ecaa1e", "b7e8f417e8e246af932560ff1384df15", "5d3affdedc944e60ae9a614a15acbfea", "c2e18a836ac642259990fc96df5a7373", "d200ba80433d4ab7933e53ff8fcf7737", "97afd22cf3fb448494d70146fbee84e4", "df1e61d0b8cf4aea9af8435b369daa55", "0a5ffeba738d438d87a4965e98c8b0ac", "fbf4cca566294ee2ad6b1be147d24527", "0bbe1990fb2349dd9c34c58aa60817c8", "afcb74266cdb42489cf421e0b0787b95", "cc10b8404f364f8c96bd2e13c8e571bc", "39a38f387ca94d20a83fb4a9f0db01e8", "4a41e8644c10452aaef9d7d6d63dfe89", "b20a8e039c9f4c86a8617be7d20f3b7b", "6f1d3138595e45c69aed07080191ac46", "c4ddcb9a2b6d4e95923b343bb277ef1f", "c19051455fa542c2bb6c45500631e216", "6e277365de13400e9c37d3af5cf4dcee", "987a8d64ae694c9aa7ee6b205cd0fe41", "4035ab64590b4ea3925af0b018dc4ca3", "c03c0ab3383344eebd8be3f89d0644ef", "4ed9c872378b41da84e06abd40e87e16", "b28e2b2d568c485b9f1b9dd4773464e9", "6b66ad980efa4ad4a86ffca93fb3596b", "9b6c8cba0d684892accba4a46062bc8c", "01e91b975c664dea8454d709e2890aaf", "0825b4e47e5649029871797c4af47f33", "173d1f40c2d7442f83468e4be3f7c7d7", "320390ab406545218526ac419ca336bc", "352b50e27bc24465a963a39ebb06fcd0", "6deb63326f934d98bc4500f5254653fb", "4ca7c74375ac4050a0d6d82544786b81", "8bbe1b0425b8410e845c1e228c85fca2", "161fe14277994b8c88da0e6fb637c400", "712dba60d82e4fe6b340fd4d88213c35", "059af47108db48a39621636c68ad0bf5", "5173377baa4e493ea82d4debf5d47625", "b905d2e7c7c044889510549559b4c0ff", "5d3da9b667d2455899e59e4f943719bf", "f4250f6780374486960ebf9c83699d9e", "0bcc56f1bc734ecfa967b6f0ae170aa6", "b8dd70abadc04f8d9aad1c2511a4ab99", "bfb2616f1f4f4f45a21d105ced96a2d3", "2ce79897d49142bb8fcc21b2e0733ad1", "b46b8bb5ecba46c7b66d50af62c5f876", "79eeb194556842aa9e658bc122287312", "912c330848fc4db6aec0b9c61792d8ec", "7dfcaa8bd3af48cd8a4f6f37e441c97c", "de559647f0234db5a251206c870907a4", "ff4a9fb7b4e64d2aa3fc2868a9c5d06b", "8aea384eec4c491bb2f589629363e0b4", "a2afa0144e784d3ca335c4ffc234db0e", "36c30c435d1d44b1bb8c45d98c146a88", "5ba303e1196a44baba3c4063a906d702", "fbe0f490b5ed472e8c95da3d9b7bb8d5", "e594f963a8c540129ed8a98d136cd01c", "6531d82b41934b29b21cad1aef6e9e9d", "ba525e64a28f41e783f04f397c2fe6fc" ] }, "id": "65cdae09", "outputId": "a3013e18-fea6-4dad-c2ca-9da59c6eee41" }, "outputs": [], "source": [ "from transformers import AutoModelForImageTextToText, AutoProcessor\n", "\n", "model_id = \"LiquidAI/LFM3-VL-2B\"\n", "\n", "print(\"๐Ÿ“š Loading processor...\")\n", "processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True, size={\"shortest_edge\": 256 * 256, \"longest_edge\": 1280 * 1280})\n", "processor.tokenizer.padding_side = \"left\"\n", "\n", "print(\"๐Ÿง  Loading model...\")\n", "model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=dtype, device_map=\"auto\", trust_remote_code=True)\n", "\n", "print(\"โœ… Model loaded successfully!\")\n", "print(f\"๐Ÿ”ข Parameters: {model.num_parameters():,}\")" ] }, { "cell_type": "markdown", "id": "a5464666", "metadata": { "id": "a5464666" }, "source": [ "# ๐Ÿ“š Load the receipt dataset\n", "\n", "[CORD](https://huggingface.co/datasets/naver-clova-ix/cord-v2) holds 1,000 photos of receipts from shops and restaurants in Indonesia, with the printed text of each line and its meaning (item name, count, price, total, ...). It has three splits: `train` (800), `validation` (100) and `test` (100)." ] }, { "cell_type": "code", "execution_count": null, "id": "311b5c43", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000, "referenced_widgets": [ "775f33da0f2e4b6e97c73fc263a639a1", "77e6c56d2bb14b7a886676bf710ee9b6", "9f762b7b7f4c401f8c267b6b7848775b", "3c09e3bfcf974e159cbe1963c9de7ae3", "e2186afd9b7543a281a936f6591746d1", "e4136ce7c558402ea75224454687c25b", "623987a704964f0797f2ee54a6327c83", "a9d61d9187c24d4aa9c59be8306eafed", "e7676382dd4742398b3c9a9afbeabb63", "88babddfed8242f5b76995dcfec6a6d7", "97fb9c6ec9c54777a19d305922886cee", "8569e913963d4d76bbe2322878f57f37", "ac239e3065914cf2820b15ddba773b02", "a9c19811017d45b29f30d35a7db8145c", "c5860b5c8d9548c0a2c0a253dcd9b80f", "23a1a4c1f1a643a6bbe9de113d1af29d", "3f79174a97c048be932309ea8b9db40d", "667870f2de464360b5ad784a84b97773", "325822defcb44ee5afadfe5529a83c77", "ebbcf8138c8b4af98429f00d1611c360", "acfafab21e064c269eddd281768c4bfb", "f9740e04caff4bcf93a5afbfa91cc782", "03d2581053424d7db19ae044157878c3", "5c87ecbeaf7c485480892e23ec3d92d8", "3d9f4d8fca4440b4a676b1439bfd6aa9", "bb0e42e7e6dd4c448fd31e8148605637", "2f891f9c2a2e4d8b843b43e96dcf708f", "1b2ba4d9bace4c12ac40d573bf57f491", "c816ee3fa6344d28b67b08c4c3b3344e", "3f0cfcc24d2448b68e2737039c5cfefc", "1d33942a938145de9e9eef29db3084e8", "2d9cafe071a149f68bf0d9c701e2b1fa", "9deb291870da458a9cb7a0ca849cb818", "d7e7f6d7b7224d99a06365f9b6aeaee4", "7d0d681cea6c47bbb77b0230b4c15b77", "44318acac6864216a497422021db0c9b", "8f34beed5bb548ceb7a0a62ee7f06df7", "9d092e0ee94942d597a4ad2367f2048d", "986dc977d8e54df2b9fd7704952c9e72", "9d83684777ee45a5941ff79d264c9a0f", "42094eb3e90b482589972dc2e2b7fb2b", "223d85f8f82440e781e004408446f4ba", "6d60b3dbf9b84e428b0d58109167f43b", "87acd4c7730a4cf69023d34d8c7365c5", "ac21ca4e282648bb8a22f785b8b258f8", "6305bae28f154a29af2add61964eb0ee", "c7c6a7ab0c0a4579b0c82901595b507a", "b110fdd72c5d418f849c0716cf49342a", "3a2fa15624e24ad78e7628b5cfa075cd", "6ac877257c75451abcac1bb61e4f1706", "519bf130b64b4ba89159bfd92b633533", "5e22f6f670de4502b30633513e4e99cf", "762d93b69741424e823c70e1ffa0cdd0", "ac54318de50d4e9887d0aab7d76ad2d0", "2ee5918443f3440aae2e6414caa976b4", "f492c0018e95402685e6e140c6d537d5", "a898465b3968469fbe48f53535426ead", "fdf28e5603ac4ec4b1681c5b48a13df2", "f87469a28b4c4b12bf6c7edd05a4301c", "61521345e4484efdbd21334ad34c7cec", "663facf1f16746889d7cdad9fcd7e5ac", "9a4422c897b242339ffb6190b8ea148f", "7ff063c4a5a94eb988ab6048187425a9", "2fcabcb51c964d05b20be1655d38f7c1", "1c8981c36172405f9c1f2879f4bc5961", "2fbc353d65394fd78d0c0160039d992a", "3fc490e15e224bc889879f67bb4dbb24", "e6ac4dcaa403438baaa470f19a32618f", "bb0e9628f5a44e51951baeda4edef53b", "8ab1c02adb274f3d804b2e193c46e2e6", "9c8a86d6746143f5973919a5cd01084e", "d795ade02ca04128b1be16db7c735429", "144fa8132ee349c0b4c7ea5f291cb147", "20464ea3593e4b65bffc1a17e95537a1", "00a5a7f3a20540b8887940b1c163fc28", "95a5608f661248cea5141270d5fb0b3a", "57025bb4ef6041329fc852ab8de7f100", "b052d5b68e3945348b5dafae030a92b0", "25aa1c9aa1bd4ebbab8b95f16dadee39", "a3581d0d1cfd4b83989789550db13faa", "a94bca66da4246eea22cabfccfaa1b56", "7710266bc6df4816bf8118b790289f93", "98818569b8c0471c97b02f04f5a582b0", "2efa6a48a95545ef9d6bfa2bbe9a355e", "ce3f4610678a4953b3a9714b86b3217b", "332c2a02e4ed4dc9a49d4f8c8535593b", "b317e39c32ac48ac810855bab2506c86", "8b85ba8f63ec4cf9a72e145475a27a90", "738f07c40ec04c298124569a32179575", "2177eb2a75984c9ca5f92f57bbc21b50", "bb62ccc07962497e822f892c29992687", "b9b184757d184016b15f93ab4eb1114d", "4f39676fc1984a7d898e0c310e27fe27", "b89021e05ccc444bafe21f41190a522f", "d956031e8195496aac77c13c69d3cda2", "4300c5df83994216b351fc41ba42053d", "54223ea8c3d04824bc78f37b288c7398", "957c650b711540aaa74f7280b0a921d8", "7a20b2ccb07f4de7961acbf236e24436", "3419c45c4a444105846bed409d97f7ea", "e2acf01b67ba4899bf2f14fbe45fe49f", "452c8246fe5d4c12a442c423a8ab187d", "944fb1c0072c4ed98bb1db4f714fbffa", "69aa040c5c454764935f446c89edd32d", "b6abf5a3421c443b8498d968dd66b930", "1015d432458f434599e48d0d67e451fc", "19ac82451e8b4bc7b35b41bfc23e3c20", "668ac9eebcce468eace119d9b001e410", "ecd8849a8f08448487cef43f90f6e142", "f413e2a06d844464b6007ef283e2a73c", "32045a156e20479dae64fe3b271695bb", "c9abcaa7998c4020a8e75fc59614c504", "bf224cde0c94438c89314d5a1768a86a", "56a7a0323ca24b6394f81165889029e4", "9ceb27d327494541922ed199830a3154", "d04bd979849b45e0af122361069cab4f", "41aa4158102b41daa1e4356e6a017def", "fdcab05d80f04854a2b5c13027fdc32a", "944170fd44ac45659a6b7caab9bb13ca", "55ddd996fda34313b70c196c7e367fbb", "ef56cf3ad6f9454b82cad372b8b5c49a", "ceb2919804a84917a55882bfc0b4d150", "16806f386d5c46d0be9833eb617e21a6", "5fb35102c0b9411d80313f58ad7fad1f", "4e353efecc244eeab30775910ce682ff", "aaab02fcd9374fd2ac27763b8e06d2e8", "8cb327a5567342f3b647de8719fc5a17", "f2a8f2411875458280a904e535a5b436", "8048281829b34a718dd9e7bca094b356", "4ce51c74dd624362b4fdf7e0f6d448cf", "6d0dc4bc9f6a42ed91d7e954f336492d", "dfa62ba2528d458bb1b6b311365d583b", "31cdc40784d9482c953524d219530ac4", "ae18cbedcc3344b8b0d7f8e6aaee08eb", "bdf4e30678b24229a110012d53d1b2b2", "efc46c15d5b84a1392827ea28944f5b6", "51aa1877fc454d8da3ee582e72af8930", "cb79b27b4f4c453b9375ae11579a6df3", "74327aab29ae479aa97a1f1c1efc411c", "079eb19b24214b00aee6d7cc60cdbe9a", "35b49bd0b8fb4c00aa00428b55154a41", "6f3203be69384d08b7836d20c50642d4", "6c2030dda70a464bae349f2c82509394", "5fb4cea78ba34e2e95919191b080d822", "b32243c282b04fe8b7d33877b12e3f4c", "9c3d03fc7b6040acb10d910a33d4b888", "54da8b1d230a41aeaa3109d44f7e8820", "ead22f0974584ea8ab5ab03335899b81", "fbb6f2290a0a415881bacd3433d8cbb1", "ae770958e2d54788a8c2d283c9fc856c", "7c947b5c6b1d46b787b6964d2ff05d59", "59fd8717fc774082b36cc4cc289b2b17", "5aa50c9747614442b1547efc2213e43c", "d0abae1869d8456e80d03df9111695cc", "edeb43f41e7a4fdd9cb5d7a3235168c2", "956ab0c04c784f03be7c8f3675f2452b", "49c501834b324d89a8a158b4ecec5e32", "acf2a4e929644679a9314ce51d59d4a1", "a27a89340fab42bf8979647ceff8a45b", "0e5474105ac34b339b3d06b84c3139b8", "ce1e2f18b6d94ffc9da1b31bb5984a29", "e9cb4ecdc9eb4eb4b4a6127c8fa1bd4d", "623966bb047042f5bdc3f7b77e6d36c8", "d354600f037843c2b468d03d060fffb5", "74e33652a434419789942e926669d662", "9c4040f22a1049f7b050dfc5d2f09d64", "afce60ddf8c34caf990e851681a0fd0e", "ca0ff1ac413b437b8669441409eaa89e", "60f8eed99a4b4145a46d72480ed15ca7", "42d7b5c76f30476eb45110b7277d5114", "1eef9cd8de9941be9b23d0bb94bd08c1", "8c91635f493b4c29863f27d6e5f2477d", "eb20bba081a54b8dbbdafbb814cdd5f7", "2c73c1e6983f405582c88606d6b6815f", "419d0948b72d49b0b989986774248322", "38749f9284484ef7bc9ae4be8b73eff1", "d8cd8e0ab0ce45aebfd043ea0baad4da", "b9f48c8c4a434de8ad78f0bad04a9352", "5ef1c8ac90574520ade28792653dd7d9", "8d11f3bbd2ed48178003492e402cee18", "83de1cf346ce4704a7c7b3df2898c26b", "7065de35585041d9b8a61b62e90637e6", "9748b6f5e76045f9b50584fc5cdb41fa", "9a8fddfc4de34a9eaa2b608fbb894611", "0c8d3e64a609447bb98e3d571a859826", "372d550d688942b385b4c501d23179d4", "f83651bd5515459885fdf424f1cf6226" ] }, "id": "311b5c43", "outputId": "253b2671-8f18-4605-b893-d887bcbfb4a5" }, "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "dataset = load_dataset(\"naver-clova-ix/cord-v2\")\n", "print(dataset)\n", "\n", "example = dataset[\"train\"][1]\n", "print(f\"\\n๐Ÿงพ Picture: {example['image'].size[0]}ร—{example['image'].size[1]} pixels\")\n", "print(f\"\\n๐ŸŽฏ Labels:\\n{json.dumps(json.loads(example['ground_truth'])['gt_parse'], indent=2, ensure_ascii=False)[:800]}\")\n", "example[\"image\"].resize((example[\"image\"].width * 500 // example[\"image\"].height, 500))" ] }, { "cell_type": "markdown", "id": "dab8d247", "metadata": { "id": "dab8d247" }, "source": [ "## Define the schema\n", "\n", "We turn CORD's labels into a fixed schema: the purchased items in printed order, then the totals and the payment. Every value is copied exactly as printed (`\"24,000\"`, `\"2 x\"`), and a field that is not on the receipt is `null`. Item modifiers printed under an item (a size, a topping, the sugar level) are left out.\n", "\n", "A few receipts have two values for one field (for example two cash payments). They don't fit the schema, so we drop them." ] }, { "cell_type": "code", "execution_count": null, "id": "bd8a0a86", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "bd8a0a86", "outputId": "a337f978-aa1c-47a8-efa9-5a2905b49950" }, "outputs": [], "source": [ "FIELDS = { # schema key -> CORD label\n", " \"subtotal\": (\"sub_total\", \"subtotal_price\"),\n", " \"discount\": (\"sub_total\", \"discount_price\"),\n", " \"service\": (\"sub_total\", \"service_price\"),\n", " \"tax\": (\"sub_total\", \"tax_price\"),\n", " \"total\": (\"total\", \"total_price\"),\n", " \"cash\": (\"total\", \"cashprice\"),\n", " \"change\": (\"total\", \"changeprice\"),\n", " \"card\": (\"total\", \"creditcardprice\"),\n", "}\n", "\n", "def to_schema(labels):\n", " \"\"\"The extraction of one receipt in our schema, or None when a field has several values.\"\"\"\n", " menu = labels.get(\"menu\", [])\n", " menu = menu if isinstance(menu, list) else [menu]\n", " items = [{\"name\": item.get(\"nm\"), \"quantity\": item.get(\"cnt\"), \"price\": item.get(\"price\")} for item in menu]\n", " totals = {key: labels.get(group, {}).get(label) if isinstance(labels.get(group), dict) else None for key, (group, label) in FIELDS.items()}\n", " extraction = {\"items\": items, **totals}\n", " values = [v for item in items for v in item.values()] + list(totals.values())\n", " if not all(v is None or isinstance(v, str) for v in values) or any(item[\"name\"] is None for item in items):\n", " return None\n", " return extraction\n", "\n", "print(json.dumps(to_schema(json.loads(example[\"ground_truth\"])[\"gt_parse\"]), indent=2, ensure_ascii=False))" ] }, { "cell_type": "markdown", "id": "0bcbd0dd", "metadata": { "id": "0bcbd0dd" }, "source": [ "## Write the reasoning\n", "\n", "LFM3-VL-2B thinks before it answers, and we keep it that way. Asked about a receipt, the original model already starts its reasoning by reading the lines of the receipt. CORD gives us the printed text of every line, so we turn it into a short reasoning in the same spirit: the lines of the receipt, top to bottom. The model learns to read first, then fill the schema from what it read, and its reasoning stays short." ] }, { "cell_type": "code", "execution_count": null, "id": "492cb6c1", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "492cb6c1", "outputId": "eaadd96f-df53-4cdc-9f44-c0cd4a20302e" }, "outputs": [], "source": [ "from collections import defaultdict\n", "\n", "def receipt_lines(ground_truth):\n", " \"\"\"The printed lines of a receipt, top to bottom, each with its words left to right.\"\"\"\n", " rows = defaultdict(list)\n", " for line in ground_truth[\"valid_line\"]:\n", " for word in line[\"words\"]:\n", " quad = word[\"quad\"]\n", " rows[word[\"row_id\"]].append((min(quad[\"x1\"], quad[\"x4\"]), (quad[\"y1\"] + quad[\"y3\"]) / 2, word[\"text\"]))\n", " rows = sorted(rows.values(), key=lambda words: sum(y for _, y, _ in words) / len(words))\n", " return [\" \".join(text for _, _, text in sorted(words)) for words in rows]\n", "\n", "def reasoning(ground_truth):\n", " return \"The receipt reads, line by line:\\n\" + \"\\n\".join(receipt_lines(ground_truth)) + \"\\n\\nNow I fill the schema from these lines.\"\n", "\n", "print(reasoning(json.loads(example[\"ground_truth\"])))" ] }, { "cell_type": "markdown", "id": "3544254d", "metadata": { "id": "3544254d" }, "source": [ "## Build the conversations\n", "\n", "The system prompt holds the schema and the extraction rules, and the user message holds the photo. In the training conversations, the assistant message holds the reasoning (in `reasoning_content`) and the JSON answer, on one line. The chat template writes the reasoning between `` and ``.\n", "\n", "The conversations are built from the labels only. The photos stay in their `image` column, and TRL puts each photo in front of the user text. This way `map` does not decode and write again 770 large photos, which would need more RAM than a free Colab runtime has." ] }, { "cell_type": "code", "execution_count": null, "id": "ce999ad9", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 272, "referenced_widgets": [ "341abe1822c14d259624ca0c841ba4a3", "12542dc2030d4a5aaa96a31ba67747fd", "c09d190fdba348bd96c2e3727aa38ef9", "a04153a9300149b29b458f8b6f718e30", "9341579086a44aeaa2b854178f1072d2", "779b2417c21349dabffd40478f12135b", "5fb54a5e6a5e4cac9401cafbedf353e6", "3a22842ab60843fc82f11988d3d91c8f", "dfc22879423e4213a8c271c2a8ba0bcd", "af764ad801fa43ecb62eb41d480ab58f", "1505fa57554f40d8a3a3d286ef35bbed", "4d24cfe8e08a41fc8c75e78e33fbbcc2", "5f6fb8a5c7c846c0b193495118cea0ef", "ad32d5af0d2742f19228688da964aab6", "0dc59ae2f7f64b9696899c09f82450c4", "799ac3451c394838a6c800b9b7b9a4df", "cce1e61e9b4543d8905b2af6eab28b84", "8e38cbfdd57b4cf28e940c659bb09c6c", "e4dc776863244ecfadd2dd4944dcfd39", "d16949c150b14c1485dba1469ac7f807", "ed61fa95526f4553934b3424aee59a94", "4931f554d3f64f34803da22292b53d9a", "27d320e9d52145eaa28496034a83304c", "37959e4221ef456299b80fc54668954e", "3e9f2aa849b847b2a94e53cba239d120", "a154a8f4e3ee4586be8b2b45b51375e7", "c40bd55270444aa49f2127738d229420", "7326e2dbea784aad97dbeafbe25ca6e5", "08cad2e30249442e96a61f5f3fd4e504", "d1b5588ba976480685f6d050a1b78104", "c57c4c56796141cbaa50bba9e722fcb7", "a794ceae545b46bc9ecbf4b5847f6ba0", "231acde3c44043049b9505f759bcf1b9", "0c59fdc6b32f4c69af3b4e7398a872de", "c7a5e24b1624434b9d54dc8e4a6dec3c", "cfc3f90cae0c471ebc7c9f2d69378452", "fb7d0dd2ec574269816a925e8ecf8056", "f03a92711d364a37bc4c3015cf644c91", "7108feeb76a24cef9a1c0ed15527db02", "ea853dfd445246fbbb68ff3063f7ecbc", "ac05af50b00b4585a2daa465524d4678", "76f33622a7654314b0cfaf2508a93ebc", "026e3a84e1c8447eb7ecb3292d4bc20a", "2b4cf35070064a289cc8d5b4965fda84" ] }, "id": "ce999ad9", "outputId": "4e6222b4-8813-45d8-f73a-8c324ec5e325" }, "outputs": [], "source": [ "SYSTEM_PROMPT = \"\"\"Extract the receipt in the picture into a JSON object with exactly these keys, in this order:\n", "\n", "{\"items\": [{\"name\": string, \"quantity\": string or null, \"price\": string or null}], \"subtotal\": string or null, \"discount\": string or null, \"service\": string or null, \"tax\": string or null, \"total\": string or null, \"cash\": string or null, \"change\": string or null, \"card\": string or null}\n", "\n", "Rules:\n", "1. Output exactly one JSON object, on one line. No prose, no code fences.\n", "2. Copy every value exactly as printed: same digits, separators, symbols and spelling. Do not convert or compute values.\n", "3. `items` lists the purchased items in printed order. Leave out modifiers printed under an item (size, topping, sugar level).\n", "4. `quantity` is the count printed for the item, `price` the amount printed on the item's line.\n", "5. `subtotal` is the amount before tax and service, `discount` the total discount, `service` the service charge, `tax` the tax, `total` the amount to pay, `cash` the cash given, `change` the change returned, `card` the amount paid by card.\n", "6. A field that is not printed on the receipt is null.\"\"\"\n", "\n", "def build_messages(image):\n", " return [\n", " {\"role\": \"system\", \"content\": [{\"type\": \"text\", \"text\": SYSTEM_PROMPT}]},\n", " {\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": image}, {\"type\": \"text\", \"text\": \"Extract this receipt.\"}]},\n", " ]\n", "\n", "def to_conversation(ground_truth):\n", " ground_truth = json.loads(ground_truth)\n", " extraction = to_schema(ground_truth[\"gt_parse\"])\n", " return {\n", " \"messages\": [\n", " {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n", " {\"role\": \"user\", \"content\": \"Extract this receipt.\"}, # TRL puts the picture before this text\n", " {\"role\": \"assistant\", \"content\": json.dumps(extraction, ensure_ascii=False), \"reasoning_content\": reasoning(ground_truth)},\n", " ],\n", " }\n", "\n", "def keep(ground_truth):\n", " return to_schema(json.loads(ground_truth)[\"gt_parse\"]) is not None\n", "\n", "dataset = dataset.filter(keep, input_columns=\"ground_truth\")\n", "train_dataset = dataset[\"train\"].map(to_conversation, input_columns=\"ground_truth\", remove_columns=\"ground_truth\")\n", "print(f\"๐Ÿงพ {len(train_dataset)} training receipts, {len(dataset['test'])} test receipts\")\n", "print(json.dumps(train_dataset[1][\"messages\"][2], indent=2, ensure_ascii=False)[:1200])" ] }, { "cell_type": "markdown", "id": "fc279dc2", "metadata": { "id": "fc279dc2" }, "source": [ "# ๐Ÿ“ Evaluate the model before training\n", "\n", "We generate an extraction for each test receipt, with the sampling settings recommended in the model card (they are the checkpoint's generation defaults), and compare it with the gold JSON:\n", "\n", "- **Valid JSON**: the answer after the reasoning parses as a JSON object (we tolerate a Markdown code fence around it, even though the rules ask for none).\n", "- **Field accuracy**: the share of fields with the right value. CORD's labels space words inconsistently (`\"JASMINE MT ( L )\"`), so strings are compared without case and spaces. `null` must stay `null`.\n", "- **Exact match**: every field is right.\n", "- **Output tokens**: the average length of the reasoning and the answer, which sets the latency on a phone.\n", "\n", "The original model has never seen our schema, so it gets the same instructions as the fine-tuned one." ] }, { "cell_type": "code", "source": [ "model.generation_config" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "OHvilXwN-7vc", "outputId": "821ebfdb-f526-4f42-c2be-82b0d25902c4" }, "id": "OHvilXwN-7vc", "execution_count": null, "outputs": [] }, { "cell_type": "code", "execution_count": null, "id": "de2ced3a", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 228, "referenced_widgets": [ "84a52bb69e8949498b7a605216e90c44", "9ba6425cb9fe4624a3e38437013af5c1", "663a280100bd46b4bd0f4276ee72000e", "d2c44642652d49518c94adb0806c2263", "dea25e445efd477188870fe6de3e9d70", "b932cda427b841b38ac4d07b31a538e7", "3f6acbd88f32451b9b041a04ec0ce279", "dc85d2cb5bf64222a0ec533fd1037e23", "91fe0f4594af42a396152e03d2139ddd", "8c436f9d7b454997bfb1418d9b61ee3c", "06043546f3104e70b4354b193bb5190c" ] }, "id": "de2ced3a", "outputId": "86fe9773-26b9-4067-f149-26c3b9c036a7" }, "outputs": [], "source": [ "from tqdm.auto import tqdm\n", "\n", "def parse_json(text):\n", " \"\"\"The JSON object in a model's answer, tolerating a Markdown code fence around it, or None.\"\"\"\n", " text = text.strip()\n", " if text.startswith(\"```\"):\n", " text = text.split(\"\\n\", 1)[-1].rsplit(\"```\", 1)[0]\n", " try:\n", " value = json.loads(text)\n", " except json.JSONDecodeError:\n", " return None\n", " return value if isinstance(value, dict) else None\n", "\n", "def flatten(value, path=\"\"):\n", " \"\"\"Leaves of a JSON value as {path: value}, e.g. {\"items[0].price\": \"24,000\"}.\"\"\"\n", " if isinstance(value, dict):\n", " return {k: v for key, child in value.items() for k, v in flatten(child, f\"{path}.{key}\" if path else key).items()}\n", " if isinstance(value, list):\n", " return {k: v for i, child in enumerate(value) for k, v in flatten(child, f\"{path}[{i}]\").items()} or {path: []}\n", " return {path: value}\n", "\n", "def same(a, b):\n", " if isinstance(a, str) and isinstance(b, str):\n", " return \"\".join(a.split()).casefold() == \"\".join(b.split()).casefold()\n", " return a == b\n", "\n", "def score(prediction, gold):\n", " \"\"\"Field accuracy of one prediction: matching fields / fields in the gold or the prediction.\"\"\"\n", " gold_fields, predicted_fields = flatten(gold), flatten(prediction)\n", " paths = set(gold_fields) | set(predicted_fields)\n", " if not paths:\n", " return 1.0\n", " return sum(p in gold_fields and p in predicted_fields and same(gold_fields[p], predicted_fields[p]) for p in paths) / len(paths)\n", "\n", "def answer(text):\n", " \"\"\"The answer after the reasoning, or None when the reasoning never ends.\"\"\"\n", " return text.split(\"\", 1)[1] if \"\" in text else None\n", "\n", "@torch.no_grad()\n", "def generate(model, images, max_new_tokens=4096):\n", " \"\"\"The reasoning and answer of the model for each picture, as text, and their lengths in tokens.\"\"\"\n", " inputs = processor.apply_chat_template(\n", " [build_messages(image) for image in images], add_generation_prompt=True, tokenize=True,\n", " return_dict=True, return_tensors=\"pt\", processor_kwargs={\"padding\": True},\n", " ).to(model.device)\n", " outputs = model.generate(**inputs, max_new_tokens=max_new_tokens)[:, inputs[\"input_ids\"].shape[1]:]\n", " lengths = (outputs != processor.tokenizer.pad_token_id).sum(dim=1).tolist()\n", " return processor.batch_decode(outputs, skip_special_tokens=True), lengths\n", "\n", "def evaluate(model, batch_size=16):\n", " model.eval()\n", " examples = dataset[\"test\"]\n", " valid, accuracy, exact, tokens = 0, 0.0, 0, 0\n", " for start in tqdm(range(0, len(examples), batch_size), desc=\"Evaluating\"):\n", " batch = examples.select(range(start, min(start + batch_size, len(examples))))\n", " texts, lengths = generate(model, batch[\"image\"])\n", " tokens += sum(lengths)\n", " for text, example in zip(texts, batch):\n", " gold = to_schema(json.loads(example[\"ground_truth\"])[\"gt_parse\"])\n", " prediction = parse_json(answer(text) or \"\")\n", " if prediction is None:\n", " continue # invalid JSON scores 0 on every metric\n", " valid += 1\n", " field_accuracy = score(prediction, gold)\n", " accuracy += field_accuracy\n", " exact += field_accuracy == 1.0\n", " n = len(examples)\n", " return {\"valid_json\": valid / n, \"field_accuracy\": accuracy / n, \"exact_match\": exact / n, \"tokens\": tokens / n}\n", "\n", "def show(r, title):\n", " print(f\"{title}: valid JSON {r['valid_json']:.1%} | field accuracy {r['field_accuracy']:.1%} | exact match {r['exact_match']:.1%} | {r['tokens']:.0f} output tokens\")\n", "\n", "torch.manual_seed(0)\n", "baseline = evaluate(model, batch_size=32)\n", "show(baseline, \"๐Ÿ“ Before training\")" ] }, { "cell_type": "markdown", "id": "31a90cdf", "metadata": { "id": "31a90cdf" }, "source": [ "# ๐ŸŽฏ Supervised Fine-Tuning (SFT + LoRA)\n", "\n", "## Wrap the model with PEFT\n", "\n", "In LFM3's language model, we adapt the attention projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`), the gated short convolutions (`in_proj`, `out_proj`), and the MLP (`w1`, `w2`, `w3`). LFM3's attention layers also have a per-head output gate, `gate_proj`; adding it to the targets is a reasonable choice. The vision encoder uses some of the same names, so we select the language model's layers with a regular expression and leave the vision encoder frozen: it already sees receipts well, the model only has to learn what to write." ] }, { "cell_type": "code", "execution_count": null, "id": "492ef134", "metadata": { "id": "492ef134" }, "outputs": [], "source": [ "from peft import LoraConfig\n", "\n", "peft_config = LoraConfig(\n", " r=16,\n", " lora_alpha=32,\n", " lora_dropout=0.05,\n", " # optional: add \"|gate_proj\" to the group for LFM3's per-head attention gate\n", " target_modules=r\".*language_model.*\\.(q_proj|k_proj|v_proj|o_proj|in_proj|out_proj|w1|w2|w3)\",\n", " task_type=\"CAUSAL_LM\",\n", ")" ] }, { "cell_type": "markdown", "id": "c04b4bec", "metadata": { "id": "c04b4bec" }, "source": [ "## Launch training\n", "\n", "With `assistant_only_loss=True`, `SFTTrainer` computes the loss on the assistant message only: the model learns to write the reasoning and the JSON, not to reproduce the system prompt. The chat template marks the assistant message for TRL with `{% generation %}`." ] }, { "cell_type": "code", "execution_count": null, "id": "14836359", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 431 }, "id": "14836359", "outputId": "9b69f909-2760-485e-dfbd-ec8dc2a21661" }, "outputs": [], "source": [ "from trl import SFTConfig, SFTTrainer\n", "\n", "sft_config = SFTConfig(\n", " output_dir=\"./LFM3-VL-2B-receipts\",\n", " num_train_epochs=2,\n", " per_device_train_batch_size=4,\n", " gradient_accumulation_steps=4, # <- effective batch size of 16 receipts\n", " learning_rate=2e-4,\n", " lr_scheduler_type=\"cosine\",\n", " warmup_steps=0.05, # <- a float is a fraction of the total steps\n", " max_length=None, # <- never cut a conversation: the vision tokens of a picture must stay whole\n", " assistant_only_loss=True,\n", " gradient_checkpointing=True,\n", " logging_steps=10,\n", " save_strategy=\"no\",\n", " bf16=use_bf16,\n", " fp16=not use_bf16, # <- float16 mixed precision with loss scaling on GPUs without bfloat16\n", " report_to=\"none\",\n", ")\n", "\n", "trainer = SFTTrainer(\n", " model=model,\n", " args=sft_config,\n", " train_dataset=train_dataset,\n", " processing_class=processor,\n", " peft_config=peft_config,\n", ")\n", "trainer.model.print_trainable_parameters()\n", "\n", "print(\"๐Ÿš€ Starting SFT training...\")\n", "trainer.train()\n", "print(\"๐ŸŽ‰ SFT training completed!\")\n", "print(f\"๐Ÿงฎ Peak GPU memory: {torch.cuda.max_memory_allocated() / 1e9:.1f} GB\")" ] }, { "cell_type": "markdown", "id": "b7a8c293", "metadata": { "id": "b7a8c293" }, "source": [ "# ๐Ÿ“Š Evaluate the fine-tuned model" ] }, { "cell_type": "code", "execution_count": null, "id": "4f2de9d7", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 468, "referenced_widgets": [ "64bac734ea2d4956be96b65556a426ce", "70245eb3b77045fb8612dd01073affc8", "03833bdb550b4af0919ecedef20d7e04", "c093c165fb7c48babcbc784923d55565", "7bd9152891cb41ca814c9147ea07fbe0", "3d2ea03423a54438b93c735438dde4da", "76b6df8f06214871a707a22a5814416f", "211dc3dc43c040a6be551f48272c01e3", "eb93c0fde4cc46a790adaf3e5ef073f7", "dee9c29d47a149f3b2c1d242195d6ff6", "9ff9227293164f5ea7866508047b196b" ] }, "id": "4f2de9d7", "outputId": "ff9af497-a498-431c-97c3-c7f9d5a65b6f" }, "outputs": [], "source": [ "torch.manual_seed(0)\n", "finetuned = evaluate(trainer.model, batch_size=32)\n", "show(baseline, \"๐Ÿ“ Before training\")\n", "show(finetuned, \"๐Ÿ“ After training\")" ] }, { "cell_type": "markdown", "id": "7a7ccace", "metadata": { "id": "7a7ccace" }, "source": [ "# ๐Ÿ” Original vs fine-tuned model\n", "\n", "To see what changed, we run the original model and the fine-tuned model on the same test receipts. With LoRA, we don't need to load the original model again: `disable_adapter()` temporarily turns the adapter off.\n", "\n", "For each field, we show the gold value and both extractions, with โœ… when the value is right." ] }, { "cell_type": "code", "execution_count": null, "id": "f9bbdbe6", "metadata": { "id": "f9bbdbe6" }, "outputs": [], "source": [ "from IPython.display import display\n", "\n", "def cell(fields, path, gold_fields):\n", " if fields is None:\n", " return \"โŒ invalid JSON\"\n", " if path not in fields:\n", " return \"โŒ (missing)\"\n", " mark = \"โœ…\" if path in gold_fields and same(fields[path], gold_fields[path]) else \"โŒ\"\n", " return f\"{mark} {json.dumps(fields[path], ensure_ascii=False)}\"\n", "\n", "trainer.model.eval()\n", "for example in dataset[\"test\"].select([3, 17, 42]):\n", " with trainer.model.disable_adapter():\n", " (original_text,), _ = generate(trainer.model, [example[\"image\"]])\n", " (finetuned_text,), _ = generate(trainer.model, [example[\"image\"]])\n", " gold_fields = flatten(to_schema(json.loads(example[\"ground_truth\"])[\"gt_parse\"]))\n", " original, finetuned_extraction = parse_json(answer(original_text) or \"\"), parse_json(answer(finetuned_text) or \"\")\n", " original_fields = flatten(original) if original is not None else None\n", " finetuned_fields = flatten(finetuned_extraction) if finetuned_extraction is not None else None\n", " display(example[\"image\"].resize((example[\"image\"].width * 400 // example[\"image\"].height, 400)))\n", " print(f\"๐Ÿง  Fine-tuned reasoning:\\n{finetuned_text.split('')[0][:600]}\\n\")\n", " print(f\"{'field':<20} {'gold':<24} {'original':<30} {'fine-tuned'}\")\n", " for path, gold_value in gold_fields.items():\n", " print(f\"{path[:19]:<20} {json.dumps(gold_value, ensure_ascii=False)[:23]:<24} {cell(original_fields, path, gold_fields)[:29]:<30} {cell(finetuned_fields, path, gold_fields)[:36]}\")" ] }, { "cell_type": "markdown", "id": "0ad89551", "metadata": { "id": "0ad89551" }, "source": [ "And the summary over all the test receipts:" ] }, { "cell_type": "code", "execution_count": null, "id": "7d7990b1", "metadata": { "id": "7d7990b1" }, "outputs": [], "source": [ "print(f\"{'':<16}{'original':>10}{'fine-tuned':>12}\")\n", "for metric in [\"valid_json\", \"field_accuracy\", \"exact_match\"]:\n", " print(f\"{metric:<16}{baseline[metric]:>10.1%}{finetuned[metric]:>12.1%}\")\n", "print(f\"{'output tokens':<16}{baseline['tokens']:>10.0f}{finetuned['tokens']:>12.0f}\")" ] }, { "cell_type": "markdown", "id": "3379c379", "metadata": { "id": "3379c379" }, "source": [ "# ๐Ÿ’พ Save and merge the model\n", "\n", "We save the LoRA adapter, then merge it into the base weights. `save_pretrained` also copies LFM3-VL's modeling code next to the weights, so the merged model loads with `trust_remote_code=True` like the original one." ] }, { "cell_type": "code", "execution_count": null, "id": "5aa8c32d", "metadata": { "id": "5aa8c32d" }, "outputs": [], "source": [ "trainer.save_model(\"./LFM3-VL-2B-receipts-lora\")\n", "\n", "output_dir = \"./LFM3-VL-2B-receipts-merged\"\n", "merged_model = trainer.model.merge_and_unload()\n", "merged_model.save_pretrained(output_dir)\n", "processor.save_pretrained(output_dir)\n", "print(f\"๐Ÿ’พ Merged model saved to: {output_dir}\")\n", "\n", "# To push the merged model to the Hugging Face Hub, uncomment:\n", "# merged_model.push_to_hub(\"your-username/LFM3-VL-2B-receipts\")\n", "# processor.push_to_hub(\"your-username/LFM3-VL-2B-receipts\")" ] }, { "cell_type": "markdown", "id": "ed6d42c7", "metadata": { "id": "ed6d42c7" }, "source": [ "# ๐Ÿš€ Next steps\n", "\n", "- **Your own documents**: write a schema and label a few hundred photos. Include photos where fields are missing, so the model learns to return `null` instead of guessing.\n", "- **Your own reasoning**: if your labels have no line-by-line text, generate reasoning with the original model, and keep only the conversations whose answer matches your labels.\n", "- **Constrained decoding**: at inference, a JSON-schema grammar (supported by llama.cpp and vLLM) guarantees valid JSON on top of the fine-tuned accuracy." ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "T4", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }