{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿ’ง LFM3-Omni - Voice commands to function calls with Transformers\n", "\n", "This tutorial demonstrates how to fine-tune [`LiquidAI/LFM3-Omni-2B`](https://huggingface.co/LiquidAI/LFM3-Omni-2B) to map **spoken smart-home\n", "commands directly to function calls**: audio in, function call out, with no separate speech-to-text step.\n", "\n", "```\n", "๐ŸŽ™๏ธ \"turn on the fan in the living room\" โ†’ HassTurnOn|$area=living room|$domain=fan\n", "```\n", "\n", "We use the same task and data as our [LFM2.5-Audio voice assistant example](https://github.com/Liquid4All/cookbook/tree/main/examples/voice-assistant):\n", "[OHF-Voice](https://huggingface.co/datasets/Paulescu/OHF-Voice-audio-20260504), 55k recorded Home Assistant commands\n", "paired with their function call, across 41 functions (timers, lights, climate, media, shopping lists, ...).\n", "\n", "## ๐ŸŽฏ What You'll Find:\n", "- **Data preparation** - (audio, function call) pairs as chat conversations\n", "- **Baseline** - what the model does before fine-tuning\n", "- **LoRA + SFT** - parameter-efficient fine-tuning with the ๐Ÿค— `Trainer`\n", "- **Evaluation** - format compliance, function-name accuracy and exact-match accuracy on a held-out set\n", "\n", "## ๐Ÿ“‹ Prerequisites:\n", "- **GPU Runtime**: Select GPU in `Runtime` โ†’ `Change runtime type` (an A100 is recommended)\n", "- **Hugging Face Account**: For accessing models and datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿ“ฆ Installation & Setup\n", "\n", "First, let's install all the required packages:" ] }, { "cell_type": "code", "metadata": {}, "source": [ "!pip install -qqq \"transformers==5.20.2\" \"torchao==0.18.0\" accelerate datasets peft librosa soundfile --progress-bar off" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now verify the packages are installed correctly" ] }, { "cell_type": "code", "metadata": {}, "source": [ "import os\n", "import torch\n", "import transformers\n", "import peft\n", "os.environ[\"WANDB_DISABLED\"] = \"true\"\n", "\n", "print(f\"๐Ÿ“ฆ PyTorch version: {torch.__version__}\")\n", "print(f\"๐Ÿค— Transformers version: {transformers.__version__}\")\n", "print(f\"๐Ÿงฉ PEFT version: {peft.__version__}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Loading the model from Transformers ๐Ÿค—\n", "\n", "The processor bundles the tokenizer, the image, video and audio preprocessors and the audio codec." ] }, { "cell_type": "code", "metadata": {}, "source": [ "from transformers import AutoModelForMultimodalLM, AutoProcessor\n", "\n", "model_id = \"LiquidAI/LFM3-Omni-2B\"\n", "\n", "print(\"๐Ÿ“š Loading processor...\")\n", "processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)\n", "\n", "print(\"๐Ÿง  Loading model...\")\n", "model = AutoModelForMultimodalLM.from_pretrained(\n", " model_id, dtype=torch.bfloat16, device_map=\"cuda\", trust_remote_code=True\n", ")\n", "\n", "print(\"\\nโœ… Model loaded successfully!\")\n", "print(f\"๐Ÿ”ข Parameters: {model.num_parameters():,}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿ“š Load the dataset\n", "\n", "The full training split has 52k samples in 16 shards of ~470 MB. Two shards (~6,600 samples) are enough to learn the task;\n", "add more for better accuracy on rare functions. For evaluation we draw up to 10 samples per function from the test split\n", "(~400 samples), as in the LFM2.5 example.\n", "\n", "Each sample holds the recording (WAV bytes), the function call, and the transcript of the command (which we only use for display)." ] }, { "cell_type": "code", "metadata": {}, "source": [ "import collections\n", "import io\n", "import random\n", "\n", "import librosa\n", "import soundfile as sf\n", "from datasets import load_dataset\n", "from IPython.display import Audio, display\n", "\n", "TRAIN_SHARDS = 2\n", "dataset_id = \"Paulescu/OHF-Voice-audio-20260504\"\n", "train_files = [f\"data/train-{i:05d}-of-00016.parquet\" for i in range(TRAIN_SHARDS)]\n", "# verification_mode=\"no_checks\": we load a subset of the files, which doesn't match the split sizes in the dataset card\n", "train_raw = load_dataset(dataset_id, data_files={\"train\": train_files}, split=\"train\", verification_mode=\"no_checks\")\n", "test_raw = load_dataset(dataset_id, data_files={\"test\": \"data/test-00000-of-00001.parquet\"}, split=\"test\", verification_mode=\"no_checks\")\n", "\n", "\n", "def flatten(row):\n", " return {\n", " \"wav\": row[\"audio_chat\"][0][\"content\"][0][\"audio\"],\n", " \"target\": row[\"audio_chat\"][1][\"content\"][0][\"text\"],\n", " \"transcript\": row[\"text_chat\"][0][\"content\"][0][\"text\"],\n", " }\n", "\n", "\n", "train_raw = train_raw.map(flatten, remove_columns=train_raw.column_names)\n", "test_raw = test_raw.map(flatten, remove_columns=test_raw.column_names)\n", "\n", "# stratified evaluation subset: up to 10 samples per function\n", "by_function = collections.defaultdict(list)\n", "for i, target in enumerate(test_raw[\"target\"]):\n", " by_function[target.split(\"|\")[0]].append(i)\n", "rng = random.Random(0)\n", "eval_raw = test_raw.select(sorted(i for idx in by_function.values() for i in rng.sample(idx, min(10, len(idx)))))\n", "\n", "print(\"โœ… Dataset loaded:\")\n", "print(f\" ๐Ÿ“š Train samples: {len(train_raw)}\")\n", "print(f\" ๐Ÿงช Eval samples: {len(eval_raw)} across {len(by_function)} functions\")\n", "for row in train_raw.select(range(3)):\n", " print(f\"\\n๐ŸŽ™๏ธ {row['transcript']!r}\\n๐Ÿ”ง {row['target']}\")\n", " display(Audio(row[\"wav\"]))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Format the conversations\n", "\n", "The recording is the whole user turn; the function call is the assistant's answer. A short system prompt names the task.\n", "Recordings are 24 kHz; we resample them to the 16 kHz the audio encoder expects." ] }, { "cell_type": "code", "metadata": {}, "source": [ "SYSTEM_PROMPT = \"Convert the spoken command into a Home Assistant function call.\"\n", "\n", "\n", "def load_16k(wav_bytes):\n", " wav, sr = sf.read(io.BytesIO(wav_bytes), dtype=\"float32\")\n", " return librosa.resample(wav, orig_sr=sr, target_sr=16_000)\n", "\n", "\n", "def format_sample(row, with_answer=True):\n", " conversation = [\n", " {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n", " {\"role\": \"user\", \"content\": [{\"type\": \"audio\", \"audio\": load_16k(row[\"wav\"])}]},\n", " ]\n", " if with_answer:\n", " conversation.append({\"role\": \"assistant\", \"content\": row[\"target\"]})\n", " return conversation\n", "\n", "\n", "print(processor.apply_chat_template(format_sample(train_raw[0]), tokenize=False))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Collate function\n", "\n", "The processor renders a batch of conversations, computes the log-mel features of each recording and expands every\n", "`<|audio|>` placeholder to its audio embeddings. We train only on the answer: every token up to and including the\n", "assistant header is masked with `-100`. Recordings are decoded on the fly, so the dataset stays small in memory." ] }, { "cell_type": "code", "metadata": {}, "source": [ "_prompt = processor.apply_chat_template(format_sample(train_raw[0], with_answer=False), add_generation_prompt=True, tokenize=False)\n", "ASSISTANT_HEADER = processor.tokenizer(_prompt[_prompt.rindex(\"<|im_start|>\"):], add_special_tokens=False)[\"input_ids\"]\n", "\n", "\n", "def answer_start(ids):\n", " n = len(ASSISTANT_HEADER)\n", " return max(i for i in range(len(ids) - n + 1) if ids[i:i + n] == ASSISTANT_HEADER) + n\n", "\n", "\n", "def collate_fn(rows):\n", " batch = processor.apply_chat_template(\n", " [format_sample(row) for row in rows], tokenize=True, return_dict=True, return_tensors=\"pt\", processor_kwargs={\"padding\": True}\n", " )\n", " labels = torch.full_like(batch[\"input_ids\"], -100)\n", " for i, ids in enumerate(batch[\"input_ids\"].tolist()):\n", " start = answer_start(ids)\n", " labels[i, start:] = batch[\"input_ids\"][i, start:]\n", " labels[batch[\"attention_mask\"] == 0] = -100\n", " batch[\"labels\"] = labels\n", " return batch\n", "\n", "\n", "batch = collate_fn([train_raw[0], train_raw[1]])\n", "print({k: tuple(v.shape) for k, v in batch.items()})\n", "print(\"๐ŸŽฏ Supervised tokens:\", [processor.tokenizer.decode(ids[lab != -100]) for ids, lab in zip(batch[\"input_ids\"], batch[\"labels\"])])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿ“ Baseline\n", "\n", "We score predictions with three layered metrics, as in the LFM2.5 example:\n", "- **Format**: the output parses as `Function|$key=value|...`\n", "- **Function**: the function name is right\n", "- **Exact match**: the function name and every argument are right (strict string equality)\n", "\n", "`generation_mode=\"text\"` keeps the answer in text." ] }, { "cell_type": "code", "metadata": {}, "source": [ "import re\n", "from tqdm.auto import tqdm\n", "\n", "\n", "def parse(call):\n", " parts = call.strip().split(\"|\")\n", " if not re.fullmatch(r\"[A-Za-z]+\", parts[0]) or any(not re.fullmatch(r\"\\$[^=|]+=.*\", p) for p in parts[1:]):\n", " return None\n", " return parts[0], dict(p[1:].split(\"=\", 1) for p in parts[1:])\n", "\n", "\n", "@torch.inference_mode()\n", "def evaluate(model, show=5):\n", " scores = collections.Counter()\n", " for k, row in enumerate(tqdm(eval_raw, desc=\"Evaluating\")):\n", " inputs = processor.apply_chat_template(\n", " format_sample(row, with_answer=False),\n", " add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors=\"pt\",\n", " ).to(model.device)\n", " out = model.generate(**inputs, max_new_tokens=64, generation_mode=\"text\")\n", " prediction = processor.decode(out.sequences[0, inputs[\"input_ids\"].shape[1]:], skip_special_tokens=True).strip()\n", " if k < show:\n", " print(f\" ๐ŸŽ™๏ธ {row['transcript']!r}\\n โ†’ {prediction!r}\\n โœ… {row['target']!r}\")\n", " predicted, expected = parse(prediction), parse(row[\"target\"])\n", " scores[\"format\"] += predicted is not None\n", " scores[\"function\"] += predicted is not None and predicted[0] == expected[0]\n", " scores[\"exact\"] += predicted == expected\n", " return {metric: scores[metric] / len(eval_raw) for metric in (\"format\", \"function\", \"exact\")}\n", "\n", "\n", "def report(name, scores):\n", " print(f\"{name:8s} format {scores['format']:6.1%} function {scores['function']:6.1%} exact match {scores['exact']:6.1%}\")\n", "\n", "\n", "scores_before = evaluate(model)\n", "report(\"before\", scores_before)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Without fine-tuning, the model has never seen this function vocabulary or the pipe-delimited format: it mostly answers in its own `` format with made-up function names, or just transcribes the command.\n", "Fine-tuning bakes the 41 function signatures into the weights.\n", "\n", "## Wrap the model with PEFT\n", "\n", "We add LoRA adapters to the language model: the attention projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`), the gated short convolutions (`in_proj`, `out_proj`), and the MLP (`w1`, `w2`, `w3`). The audio encoder stays frozen. LFM3's attention layers also have a per-head output gate, `gate_proj`; adding it to the targets is a reasonable choice. Feel free to modify the `r` (rank) value:\n", "- Higher โ†’ better approximation of full fine-tuning\n", "- Lower โ†’ needs even less compute resources" ] }, { "cell_type": "code", "metadata": {}, "source": [ "from peft import LoraConfig, get_peft_model\n", "\n", "peft_config = LoraConfig(\n", " r=16,\n", " lora_alpha=32,\n", " lora_dropout=0.05,\n", " bias=\"none\",\n", " task_type=\"CAUSAL_LM\",\n", " # optional: add \"|gate_proj\" to the group for LFM3's per-head attention gate\n", " target_modules=r\"model\\.language_model\\..*\\.(q_proj|k_proj|v_proj|o_proj|in_proj|out_proj|w1|w2|w3)\",\n", ")\n", "\n", "model = get_peft_model(model, peft_config)\n", "model.print_trainable_parameters()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿš€ Launch Training\n", "\n", "We use the ๐Ÿค— `Trainer`, which trains on the loss returned by the model. Our collate function builds the batches, so we keep all\n", "columns (`remove_unused_columns=False`). One epoch over two shards takes ~10 minutes on an A100." ] }, { "cell_type": "code", "metadata": {}, "source": [ "from transformers import Trainer, TrainingArguments\n", "\n", "training_args = TrainingArguments(\n", " output_dir=\"lfm3-omni-voice-commands\",\n", " num_train_epochs=1,\n", " per_device_train_batch_size=16,\n", " learning_rate=2e-4,\n", " lr_scheduler_type=\"cosine\",\n", " warmup_steps=20,\n", " logging_steps=20,\n", " save_strategy=\"no\",\n", " bf16=True,\n", " remove_unused_columns=False,\n", " dataloader_num_workers=2,\n", " report_to=\"none\",\n", ")\n", "\n", "print(\"๐Ÿ—๏ธ Creating trainer...\")\n", "trainer = Trainer(model=model, args=training_args, train_dataset=train_raw, data_collator=collate_fn)\n", "\n", "print(\"\\n๐Ÿš€ Starting SFT training...\")\n", "trainer.train()\n", "print(\"๐ŸŽ‰ SFT training completed!\")\n", "print(f\"๐Ÿ’พ Peak GPU memory: {torch.cuda.max_memory_allocated() / 1e9:.1f} GB\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿ“ˆ Evaluate the fine-tuned model" ] }, { "cell_type": "code", "metadata": {}, "source": [ "model.eval()\n", "scores_after = evaluate(model)\n", "report(\"before\", scores_before)\n", "report(\"after\", scores_after)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿ’พ Save the model\n", "\n", "Save the LoRA adapter, or merge it into the base weights to obtain a regular checkpoint." ] }, { "cell_type": "code", "metadata": {}, "source": [ "model.save_pretrained(\"./lfm3-omni-voice-commands-lora\")\n", "print(\"๐Ÿ’พ Adapter saved to: ./lfm3-omni-voice-commands-lora\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "print(\"๐Ÿ”„ Merging LoRA weights...\")\n", "merged = model.merge_and_unload()\n", "merged.save_pretrained(\"./lfm3-omni-voice-commands\")\n", "processor.save_pretrained(\"./lfm3-omni-voice-commands\")\n", "print(\"๐Ÿ’พ Model saved to: ./lfm3-omni-voice-commands\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿš€ Next steps\n", "\n", "- **More data**: raise `TRAIN_SHARDS` (up to 16) to improve accuracy on rare functions and argument values.\n", "- **Your own commands**: record your users' phrasings, or synthesize them with the model's own text-to-speech, and pair them with your API's function calls.\n", "- **Deploy on device**: the [LFM2.5 voice assistant example](https://github.com/Liquid4All/cookbook/tree/main/examples/voice-assistant) shows how to quantize the fine-tuned model to GGUF and serve it with llama.cpp.\n", "- [`lfm3_omni_vqa_sft_with_transformers.ipynb`](lfm3_omni_vqa_sft_with_transformers.ipynb) combines images and spoken questions." ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "A100", "provenance": [], "machine_shape": "hm" }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }