{ "cells": [ { "cell_type": "markdown", "id": "62129596-d10f-45b1-a1af-ee10f358f773", "metadata": { "id": "62129596-d10f-45b1-a1af-ee10f358f773" }, "source": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka
\n", "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", "\n", "		\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "b0bd2379-ed2f-4c77-8b71-f1f0242b9ff9", "metadata": { "id": "b0bd2379-ed2f-4c77-8b71-f1f0242b9ff9" }, "source": [ "# 直接偏好优化（DPO）用于 LLM 对齐（从零实现）\n", "\n", "- 本代码笔记本 **从零实现** **直接偏好优化（Direct Preference Optimization, DPO）**， \n", " 并将其应用于 **大语言模型（LLM）**，以提高其 **生成符合用户偏好的响应能力**。 " ] }, { "cell_type": "code", "execution_count": 1, "id": "pxMGAf3bnVwn", "metadata": { "id": "pxMGAf3bnVwn" }, "outputs": [], "source": [ "# !pip install -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/requirements.txt" ] }, { "cell_type": "code", "execution_count": 2, "id": "edb3e145-fbaa-4bb3-9e95-186b4145087f", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "edb3e145-fbaa-4bb3-9e95-186b4145087f", "outputId": "3d449525-76cc-4124-ab30-a93c6a9623ee" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tiktoken version: 0.7.0\n", "torch version: 2.3.1+cu121\n" ] } ], "source": [ "from importlib.metadata import version\n", "\n", "pkgs = [\n", " \"tiktoken\", # 分词器\n", " \"torch\", # 深度学习库\n", "]\n", "for p in pkgs:\n", " print(f\"{p} version: {version(p)}\")" ] }, { "cell_type": "markdown", "id": "49ec20a3-a26c-4f9b-8a33-bfd3d67860e2", "metadata": { "id": "49ec20a3-a26c-4f9b-8a33-bfd3d67860e2" }, "source": [ " \n", "# 1) 导论·DPO" ] }, { "cell_type": "markdown", "id": "17804afd-786b-4600-bad0-f5805454e3d6", "metadata": { "id": "17804afd-786b-4600-bad0-f5805454e3d6" }, "source": [ "- **直接偏好优化（DPO）** 是 **论文 [《Direct Preference Optimization: Your Language Model is Secretly a Reward Model》](https://arxiv.org/abs/2305.18290)** 提出的方法， \n", " 它是 **一种替代强化学习（RLHF）** 的方法，可用于 **微调大语言模型（LLM）**。 \n", "- **DPO 可用于微调（对齐）模型**，使其生成的响应 **更符合用户期望和指令**。 \n", "\n", "\n", "\n", "- 在 **指令微调（Instruction Finetuning）** 过程中，我们训练 LLM **根据提示词生成正确答案**。 \n", "- **然而，在实际应用中**，正确答案可能有 **多种表达方式**，风格也可能不同： \n", " - 例如，回答相同的问题，模型可以给出 **技术性解答**，也可以给出 **更友好的用户导向解答**。 \n", " - 下图展示了 **当用户咨询购买笔记本电脑建议时**，LLM 可能生成的两种不同风格的响应： \n", "\n", "\n", "\n", "- **RLHF（强化学习人类反馈）** 和 **DPO（直接偏好优化）** 是两种常见的方法， \n", " 它们可以 **引导 LLM 更倾向于某种特定回答风格，以更好地满足用户偏好**。 \n", "- **RLHF（强化学习人类反馈）** 需要 **训练一个单独的奖励模型（Reward Model）**，其流程如下图所示：\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "9073622f-d537-42bf-8778-43c2adaa2191", "metadata": { "id": "9073622f-d537-42bf-8778-43c2adaa2191" }, "source": [ "- **与 RLHF 相比**，DPO **简化了优化流程**， \n", " **无需构建复杂的奖励模型（Reward Model）和策略优化（Policy Optimization）**， \n", " **直接优化模型以符合用户偏好**。 \n", "- 换句话说，**DPO 直接调整模型的输出**，使其更符合 **人类偏好** 或 **特定目标**。 \n", "- 下图概述了 **DPO 的核心思想** 及其 **优化过程**：\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "c894134a-315c-453e-bbc1-387794b3f4d6", "metadata": { "id": "c894134a-315c-453e-bbc1-387794b3f4d6" }, "source": [ "- 实现 DPO 损失的具体方程如下所示；我们将在本代码笔记本的后面部分用 Python 实现该方程时再次讨论它。\n", "\n", "" ] }, { "cell_type": "markdown", "id": "dd7491b5-f619-4501-ad39-2942de57c115", "metadata": { "id": "dd7491b5-f619-4501-ad39-2942de57c115" }, "source": [ "- 在上方公式中：\n", " - **“期望值”（Expected Value，$\\mathbb{E}$）** 是统计学术语，表示 **随机变量的平均值**（即括号内的表达式）。 \n", " **优化 $-\\mathbb{E}$ 可使模型更符合用户偏好**。 \n", " - **$\\pi_{\\theta}$ 代表策略（Policy）**，该术语来源于强化学习， \n", " **在 DPO 中，它表示我们要优化的 LLM**； \n", " **$\\pi_{ref}$ 是参考模型（Reference LLM）**，通常是优化前的原始 LLM。 \n", " **在训练初期，$\\pi_{\\theta}$ 和 $\\pi_{ref}$ 通常是相同的**。 \n", " - **$\\beta$ 是控制 $\\pi_{\\theta}$ 和参考模型之间散度的超参数**： \n", " - **增大 $\\beta$ 会加大两者对数概率（Log Probabilities）之间的影响**， \n", " **从而提高优化后的 LLM 与原始 LLM 之间的差异**。 \n", " - **逻辑 Sigmoid 函数（$\\sigma(\\centerdot)$）** 用于 **将偏好响应与非偏好响应的对数几率（Log-Odds）转换为概率分数**。 \n", "\n", "- **为了避免代码笔记本过于冗长**，关于这些概念的更详细讨论， \n", " 我可能会在未来撰写 **独立的文章** 进行介绍。 \n", "\n", "- **如果您想比较 RLHF 与 DPO**，可以参考我的文章：\n", " - **[LLM 预训练与奖励模型评估技巧](https://magazine.sebastianraschka.com/p/tips-for-llm-pretraining-and-evaluating-rms)**\n", " - 其中的 **[2.2. RLHF vs 直接偏好优化（DPO）](https://magazine.sebastianraschka.com/i/142924793/rlhf-vs-direct-preference-optimization-dpo)** 部分详细分析了两者的区别。 \n" ] }, { "cell_type": "markdown", "id": "xqVAgsyQ6LuG", "metadata": { "id": "xqVAgsyQ6LuG", "tags": [] }, "source": [ "  \n", "# 2) 准备 DPO 偏好数据集 \n", "\n", "- **首先，我们加载并准备数据集**， \n", " 这可能会 **回答您在进一步研究 DPO 损失函数之前的许多问题**。 \n", "- 这里，我们使用的数据集包含 **对相同指令的两种不同风格的响应**： \n", " - **较为礼貌的（Preferred）** \n", " - **较不礼貌的（Dispreferred）** \n", " - **具体示例将在下一节展示**。 \n", "- **该数据集** 通过 **[create-preference-data-ollama.ipynb](create-preference-data-ollama.ipynb)** 生成。 \n" ] }, { "cell_type": "markdown", "id": "wHLB62Nj7haD", "metadata": { "id": "wHLB62Nj7haD" }, "source": [ "  \n", "## 2.1) 加载偏好数据集 \n", "\n", "- **该数据集为 JSON 文件**，共包含 **1100 条样本**： " ] }, { "cell_type": "code", "execution_count": 3, "id": "5266e66c-5ec0-45e6-a654-148971f6aee7", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5266e66c-5ec0-45e6-a654-148971f6aee7", "outputId": "04e8ee70-3076-441d-d2bf-7641da3d0c1d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of entries: 1100\n" ] } ], "source": [ "import json\n", "import os\n", "import urllib\n", "\n", "\n", "def download_and_load_file(file_path, url):\n", "\n", " if not os.path.exists(file_path):\n", " with urllib.request.urlopen(url) as response:\n", " text_data = response.read().decode(\"utf-8\")\n", " with open(file_path, \"w\", encoding=\"utf-8\") as file:\n", " file.write(text_data)\n", " else:\n", " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", " text_data = file.read()\n", "\n", " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", " data = json.load(file)\n", "\n", " return data\n", "\n", "\n", "file_path = \"instruction-data-with-preference.json\"\n", "url = (\n", " \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch\"\n", " \"/main/ch07/04_preference-tuning-with-dpo/instruction-data-with-preference.json\"\n", ")\n", "\n", "data = download_and_load_file(file_path, url)\n", "print(\"Number of entries:\", len(data))" ] }, { "cell_type": "markdown", "id": "725d2b9a-d6d2-46e2-89f8-5ab87e040e3b", "metadata": { "id": "725d2b9a-d6d2-46e2-89f8-5ab87e040e3b" }, "source": [ "- 让我们来看两个示例数据: " ] }, { "cell_type": "code", "execution_count": 4, "id": "5c11916f-9a26-4367-a16e-7b0c121a20a6", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5c11916f-9a26-4367-a16e-7b0c121a20a6", "outputId": "00a432cc-19b1-484f-80e2-e897ee5e4024" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'instruction': 'Identify the correct spelling of the following word.',\n", " 'input': 'Ocassion',\n", " 'output': \"The correct spelling is 'Occasion.'\",\n", " 'rejected': \"The correct spelling is obviously 'Occasion.'\",\n", " 'chosen': \"The correct spelling is 'Occasion.'\"}\n" ] } ], "source": [ "import pprint\n", "\n", "pprint.pp(data[50])" ] }, { "cell_type": "code", "execution_count": 5, "id": "01ef804a-8c13-4a0b-9b2e-b65a4d0a870d", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "01ef804a-8c13-4a0b-9b2e-b65a4d0a870d", "outputId": "078cd643-83fb-4b42-ecf9-3256e8c9d239" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'instruction': \"What is an antonym of 'complicated'?\",\n", " 'input': '',\n", " 'output': \"An antonym of 'complicated' is 'simple'.\",\n", " 'chosen': \"A suitable antonym for 'complicated' would be 'simple'.\",\n", " 'rejected': \"An antonym of 'complicated' is 'simple'.\"}\n" ] } ], "source": [ "pprint.pp(data[999])" ] }, { "cell_type": "markdown", "id": "56db5697-a089-4b40-a1f3-e928e8018220", "metadata": { "id": "56db5697-a089-4b40-a1f3-e928e8018220" }, "source": [ "\n", "\n", "```\n", "# This is formatted as code\n", "```\n", "\n", "- 如上所示，数据集包含 **5 个键（keys）**：\n", " - **`'instruction'`** 和 **`'input'`**：用于 **提供给 LLM 的输入**。 \n", " - **`'output'`**：模型在 **第 7 章的指令微调阶段** 训练时生成的 **标准响应**。 \n", " - **`'chosen'`** 和 **`'rejected'`**：DPO 训练所需的 **偏好数据**：\n", " - **`'chosen'`**：**偏好（Preferred）** 响应。 \n", " - **`'rejected'`**：**非偏好（Dispreferred）** 响应。 \n", "\n", "- **DPO 训练的目标** 是让 **模型更倾向于生成 `'chosen'` 风格的响应**，而 **避免 `'rejected'` 风格**。 \n" ] }, { "cell_type": "markdown", "id": "86257468-a6ab-4ba3-9c9f-2fdc2c0cc284", "metadata": { "id": "86257468-a6ab-4ba3-9c9f-2fdc2c0cc284" }, "source": [ "- 下面是一个 **工具函数**，用于 **格式化模型输入**， \n", " 采用的格式与 **第 7 章（[../01_main-chapter-code/ch07.ipynb](../01_main-chapter-code/ch07.ipynb)）** \n", " **相同，基于 Alpaca 提示风格（Alpaca Prompt Style）**： \n" ] }, { "cell_type": "code", "execution_count": 6, "id": "4564d55c-1c5d-46a6-b5e8-46ab568ad627", "metadata": { "id": "4564d55c-1c5d-46a6-b5e8-46ab568ad627" }, "outputs": [], "source": [ "def format_input(entry):\n", " instruction_text = (\n", " f\"Below is an instruction that describes a task. \"\n", " f\"Write a response that appropriately completes the request.\"\n", " f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n", " )\n", "\n", " input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n", "\n", " return instruction_text + input_text" ] }, { "cell_type": "code", "execution_count": 7, "id": "3f38b49f-63fd-48c5-bde8-a4717b7923ea", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3f38b49f-63fd-48c5-bde8-a4717b7923ea", "outputId": "9ad07c59-05b3-42ae-c5bc-68780aaf6780" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Identify the correct spelling of the following word.\n", "\n", "### Input:\n", "Ocassion\n" ] } ], "source": [ "model_input = format_input(data[50])\n", "print(model_input)" ] }, { "cell_type": "markdown", "id": "7dd9e4c9-88a3-463a-8c16-c60ed7e6b51e", "metadata": { "id": "7dd9e4c9-88a3-463a-8c16-c60ed7e6b51e" }, "source": [ "- 同样，我们可以使用 **Alpaca 提示风格** 格式化 **`chosen`（偏好响应）** 和 **`rejected`（非偏好响应）**： \n" ] }, { "cell_type": "code", "execution_count": 8, "id": "8ad5831a-e936-44e5-a5cf-02953fe7d848", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "8ad5831a-e936-44e5-a5cf-02953fe7d848", "outputId": "2c0a0cbf-c13d-43cf-fcc1-a4585c21e66f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "### Response:\n", "The correct spelling is 'Occasion.'\n" ] } ], "source": [ "desired_response = f\"### Response:\\n{data[50]['chosen']}\"\n", "print(desired_response)" ] }, { "cell_type": "code", "execution_count": 9, "id": "fc0991f6-fef7-48ab-8dee-fbd2863f784c", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fc0991f6-fef7-48ab-8dee-fbd2863f784c", "outputId": "cd85406c-3470-48f8-9792-63f91affd50a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "### Response:\n", "The correct spelling is obviously 'Occasion.'\n" ] } ], "source": [ "possible_response = f\"### Response:\\n{data[50]['rejected']}\"\n", "print(possible_response)" ] }, { "cell_type": "markdown", "id": "6G3j2Q987t_g", "metadata": { "id": "6G3j2Q987t_g" }, "source": [ "  \n", "## 2.2) 划分训练集、验证集和测试集 " ] }, { "cell_type": "markdown", "id": "e558ecc7", "metadata": {}, "source": [ "- 接下来，我们将数据集划分为 **3 个子集**：\n", " - **85%** 用作 **训练集（Training Set）** \n", " - **5%** 用作 **验证集（Validation Set）** \n", " - **10%** 用作 **测试集（Test Set）** " ] }, { "cell_type": "code", "execution_count": 10, "id": "36c7b919-8531-4e33-aebf-aaf8e6dbcfbd", "metadata": { "id": "36c7b919-8531-4e33-aebf-aaf8e6dbcfbd" }, "outputs": [], "source": [ "train_portion = int(len(data) * 0.85) # 85% 用作训练集\n", "test_portion = int(len(data) * 0.1) # 10% 用作测试集\n", "val_portion = len(data) - train_portion - test_portion # 剩下的 5% 用作验证集\n", "\n", "train_data = data[:train_portion]\n", "test_data = data[train_portion:train_portion + test_portion]\n", "val_data = data[train_portion + test_portion:]" ] }, { "cell_type": "code", "execution_count": 11, "id": "831a6c1b-119b-4622-9862-87f1db36e066", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "831a6c1b-119b-4622-9862-87f1db36e066", "outputId": "8e017483-1a75-4336-9540-ac6a69104e27" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training set length: 935\n", "Validation set length: 55\n", "Test set length: 110\n" ] } ], "source": [ "print(\"Training set length:\", len(train_data))\n", "print(\"Validation set length:\", len(val_data))\n", "print(\"Test set length:\", len(test_data))" ] }, { "cell_type": "markdown", "id": "c07d09f7-66af-49ed-8b9e-484f46e6a68d", "metadata": { "id": "c07d09f7-66af-49ed-8b9e-484f46e6a68d" }, "source": [ " \n", "## 2.3) 开发 `PreferenceDataset` 类与批量处理函数 \n" ] }, { "cell_type": "markdown", "id": "86101174-00c8-485d-8273-d086d5311926", "metadata": { "id": "86101174-00c8-485d-8273-d086d5311926" }, "source": [ "- 在本节中，我们基于 **第 7 章的 `InstructionDataset` 类** \n", " （[../01_main-chapter-code/ch07.ipynb](../01_main-chapter-code/ch07.ipynb)）**重新编写数据集类，以适用于 DPO 训练**。 \n", "- **不同之处在于**，我们不再仅关注 **单个输出序列（response）**， \n", " **而是返回两条响应，其中一条是偏好（\"chosen\"），另一条是非偏好（\"rejected\"）**。 \n", "- **整体而言**，`PreferenceDataset` **与** `InstructionDataset` **几乎相同**， \n", " 只是针对 **DPO 训练的需求做了调整**： " ] }, { "cell_type": "code", "execution_count": 12, "id": "db08ad74-6dd4-4e40-b1e5-bc5f037d3d27", "metadata": { "id": "db08ad74-6dd4-4e40-b1e5-bc5f037d3d27" }, "outputs": [], "source": [ "import torch\n", "from torch.utils.data import Dataset\n", "\n", "\n", "class PreferenceDataset(Dataset):\n", " def __init__(self, data, tokenizer):\n", " self.data = data\n", "\n", " # 预分词文本\n", " self.encoded_texts = []\n", " for entry in data:\n", " prompt = format_input(entry)\n", " rejected_response = entry[\"rejected\"]\n", " chosen_response = entry[\"chosen\"]\n", "\n", " prompt_tokens = tokenizer.encode(prompt)\n", " chosen_full_text = f\"{prompt}\\n\\n### Response:\\n{chosen_response}\"\n", " rejected_full_text = f\"{prompt}\\n\\n### Response:\\n{rejected_response}\"\n", " chosen_full_tokens = tokenizer.encode(chosen_full_text)\n", " rejected_full_tokens = tokenizer.encode(rejected_full_text)\n", "\n", " self.encoded_texts.append({\n", " \"prompt\": prompt_tokens,\n", " \"chosen\": chosen_full_tokens,\n", " \"rejected\": rejected_full_tokens,\n", " })\n", "\n", " def __getitem__(self, index):\n", " return self.encoded_texts[index]\n", "\n", " def __len__(self):\n", " return len(self.data)\n" ] }, { "cell_type": "markdown", "id": "2325d183-75b9-400a-80ac-0b8d2f526561", "metadata": { "id": "2325d183-75b9-400a-80ac-0b8d2f526561" }, "source": [ "- 除了 **更新 `PreferenceDataset` 类**，我们还需要 **更新批处理（batch collation）函数**， \n", " 该函数的作用是 **对每个批次（batch）中的序列进行填充（padding），使其长度一致**， \n", " 以便将数据 **批量组织** 进行训练。 \n", "- **下方代码已添加注释**，帮助理解数据处理流程； \n", " **不过，最直观的方式是查看后续示例输入和输出**，了解其工作原理。 " ] }, { "cell_type": "code", "execution_count": 13, "id": "8d3a43a6-7704-4bff-9bbc-a38632374f30", "metadata": { "id": "8d3a43a6-7704-4bff-9bbc-a38632374f30" }, "outputs": [], "source": [ "def custom_collate_fn(\n", " batch,\n", " pad_token_id=50256,\n", " allowed_max_length=None,\n", " mask_prompt_tokens=True,\n", " device=\"cpu\"\n", "):\n", " # 初始化列表以保存批次数据\n", " batch_data = {\n", " \"prompt\": [],\n", " \"chosen\": [],\n", " \"rejected\": [],\n", " \"rejected_mask\": [],\n", " \"chosen_mask\": []\n", "\n", " }\n", "\n", " # 确定最长的序列以设置相同的填充长度\n", " max_length_common = 0\n", " if batch:\n", " for key in [\"chosen\", \"rejected\"]:\n", " current_max = max(len(item[key])+1 for item in batch)\n", " max_length_common = max(max_length_common, current_max)\n", "\n", " # 处理批次中的每个项目\n", " for item in batch:\n", " prompt = torch.tensor(item[\"prompt\"])\n", " batch_data[\"prompt\"].append(prompt)\n", "\n", " for key in [\"chosen\", \"rejected\"]:\n", " # 根据相同的最大长度调整填充\n", " sequence = item[key]\n", " padded = sequence + [pad_token_id] * (max_length_common - len(sequence))\n", " mask = torch.ones(len(padded)).bool()\n", "\n", " # 将所有填充标记的掩码设置为 False\n", " mask[len(sequence):] = False\n", "\n", " # 将所有填充标记的掩码设置为 False\n", " # +2 将 \"### Response\" 之前的 2 个换行符 (\"\\n\") 标记设置为 False\n", " if mask_prompt_tokens:\n", " mask[:prompt.shape[0]+2] = False\n", "\n", " batch_data[key].append(torch.tensor(padded))\n", " batch_data[f\"{key}_mask\"].append(mask)\n", "\n", " # 最终处理\n", " for key in [\"chosen\", \"rejected\", \"chosen_mask\", \"rejected_mask\"]:\n", " # 将所有序列堆叠为给定键的张量\n", " tensor_stack = torch.stack(batch_data[key])\n", "\n", " # 可选：截断到最大序列长度\n", " if allowed_max_length is not None:\n", " tensor_stack = tensor_stack[:, :allowed_max_length]\n", "\n", " # 移动到指定设备\n", " batch_data[key] = tensor_stack.to(device)\n", "\n", " return batch_data" ] }, { "cell_type": "markdown", "id": "76f3744b-9bb0-4f1e-b66b-cff35ad8fd9f", "metadata": { "id": "76f3744b-9bb0-4f1e-b66b-cff35ad8fd9f" }, "source": [ "- 在正式使用 **自定义批处理函数（collate function）** 之前， \n", " **我们先创建一个预填充部分参数的版本**，以便后续调用更加方便： " ] }, { "cell_type": "code", "execution_count": 14, "id": "d3cc137c-7ed7-4758-a518-cc4071b2817a", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "d3cc137c-7ed7-4758-a518-cc4071b2817a", "outputId": "598e9def-9768-441a-f886-01f6ba6e250b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Device: cuda\n" ] } ], "source": [ "from functools import partial\n", "\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "print(\"Device:\", device)\n", "\n", "customized_collate_fn = partial(\n", " custom_collate_fn,\n", " device=device, # 如果有 GPU，直接将数据放在 GPU 上\n", " mask_prompt_tokens=True, # 这是可选的\n", " allowed_max_length=1024 # 模型支持的上下文长度\n", ")" ] }, { "cell_type": "markdown", "id": "5d29e996-e267-4348-bc1d-4ac6b725cf6a", "metadata": { "id": "5d29e996-e267-4348-bc1d-4ac6b725cf6a" }, "source": [ "- 现在，让我们实际运行 **`customized_collate_fn`**， \n", " 并将其应用于 **偏好数据集中的示例数据**； \n", " **为此，我们选取数据集的前两条样本进行测试**： " ] }, { "cell_type": "code", "execution_count": 15, "id": "1171057d-2a0f-48ff-bad6-4917a072f0f5", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1171057d-2a0f-48ff-bad6-4917a072f0f5", "outputId": "3db3eee8-db29-4ff6-8078-6577a05d953a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "{'instruction': 'Evaluate the following phrase by transforming it into the '\n", " 'spelling given.',\n", " 'input': 'freind --> friend',\n", " 'output': 'The spelling of the given phrase \"freind\" is incorrect, the '\n", " 'correct spelling is \"friend\".',\n", " 'rejected': 'The spelling of the given phrase \"freind\" is flat out wrong, get '\n", " 'it together, the correct spelling is \"friend\".',\n", " 'chosen': 'The spelling of the given phrase \"freind\" is incorrect, the '\n", " 'correct spelling is \"friend\".'}\n", "\n", "{'instruction': 'Edit the following sentence for grammar.',\n", " 'input': 'He go to the park every day.',\n", " 'output': 'He goes to the park every day.',\n", " 'rejected': 'He goes to the stupid park every single day.',\n", " 'chosen': 'He goes to the park every day.'}\n" ] } ], "source": [ "example_data = data[:2]\n", "\n", "for i in example_data:\n", " print()\n", " pprint.pp(i)" ] }, { "cell_type": "markdown", "id": "8f1436cc-fbe5-4581-89d8-1992b5f04042", "metadata": { "id": "8f1436cc-fbe5-4581-89d8-1992b5f04042" }, "source": [ "- 接下来，我们实例化 **`example_dataset`**， \n", " 并使用 **PyTorch `DataLoader`** 创建 **`example_dataloader`**， \n", " **以模拟后续用于模型训练的数据加载器**： " ] }, { "cell_type": "code", "execution_count": 16, "id": "db327575-c34b-4fea-b3c7-e30569c9be78", "metadata": { "id": "db327575-c34b-4fea-b3c7-e30569c9be78" }, "outputs": [], "source": [ "import tiktoken\n", "from torch.utils.data import DataLoader\n", "\n", "\n", "tokenizer = tiktoken.get_encoding(\"gpt2\")\n", "\n", "example_dataset = PreferenceDataset(example_data, tokenizer)\n", "\n", "example_dataloader = DataLoader(\n", " example_dataset,\n", " batch_size=2,\n", " collate_fn=customized_collate_fn,\n", " shuffle=False\n", ")" ] }, { "cell_type": "markdown", "id": "43a446b7-7037-4d9a-9f14-b4ee0f6f37af", "metadata": { "id": "43a446b7-7037-4d9a-9f14-b4ee0f6f37af" }, "source": [ "- 数据集包含以下键（keys）： " ] }, { "cell_type": "code", "execution_count": 17, "id": "87ed4cf9-d70a-4bc7-b676-67e76ed3ee10", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "87ed4cf9-d70a-4bc7-b676-67e76ed3ee10", "outputId": "fa724d65-b0e1-4239-8090-9263135ad199" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "batch.keys: dict_keys(['prompt', 'chosen', 'rejected', 'rejected_mask', 'chosen_mask'])\n" ] } ], "source": [ "for batch in example_dataloader:\n", " break\n", "\n", "print(\"batch.keys:\", batch.keys())" ] }, { "cell_type": "markdown", "id": "5bda3193-8c68-478c-98d8-0d9d880e7077", "metadata": { "id": "5bda3193-8c68-478c-98d8-0d9d880e7077" }, "source": [ "- `prompts` 是一个 **张量（tensor）列表**，其中 **每个张量** \n", " **包含一个样本的 token ID**； \n", " **由于我们设定的批量大小（batch size）为 2**， \n", " **因此这里包含两个 token ID 张量**： " ] }, { "cell_type": "code", "execution_count": 18, "id": "468995ce-2906-498f-ac99-0a3f80d13d12", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "468995ce-2906-498f-ac99-0a3f80d13d12", "outputId": "7f3df961-fcb5-4e49-9b0c-c99447c67cc1" }, "outputs": [ { "data": { "text/plain": [ "[tensor([21106, 318, 281, 12064, 326, 8477, 257, 4876, 13, 19430,\n", " 257, 2882, 326, 20431, 32543, 262, 2581, 13, 198, 198,\n", " 21017, 46486, 25, 198, 36, 2100, 4985, 262, 1708, 9546,\n", " 416, 25449, 340, 656, 262, 24993, 1813, 13, 198, 198,\n", " 21017, 23412, 25, 198, 19503, 521, 14610, 1545]),\n", " tensor([21106, 318, 281, 12064, 326, 8477, 257, 4876, 13, 19430,\n", " 257, 2882, 326, 20431, 32543, 262, 2581, 13, 198, 198,\n", " 21017, 46486, 25, 198, 18378, 262, 1708, 6827, 329, 23491,\n", " 13, 198, 198, 21017, 23412, 25, 198, 1544, 467, 284,\n", " 262, 3952, 790, 1110, 13])]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "batch[\"prompt\"]" ] }, { "cell_type": "markdown", "id": "89cadebe-2516-4ae0-a71f-a8a623f2e1da", "metadata": { "id": "89cadebe-2516-4ae0-a71f-a8a623f2e1da" }, "source": [ "- **训练时，我们实际上不需要 `responses`**， \n", " **模型训练时需要输入的是 `\"chosen\"`（偏好）和 `\"rejected\"`（非偏好）条目**。 \n", "- **`\"chosen\"` 和 `\"rejected\"` 响应序列已进行填充（padding）**， \n", " **这样它们可以被堆叠为张量（tensor）**； \n", " **与 `prompts` 类似，这些响应文本已被转换为 token ID**： " ] }, { "cell_type": "code", "execution_count": 19, "id": "e8f49c56-3989-4fe9-81ac-6bb3cce1a5b8", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "e8f49c56-3989-4fe9-81ac-6bb3cce1a5b8", "outputId": "ccc0bd06-6e85-4ee9-893b-d985f26a835d" }, "outputs": [ { "data": { "text/plain": [ "tensor([[21106, 318, 281, 12064, 326, 8477, 257, 4876, 13, 19430,\n", " 257, 2882, 326, 20431, 32543, 262, 2581, 13, 198, 198,\n", " 21017, 46486, 25, 198, 36, 2100, 4985, 262, 1708, 9546,\n", " 416, 25449, 340, 656, 262, 24993, 1813, 13, 198, 198,\n", " 21017, 23412, 25, 198, 19503, 521, 14610, 1545, 198, 198,\n", " 21017, 18261, 25, 198, 464, 24993, 286, 262, 1813, 9546,\n", " 366, 19503, 521, 1, 318, 11491, 11, 262, 3376, 24993,\n", " 318, 366, 6726, 1911, 50256, 50256, 50256, 50256, 50256, 50256,\n", " 50256],\n", " [21106, 318, 281, 12064, 326, 8477, 257, 4876, 13, 19430,\n", " 257, 2882, 326, 20431, 32543, 262, 2581, 13, 198, 198,\n", " 21017, 46486, 25, 198, 18378, 262, 1708, 6827, 329, 23491,\n", " 13, 198, 198, 21017, 23412, 25, 198, 1544, 467, 284,\n", " 262, 3952, 790, 1110, 13, 198, 198, 21017, 18261, 25,\n", " 198, 1544, 2925, 284, 262, 3952, 790, 1110, 13, 50256,\n", " 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,\n", " 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256,\n", " 50256]], device='cuda:0')" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "batch[\"chosen\"]" ] }, { "cell_type": "markdown", "id": "35a4cd6d-b2ad-45a6-b00a-ba5b720be4ea", "metadata": { "id": "35a4cd6d-b2ad-45a6-b00a-ba5b720be4ea" }, "source": [ "- 上面显示的 token ID 代表了模型的输入，但以这种格式呈现时，对于我们人类来说是很难理解的\n", "- 因此，让我们实现一个小工具函数，将它们转换为文本格式，以便我们可以更容易地检查和理解这些内容：" ] }, { "cell_type": "code", "execution_count": 20, "id": "52ea54ba-32cb-4ecb-b38b-923f42fd4615", "metadata": { "id": "52ea54ba-32cb-4ecb-b38b-923f42fd4615" }, "outputs": [], "source": [ "def decode_tokens_from_batch(token_ids, tokenizer):\n", " ids_in_python_list = token_ids.flatten().tolist()\n", " return tokenizer.decode(ids_in_python_list)" ] }, { "cell_type": "markdown", "id": "bc9dd0ce-1fd4-419c-833f-ea5a1f8d800d", "metadata": { "id": "bc9dd0ce-1fd4-419c-833f-ea5a1f8d800d" }, "source": [ "- 让我们对批次中的第一个提示语条目应用 `decode_tokens_from_batch` 工具函数：" ] }, { "cell_type": "code", "execution_count": 21, "id": "55ee481e-3e2c-4ff6-b614-8cb18eb16a41", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "55ee481e-3e2c-4ff6-b614-8cb18eb16a41", "outputId": "17ddec15-a09d-45b5-b1e8-600cd59a9600" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Evaluate the following phrase by transforming it into the spelling given.\n", "\n", "### Input:\n", "freind --> friend\n" ] } ], "source": [ "text = decode_tokens_from_batch(\n", " token_ids=batch[\"prompt\"][0], # [0]表示批次中的第一个条目\n", " tokenizer=tokenizer,\n", ")\n", "print(text)" ] }, { "cell_type": "markdown", "id": "637b95c4-d5c2-4492-9d19-a45b090eee7e", "metadata": { "id": "637b95c4-d5c2-4492-9d19-a45b090eee7e" }, "source": [ "- 如上所示，提示语已经正确格式化；现在我们对 `\"chosen\"` 响应做同样的处理：" ] }, { "cell_type": "code", "execution_count": 22, "id": "33a24f20-5ec3-4a89-b57a-52e997163d07", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "33a24f20-5ec3-4a89-b57a-52e997163d07", "outputId": "e04366ee-3719-4b07-fcef-6e9dddc06310" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Evaluate the following phrase by transforming it into the spelling given.\n", "\n", "### Input:\n", "freind --> friend\n", "\n", "### Response:\n", "The spelling of the given phrase \"freind\" is incorrect, the correct spelling is \"friend\".<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>\n" ] } ], "source": [ "text = decode_tokens_from_batch(\n", " token_ids=batch[\"chosen\"][0],\n", " tokenizer=tokenizer,\n", ")\n", "print(text)" ] }, { "cell_type": "markdown", "id": "ac9fbdbd-1cff-401f-8e6c-cd98c134c0f2", "metadata": { "id": "ac9fbdbd-1cff-401f-8e6c-cd98c134c0f2" }, "source": [ "- 与指令微调类似，训练过程中传入模型的响应也包含了输入提示。\n", "- 另外，我们使用了 `<|endoftext|>` 作为填充token，这样可以将响应扩展到类似的长度，以便将它们堆叠成一个批次。\n", "- 不用担心，`<|endoftext|>` token在计算损失时会被忽略，因此不会影响训练结果。\n", "- 现在，让我们来看一下对应的拒绝响应：" ] }, { "cell_type": "code", "execution_count": 23, "id": "db382be5-c727-4299-8597-c05424ba9308", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "db382be5-c727-4299-8597-c05424ba9308", "outputId": "edbd8c4a-0528-4361-aeba-9b3c3bbde33b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Evaluate the following phrase by transforming it into the spelling given.\n", "\n", "### Input:\n", "freind --> friend\n", "\n", "### Response:\n", "The spelling of the given phrase \"freind\" is flat out wrong, get it together, the correct spelling is \"friend\".<|endoftext|>\n" ] } ], "source": [ "text = decode_tokens_from_batch(\n", " token_ids=batch[\"rejected\"][0],\n", " tokenizer=tokenizer,\n", ")\n", "print(text)" ] }, { "cell_type": "markdown", "id": "715dc968-aa64-4388-b577-7c295831bdcf", "metadata": { "id": "715dc968-aa64-4388-b577-7c295831bdcf" }, "source": [ "- 在这种情况下，如上所示，拒绝的响应是比选定响应更加不礼貌的版本（我们不希望模型生成不礼貌的回答）。\n", "- 最后，我们来讨论一下数据掩码：如果你仔细查看我们上面实现的自定义 collate 函数，你会发现我们为每个数据集条目创建了一个 `\"chosen_mask\"` 和一个 `\"rejected_mask\"`。\n", "- 这些掩码与响应条目的形状相同，下面展示的是 `\"chosen\"` 条目的掩码：" ] }, { "cell_type": "code", "execution_count": 24, "id": "5c324eab-cf1d-4071-b3ba-797d8ec4d1da", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5c324eab-cf1d-4071-b3ba-797d8ec4d1da", "outputId": "742a5742-1bc0-4f74-9eb9-cbf81f936ecb" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "chosen inputs: torch.Size([81])\n", "chosen mask: torch.Size([81])\n" ] } ], "source": [ "print(\"chosen inputs:\", batch[\"chosen\"][0].shape)\n", "print(\"chosen mask: \", batch[\"chosen_mask\"][0].shape)" ] }, { "cell_type": "markdown", "id": "880e95f7-cfc3-4f5f-be5e-c279fba5f674", "metadata": { "id": "880e95f7-cfc3-4f5f-be5e-c279fba5f674" }, "source": [ "- 这些掩码的内容是布尔值（`True` 和 `False`）：" ] }, { "cell_type": "code", "execution_count": 25, "id": "da75b550-5da4-4292-9a7e-a05b842bdcb7", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "da75b550-5da4-4292-9a7e-a05b842bdcb7", "outputId": "e5f012c3-33ba-4e6b-aa55-3e331865218f" }, "outputs": [ { "data": { "text/plain": [ "tensor([False, False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False, False,\n", " True, True, True, True, True, True, True, True, True, True,\n", " True, True, True, True, True, True, True, True, True, True,\n", " True, True, True, True, False, False, False, False, False, False,\n", " False], device='cuda:0')" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "batch[\"chosen_mask\"][0]" ] }, { "cell_type": "markdown", "id": "0e67b862-4430-4c99-9157-90955dde29b6", "metadata": { "id": "0e67b862-4430-4c99-9157-90955dde29b6" }, "source": [ "- `True` 值表示对应于实际响应的 token ID。\n", "- `False` 值表示对应于提示 token（如果我们在 `customized_collate_fn` 函数中设置了 `mask_prompt_tokens=True`，我们之前做过此设置）或填充 token 的 token ID。\n", "- 因此，我们可以使用掩码作为选择掩码，只选择对应于响应的 token ID，也就是去掉所有提示和填充 token，如下所示：" ] }, { "cell_type": "code", "execution_count": 26, "id": "1114c6fe-524b-401c-b9fe-02260e6f0541", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1114c6fe-524b-401c-b9fe-02260e6f0541", "outputId": "6d99af1d-940a-4012-c5d9-21d463a66e40" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "### Response:\n", "The spelling of the given phrase \"freind\" is incorrect, the correct spelling is \"friend\".\n" ] } ], "source": [ "text = decode_tokens_from_batch(\n", " token_ids=batch[\"chosen\"][0][batch[\"chosen_mask\"][0]],\n", " tokenizer=tokenizer,\n", ")\n", "print(text)" ] }, { "cell_type": "code", "execution_count": 27, "id": "a89f83a4-d16e-40d2-ba43-bd410affd967", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "a89f83a4-d16e-40d2-ba43-bd410affd967", "outputId": "1d439c7e-c079-4594-d02a-fa83a3cb275d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "### Response:\n", "The spelling of the given phrase \"freind\" is flat out wrong, get it together, the correct spelling is \"friend\".\n" ] } ], "source": [ "text = decode_tokens_from_batch(\n", " token_ids=batch[\"rejected\"][0][batch[\"rejected_mask\"][0]],\n", " tokenizer=tokenizer,\n", ")\n", "print(text)" ] }, { "cell_type": "markdown", "id": "e525287f-137c-4d71-94ae-cfd6db7b057c", "metadata": { "id": "e525287f-137c-4d71-94ae-cfd6db7b057c" }, "source": [ "- 我们将在后续计算DPO损失时使用该掩码，忽略提示符和填充token。" ] }, { "cell_type": "markdown", "id": "jbafhM_R8z5q", "metadata": { "id": "jbafhM_R8z5q" }, "source": [ "## 2.4) 创建训练集、验证集和测试集数据加载器" ] }, { "cell_type": "markdown", "id": "b3c29eb8-d1b9-4abe-a155-52b3270d759a", "metadata": { "id": "b3c29eb8-d1b9-4abe-a155-52b3270d759a" }, "source": [ "- 上面我们用了偏好数据集的一个小子集来做示范。\n", "- 接下来，我们将创建实际的训练集、验证集和测试集数据加载器。\n", "- 这一过程与预训练和指令微调章节中的数据加载器创建方式相同，应该不难理解。" ] }, { "cell_type": "code", "execution_count": 28, "id": "5c0068bf-bda0-4d9e-9f79-2fc4b94cbd1c", "metadata": { "id": "5c0068bf-bda0-4d9e-9f79-2fc4b94cbd1c" }, "outputs": [], "source": [ "from torch.utils.data import DataLoader\n", "\n", "\n", "num_workers = 0\n", "batch_size = 8\n", "\n", "torch.manual_seed(123)\n", "\n", "train_dataset = PreferenceDataset(train_data, tokenizer)\n", "train_loader = DataLoader(\n", " train_dataset,\n", " batch_size=batch_size,\n", " collate_fn=customized_collate_fn,\n", " shuffle=True,\n", " drop_last=True,\n", " num_workers=num_workers\n", ")" ] }, { "cell_type": "code", "execution_count": 29, "id": "2f4a257b-6835-4194-abe2-5831d6a44885", "metadata": { "id": "2f4a257b-6835-4194-abe2-5831d6a44885" }, "outputs": [], "source": [ "val_dataset = PreferenceDataset(val_data, tokenizer)\n", "val_loader = DataLoader(\n", " val_dataset,\n", " batch_size=batch_size,\n", " collate_fn=customized_collate_fn,\n", " shuffle=False,\n", " drop_last=False,\n", " num_workers=num_workers\n", ")\n", "\n", "test_dataset = PreferenceDataset(test_data, tokenizer)\n", "test_loader = DataLoader(\n", " test_dataset,\n", " batch_size=batch_size,\n", " collate_fn=customized_collate_fn,\n", " shuffle=False,\n", " drop_last=False,\n", " num_workers=num_workers\n", ")" ] }, { "cell_type": "markdown", "id": "1fe1ba19-a6d5-4a77-8283-7a17d7ec06e2", "metadata": { "id": "1fe1ba19-a6d5-4a77-8283-7a17d7ec06e2" }, "source": [ "- 让我们遍历数据加载器，查看每个批次的数据形状：" ] }, { "cell_type": "code", "execution_count": 30, "id": "80d61f15-facb-4eb8-a9be-6427887d24b2", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "80d61f15-facb-4eb8-a9be-6427887d24b2", "outputId": "dacd3bdf-f069-4b36-da2c-d6c1c6cc5405" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train loader:\n", "torch.Size([8, 77]) torch.Size([8, 77])\n", "torch.Size([8, 81]) torch.Size([8, 81])\n", "torch.Size([8, 94]) torch.Size([8, 94])\n", "torch.Size([8, 75]) torch.Size([8, 75])\n", "torch.Size([8, 75]) torch.Size([8, 75])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 99]) torch.Size([8, 99])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 67]) torch.Size([8, 67])\n", "torch.Size([8, 88]) torch.Size([8, 88])\n", "torch.Size([8, 65]) torch.Size([8, 65])\n", "torch.Size([8, 79]) torch.Size([8, 79])\n", "torch.Size([8, 80]) torch.Size([8, 80])\n", "torch.Size([8, 97]) torch.Size([8, 97])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 89]) torch.Size([8, 89])\n", "torch.Size([8, 75]) torch.Size([8, 75])\n", "torch.Size([8, 69]) torch.Size([8, 69])\n", "torch.Size([8, 84]) torch.Size([8, 84])\n", "torch.Size([8, 79]) torch.Size([8, 79])\n", "torch.Size([8, 101]) torch.Size([8, 101])\n", "torch.Size([8, 87]) torch.Size([8, 87])\n", "torch.Size([8, 73]) torch.Size([8, 73])\n", "torch.Size([8, 69]) torch.Size([8, 69])\n", "torch.Size([8, 80]) torch.Size([8, 80])\n", "torch.Size([8, 68]) torch.Size([8, 68])\n", "torch.Size([8, 73]) torch.Size([8, 73])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 91]) torch.Size([8, 91])\n", "torch.Size([8, 78]) torch.Size([8, 78])\n", "torch.Size([8, 78]) torch.Size([8, 78])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 84]) torch.Size([8, 84])\n", "torch.Size([8, 92]) torch.Size([8, 92])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 66]) torch.Size([8, 66])\n", "torch.Size([8, 73]) torch.Size([8, 73])\n", "torch.Size([8, 73]) torch.Size([8, 73])\n", "torch.Size([8, 78]) torch.Size([8, 78])\n", "torch.Size([8, 66]) torch.Size([8, 66])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 100]) torch.Size([8, 100])\n", "torch.Size([8, 77]) torch.Size([8, 77])\n", "torch.Size([8, 92]) torch.Size([8, 92])\n", "torch.Size([8, 93]) torch.Size([8, 93])\n", "torch.Size([8, 115]) torch.Size([8, 115])\n", "torch.Size([8, 81]) torch.Size([8, 81])\n", "torch.Size([8, 95]) torch.Size([8, 95])\n", "torch.Size([8, 81]) torch.Size([8, 81])\n", "torch.Size([8, 94]) torch.Size([8, 94])\n", "torch.Size([8, 70]) torch.Size([8, 70])\n", "torch.Size([8, 89]) torch.Size([8, 89])\n", "torch.Size([8, 90]) torch.Size([8, 90])\n", "torch.Size([8, 70]) torch.Size([8, 70])\n", "torch.Size([8, 85]) torch.Size([8, 85])\n", "torch.Size([8, 65]) torch.Size([8, 65])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 72]) torch.Size([8, 72])\n", "torch.Size([8, 84]) torch.Size([8, 84])\n", "torch.Size([8, 84]) torch.Size([8, 84])\n", "torch.Size([8, 65]) torch.Size([8, 65])\n", "torch.Size([8, 63]) torch.Size([8, 63])\n", "torch.Size([8, 74]) torch.Size([8, 74])\n", "torch.Size([8, 79]) torch.Size([8, 79])\n", "torch.Size([8, 93]) torch.Size([8, 93])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 99]) torch.Size([8, 99])\n", "torch.Size([8, 81]) torch.Size([8, 81])\n", "torch.Size([8, 77]) torch.Size([8, 77])\n", "torch.Size([8, 74]) torch.Size([8, 74])\n", "torch.Size([8, 75]) torch.Size([8, 75])\n", "torch.Size([8, 73]) torch.Size([8, 73])\n", "torch.Size([8, 87]) torch.Size([8, 87])\n", "torch.Size([8, 80]) torch.Size([8, 80])\n", "torch.Size([8, 75]) torch.Size([8, 75])\n", "torch.Size([8, 81]) torch.Size([8, 81])\n", "torch.Size([8, 86]) torch.Size([8, 86])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 63]) torch.Size([8, 63])\n", "torch.Size([8, 82]) torch.Size([8, 82])\n", "torch.Size([8, 68]) torch.Size([8, 68])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 68]) torch.Size([8, 68])\n", "torch.Size([8, 97]) torch.Size([8, 97])\n", "torch.Size([8, 72]) torch.Size([8, 72])\n", "torch.Size([8, 85]) torch.Size([8, 85])\n", "torch.Size([8, 67]) torch.Size([8, 67])\n", "torch.Size([8, 85]) torch.Size([8, 85])\n", "torch.Size([8, 87]) torch.Size([8, 87])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 74]) torch.Size([8, 74])\n", "torch.Size([8, 92]) torch.Size([8, 92])\n", "torch.Size([8, 85]) torch.Size([8, 85])\n", "torch.Size([8, 72]) torch.Size([8, 72])\n", "torch.Size([8, 93]) torch.Size([8, 93])\n", "torch.Size([8, 82]) torch.Size([8, 82])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 93]) torch.Size([8, 93])\n", "torch.Size([8, 80]) torch.Size([8, 80])\n", "torch.Size([8, 87]) torch.Size([8, 87])\n", "torch.Size([8, 69]) torch.Size([8, 69])\n", "torch.Size([8, 90]) torch.Size([8, 90])\n", "torch.Size([8, 99]) torch.Size([8, 99])\n", "torch.Size([8, 104]) torch.Size([8, 104])\n", "torch.Size([8, 101]) torch.Size([8, 101])\n", "torch.Size([8, 98]) torch.Size([8, 98])\n", "torch.Size([8, 79]) torch.Size([8, 79])\n", "torch.Size([8, 71]) torch.Size([8, 71])\n", "torch.Size([8, 76]) torch.Size([8, 76])\n", "torch.Size([8, 79]) torch.Size([8, 79])\n", "torch.Size([8, 79]) torch.Size([8, 79])\n", "torch.Size([8, 67]) torch.Size([8, 67])\n", "torch.Size([8, 84]) torch.Size([8, 84])\n", "torch.Size([8, 78]) torch.Size([8, 78])\n", "torch.Size([8, 85]) torch.Size([8, 85])\n", "torch.Size([8, 70]) torch.Size([8, 70])\n" ] } ], "source": [ "print(\"Train loader:\")\n", "for batch in train_loader:\n", " print(\n", " batch[\"chosen\"].shape,\n", " batch[\"rejected\"].shape,\n", " )" ] }, { "cell_type": "markdown", "id": "7ff958a6-5e61-49f5-9a97-360aa34e3758", "metadata": { "id": "7ff958a6-5e61-49f5-9a97-360aa34e3758" }, "source": [ "- 每一行展示了每个批次中 `\"chosen\"` 和 `\"rejected\"` 项目的形状\n", "- 由于我们在批次层面上应用了填充，每一行的形状不同\n", "- 这样做是出于效率考虑，因为如果将所有样本填充到整个数据集中最长的样本长度，会非常低效" ] }, { "cell_type": "markdown", "id": "29cb0543-1142-4374-8825-3384e20c6ac0", "metadata": { "id": "29cb0543-1142-4374-8825-3384e20c6ac0" }, "source": [ " \n", "# 3) 加载微调后的LLM进行DPO对齐" ] }, { "cell_type": "markdown", "id": "22b08881-b769-4b26-8153-5ec0e8573ed2", "metadata": { "id": "22b08881-b769-4b26-8153-5ec0e8573ed2" }, "source": [ "- LLM对齐步骤，如RLHF或DPO，假设我们已经有了一个经过指令微调的模型。\n", "- 本节包含加载在第7章中指令微调并保存的模型的最小代码（通过[../01_main-chapter-code/ch07.ipynb](../01_main-chapter-code/ch07.ipynb)）。\n", "- 在继续之前，请确保先运行第7章的代码，以创建指令微调的模型。\n", "- 以下代码将指令微调的模型复制到当前目录：" ] }, { "cell_type": "code", "execution_count": 31, "id": "b3c6d82b-63f7-459a-b901-7125ab225e56", "metadata": { "id": "b3c6d82b-63f7-459a-b901-7125ab225e56" }, "outputs": [], "source": [ "from pathlib import Path\n", "import shutil\n", "\n", "\n", "finetuned_model_path = Path(\"gpt2-medium355M-sft.pth\")\n", "if not finetuned_model_path.exists():\n", "\n", " # 尝试在本地找到模型检查点:\n", " relative_path = Path(\"..\") / \"01_main-chapter-code\" / finetuned_model_path\n", " if relative_path.exists():\n", " shutil.copy(relative_path, \".\")\n", "\n", " # 如果此笔记本在 Google Colab 上运行，则从 Google Drive 文件夹中获取\n", " elif \"COLAB_GPU\" in os.environ or \"COLAB_TPU_ADDR\" in os.environ:\n", " from google.colab import drive\n", " drive.mount(\"/content/drive\")\n", " google_drive_path = \"/content/drive/My Drive/Books/LLMs-From-Scratch/ch07/colab/gpt2-medium355M-sft.pth\" # 读者需要调整此路径\n", " shutil.copy(google_drive_path, \".\")\n", "\n", " else:\n", " print(\n", " f\"Could not find '{finetuned_model_path}'.\\n\"\n", " \"Run the `ch07.ipynb` notebook to finetune and save the finetuned model.\"\n", " )" ] }, { "cell_type": "markdown", "id": "71c8585e-4569-4033-84a7-3903d0e8aaf8", "metadata": { "id": "71c8585e-4569-4033-84a7-3903d0e8aaf8" }, "source": [ "- 接下来，我们重用前几章中的基本配置来加载模型权重：" ] }, { "cell_type": "code", "execution_count": 32, "id": "a8333fee-e7fe-4f8c-9411-8c1db6252d98", "metadata": { "id": "a8333fee-e7fe-4f8c-9411-8c1db6252d98" }, "outputs": [], "source": [ "from previous_chapters import GPTModel\n", "\n", "\n", "BASE_CONFIG = {\n", " \"vocab_size\": 50257, # 词表大小\n", " \"context_length\": 1024, # 上下文长度\n", " \"drop_rate\": 0.0, # Dropout率\n", " \"qkv_bias\": True # 查询-键-值偏置\n", "}\n", "\n", "model_configs = {\n", " \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n", " \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n", " \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n", " \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n", "}\n", "\n", "CHOOSE_MODEL = \"gpt2-medium (355M)\"\n", "\n", "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n", "\n", "model = GPTModel(BASE_CONFIG)" ] }, { "cell_type": "code", "execution_count": 33, "id": "c2821403-605c-4071-a4ff-e23f4c9a11fd", "metadata": { "id": "c2821403-605c-4071-a4ff-e23f4c9a11fd" }, "outputs": [], "source": [ "model.load_state_dict(\n", " torch.load(\n", " \"gpt2-medium355M-sft.pth\",\n", " map_location=torch.device(\"cpu\"),\n", " weights_only=True\n", " )\n", ")\n", "model.eval();" ] }, { "cell_type": "markdown", "id": "61863bec-bd42-4194-b994-645bfe2df8be", "metadata": { "id": "61863bec-bd42-4194-b994-645bfe2df8be" }, "source": [ "- 在使用 DPO 训练加载的模型之前，让我们通过在一些样本数据上测试，确保微调后的模型已正确保存和加载：" ] }, { "cell_type": "code", "execution_count": 34, "id": "4357aec5-0db2-4d73-b37b-539cd8fa80a3", "metadata": { "id": "4357aec5-0db2-4d73-b37b-539cd8fa80a3" }, "outputs": [], "source": [ "prompt = \"\"\"Below is an instruction that describes a task. Write a response\n", "that appropriately completes the request.\n", "\n", "### Instruction:\n", "Convert the active sentence to passive: 'The chef cooks the meal every day.'\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 35, "id": "541e7988-38d3-47f6-bd52-9da6564479fa", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "541e7988-38d3-47f6-bd52-9da6564479fa", "outputId": "278f7ddf-37c2-4c3a-d069-c510ef6f8d7a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response\n", "that appropriately completes the request.\n", "\n", "### Instruction:\n", "Convert the active sentence to passive: 'The chef cooks the meal every day.'\n", "\n", "### Response:\n", "The meal is cooked every day by the chef.\n" ] } ], "source": [ "from previous_chapters import (\n", " generate,\n", " text_to_token_ids,\n", " token_ids_to_text\n", ")\n", "\n", "torch.manual_seed(123)\n", "\n", "token_ids = generate(\n", " model=model,\n", " idx=text_to_token_ids(prompt, tokenizer),\n", " max_new_tokens=35,\n", " context_size=BASE_CONFIG[\"context_length\"],\n", " eos_id=50256\n", ")\n", "\n", "response = token_ids_to_text(token_ids, tokenizer)\n", "print(response)" ] }, { "cell_type": "markdown", "id": "be87ed19-fded-4e56-8585-6c7c0367b354", "metadata": { "id": "be87ed19-fded-4e56-8585-6c7c0367b354" }, "source": [ "- 如上所示，模型给出了一个合理且正确的回应。\n", "- 如第七章所解释的，实际上，我们会对响应进行清理，只返回响应文本，并去除提示和提示样式（这类似于你在 ChatGPT 中熟悉的方式）。" ] }, { "cell_type": "code", "execution_count": 36, "id": "0c30c4e2-af84-4ab4-95d0-9641e32c1e7f", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0c30c4e2-af84-4ab4-95d0-9641e32c1e7f", "outputId": "70192bbe-fdf6-43eb-c673-f573f8c70156" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The meal is cooked every day by the chef.\n" ] } ], "source": [ "def extract_response(response_text, input_text):\n", " return response_text[len(input_text):].replace(\"### Response:\", \"\").strip()\n", "\n", "response = extract_response(response, prompt)\n", "print(response)" ] }, { "cell_type": "markdown", "id": "80442cb9-83b1-46b8-bad0-7d44297ca52d", "metadata": { "id": "80442cb9-83b1-46b8-bad0-7d44297ca52d" }, "source": [ "- 现在，我们已经快实现 DPO 部分了。\n", "- 正如在本笔记本开头所提到的，DPO 需要使用两个 LLM：一个是策略模型（即我们希望优化的模型），另一个是参考模型（即保持不变的原始模型）。\n", "- 在接下来的代码中，我们将 `model` 重命名为 `policy_model`，并实例化第二个模型，称之为 `reference_model`。" ] }, { "cell_type": "code", "execution_count": 37, "id": "5d88cc3a-312e-4b29-bc6d-de8354c1eb9f", "metadata": { "id": "5d88cc3a-312e-4b29-bc6d-de8354c1eb9f" }, "outputs": [], "source": [ "policy_model = model\n", "\n", "reference_model = GPTModel(BASE_CONFIG)\n", "reference_model.load_state_dict(\n", " torch.load(\n", " \"gpt2-medium355M-sft.pth\",\n", " map_location=torch.device(\"cpu\"),\n", " weights_only=True\n", " )\n", ")\n", "reference_model.eval()\n", "\n", "policy_model.to(device)\n", "reference_model.to(device);" ] }, { "cell_type": "markdown", "id": "9c6c1469-0038-4914-8aa5-15b1f81877cc", "metadata": { "id": "9c6c1469-0038-4914-8aa5-15b1f81877cc" }, "source": [ " \n", "# 4) 编写 DPO 损失函数" ] }, { "cell_type": "markdown", "id": "75dbe60c-e4ce-413e-beec-22eff0237d11", "metadata": { "id": "75dbe60c-e4ce-413e-beec-22eff0237d11" }, "source": [ "- 在前面的章节中，我们已经完成了模型加载和数据集准备，现在我们可以进入更有趣的部分，开始编写 DPO 损失函数。\n", "- 请注意，下面的 DPO 损失函数代码是基于论文 [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290) 中提出的方法。\n", "- 作为参考，下面再次展示了核心的 DPO 方程：\n", "\n", "\n", "\n", "- 在上面的方程中，\n", " - “期望值” $\\mathbb{E}$ 是统计学术语，表示随机变量（括号内的表达式）的平均值或均值；优化 $-\\mathbb{E}$ 可以更好地使模型与用户偏好对齐。\n", " - $\\pi_{\\theta}$ 变量是所谓的策略（这是从强化学习中借用的术语），表示我们希望优化的语言模型（LLM）；$\\pi_{ref}$ 是参考 LLM，通常是优化前的原始 LLM（在训练开始时，$\\pi_{\\theta}$ 和 $\\pi_{ref}$ 通常是相同的）。\n", " - $\\beta$ 是一个超参数，用于控制 $\\pi_{\\theta}$ 和参考模型之间的差异；增大 $\\beta$ 会增加两者在对数概率上的差异对整体损失函数的影响，从而增大两者模型之间的分歧。\n", " - 对数 sigmoid 函数 $\\sigma(\\centerdot)$ 将首选响应和拒绝响应的对数比率（对数几率函数内的项）转换为概率得分。\n", "- 在代码中，我们可以按以下方式实现 DPO 损失函数：" ] }, { "cell_type": "code", "execution_count": 38, "id": "38CsrrwJIZiV", "metadata": { "id": "38CsrrwJIZiV" }, "outputs": [], "source": [ "import torch.nn.functional as F\n", "\n", "def compute_dpo_loss(\n", " model_chosen_logprobs,\n", " model_rejected_logprobs,\n", " reference_chosen_logprobs,\n", " reference_rejected_logprobs,\n", " beta=0.1,\n", " ):\n", " \"\"\"Compute the DPO loss for a batch of policy and reference model log probabilities.\n", "\n", " Args:\n", " policy_chosen_logprobs: Log probabilities of the policy model for the chosen responses. Shape: (batch_size,)\n", " policy_rejected_logprobs: Log probabilities of the policy model for the rejected responses. Shape: (batch_size,)\n", " reference_chosen_logprobs: Log probabilities of the reference model for the chosen responses. Shape: (batch_size,)\n", " reference_rejected_logprobs: Log probabilities of the reference model for the rejected responses. Shape: (batch_size,)\n", " beta: Temperature parameter for the DPO loss; typically something in the range of 0.1 to 0.5. We ignore the reference model as beta -> 0.\n", " label_smoothing: conservativeness for DPO loss.\n", "\n", " Returns:\n", " A tuple of three tensors: (loss, chosen_rewards, rejected_rewards).\n", " \"\"\"\n", "\n", " model_logratios = model_chosen_logprobs - model_rejected_logprobs\n", " reference_logratios = reference_chosen_logprobs - reference_rejected_logprobs\n", " logits = model_logratios - reference_logratios\n", "\n", " # DPO（参见 https://arxiv.org/pdf/2305.18290.pdf 中的公式 7）\n", " losses = -F.logsigmoid(beta * logits)\n", "\n", " # 可选值，用于在训练期间跟踪进度\n", " chosen_rewards = (model_chosen_logprobs - reference_chosen_logprobs).detach()\n", " rejected_rewards = (model_rejected_logprobs - reference_rejected_logprobs).detach()\n", "\n", " # 使用 .mean() 对批次中的样本进行平均\n", " return losses.mean(), chosen_rewards.mean(), rejected_rewards.mean()" ] }, { "cell_type": "markdown", "id": "693be65b-38fc-4d18-bf53-a260a15436e1", "metadata": { "id": "693be65b-38fc-4d18-bf53-a260a15436e1" }, "source": [ "- 如果你熟悉对数运算，你会注意到我们有一个通用关系 $\\log\\left(\\frac{a}{b}\\right) = \\log a - \\log b$，我们在上面的代码中应用了这个公式。\n", "- 牢记这一点，让我们逐步解析一些步骤（稍后我们将使用一个单独的函数来计算 `logprobs`）。\n", "- 让我们从以下几行代码开始：\n", "\n", " ```python\n", " model_logratios = model_chosen_logprobs - model_rejected_logprobs\n", " reference_logratios = reference_chosen_logprobs - reference_rejected_logprobs\n", " ```\n", "\n", "- 上面的代码行计算了在政策模型和参考模型中，选择样本和拒绝样本的对数概率（logits）的差异（这就是 $\\log\\left(\\frac{a}{b}\\right) = \\log a - \\log b$ 的应用）：\n", "\n", "$$\\log \\left( \\frac{\\pi_\\theta (y_w \\mid x)}{\\pi_\\theta (y_l \\mid x)} \\right) \\quad \\text{和} \\quad \\log \\left( \\frac{\\pi_{\\text{ref}}(y_w \\mid x)}{\\pi_{\\text{ref}}(y_l \\mid x)} \\right)$$\n", "\n", "- 接下来，代码 `logits = model_logratios - reference_logratios` 计算了政策模型的对数比率与参考模型的对数比率之间的差异，即：\n", "\n", "$$\\beta \\log \\left( \\frac{\\pi_\\theta (y_w \\mid x)}{\\pi_{\\text{ref}} (y_w \\mid x)} \\right)\n", "- \\beta \\log \\left( \\frac{\\pi_\\theta (y_l \\mid x)}{\\pi_{\\text{ref}} (y_l \\mid x)} \\right)$$\n", "\n", "- 最后，`losses = -F.logsigmoid(beta * logits)` 使用对数- sigmoid 函数计算损失；在原始方程中，期望值内部的项是：\n", "\n", "$$\\log \\sigma \\left( \\beta \\log \\left( \\frac{\\pi_\\theta (y_w \\mid x)}{\\pi_{\\text{ref}} (y_w \\mid x)} \\right)\n", "- \\beta \\log \\left( \\frac{\\pi_\\theta (y_l \\mid x)}{\\pi_{\\text{ref}} (y_l \\mid x)} \\right) \\right)$$\n", "\n", "- 上面我们假设对数概率已经计算出来了；现在让我们定义一个 `compute_logprobs` 函数，用来计算传递给 `compute_dpo_loss` 函数中的对数概率，即值 $\\pi_\\theta (y_w \\mid x)$、$\\pi_\\theta (y_l \\mid x)$ 等等：" ] }, { "cell_type": "code", "execution_count": 39, "id": "71e6507b-d2e2-4469-86b9-f057b08b5df9", "metadata": { "id": "71e6507b-d2e2-4469-86b9-f057b08b5df9" }, "outputs": [], "source": [ "def compute_logprobs(logits, labels, selection_mask=None):\n", " \"\"\"\n", " Compute log probabilities.\n", "\n", " Args:\n", " logits: Tensor of shape (batch_size, num_tokens, vocab_size)\n", " labels: Tensor of shape (batch_size, num_tokens)\n", " selection_mask: Tensor for shape (batch_size, num_tokens)\n", "\n", " Returns:\n", " mean_log_prob: Mean log probability excluding padding tokens.\n", " \"\"\"\n", "\n", " # 标签是输入向右移动一位\n", " labels = labels[:, 1:].clone()\n", "\n", " # 截断 Logits 以匹配标签的token数量\n", " logits = logits[:, :-1, :]\n", "\n", " log_probs = F.log_softmax(logits, dim=-1)\n", "\n", " # 收集实际标签的对数概率\n", " selected_log_probs = torch.gather(\n", " input=log_probs,\n", " dim=-1,\n", " index=labels.unsqueeze(-1)\n", " ).squeeze(-1)\n", "\n", " if selection_mask is not None:\n", " mask = selection_mask[:, 1:].clone()\n", "\n", " # 应用掩码以过滤掉填充token\n", " selected_log_probs = selected_log_probs * mask\n", "\n", " # 计算排除填充token的平均对数概率\n", " # 这是在token上取平均，因此形状为 (batch_size, num_tokens)\n", " avg_log_prob = selected_log_probs.sum(-1) / mask.sum(-1)\n", "\n", " return avg_log_prob\n", "\n", " else:\n", " return selected_log_probs.mean(-1)" ] }, { "cell_type": "markdown", "id": "cf6a71ac-3fcc-44a4-befc-1c56bbd378d7", "metadata": { "id": "cf6a71ac-3fcc-44a4-befc-1c56bbd378d7" }, "source": [ "- 请注意，以上这个函数由于包含了 `torch.gather` 函数，刚开始可能看起来有点让人畏惧，但它与 PyTorch 的 `cross_entropy` 函数内部的实现非常相似。\n", "- 例如，考虑以下示例：" ] }, { "cell_type": "code", "execution_count": 40, "id": "59873470-464d-4be2-860f-cbb7ac2d80ba", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "59873470-464d-4be2-860f-cbb7ac2d80ba", "outputId": "8f7b47d4-73fe-4605-c17d-ad6cfd909a9b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor(1.4185) tensor(1.4185)\n" ] } ], "source": [ "# 示例数据\n", "logits = torch.tensor(\n", " [[2.0, 1.0, 0.1],\n", " [0.5, 2.5, 0.3]]) # 形状: (2, 3)\n", "targets = torch.tensor([0, 2]) # 形状: (2,)\n", "\n", "\n", "# 使用 torch.gather 手动计算损失\n", "log_softmax_logits = F.log_softmax(logits, dim=1) # 形状: (2, 3)\n", "selected_log_probs = torch.gather(\n", " input=log_softmax_logits,\n", " dim=1,\n", " index=targets.unsqueeze(1), # 形状: (2, 1)\n", ").squeeze(1) # 形状: (2,)\n", "manual_loss = -selected_log_probs.mean() # 在批次上取平均\n", "\n", "\n", "# PyTorch 损失函数\n", "cross_entropy_loss = F.cross_entropy(logits, targets)\n", "\n", "print(manual_loss, cross_entropy_loss)" ] }, { "cell_type": "markdown", "id": "f86d7add-f7ff-4a87-9193-7878c42bf0e7", "metadata": { "id": "f86d7add-f7ff-4a87-9193-7878c42bf0e7" }, "source": [ "- 如上所示，我们可以看到这两种实现是等效的，但让我们进一步聚焦于 `torch.gather` 的工作原理。\n", "- 现在考虑以下两个张量：" ] }, { "cell_type": "code", "execution_count": 41, "id": "508db6ba-cc40-479f-a996-2250cf862388", "metadata": { "id": "508db6ba-cc40-479f-a996-2250cf862388" }, "outputs": [], "source": [ "t = torch.tensor(\n", " [[1., 2.,],\n", " [3., 4.]]\n", ")\n", "\n", "m = torch.tensor(\n", " [[1, 1],\n", " [0, 1]]\n", ")" ] }, { "cell_type": "markdown", "id": "821cbf45-8fbb-47b7-bae8-6c3271e36979", "metadata": { "id": "821cbf45-8fbb-47b7-bae8-6c3271e36979" }, "source": [ "- 上述中，`t` 是我们想要选择的张量，`m` 是一个掩码，用于指定我们如何选择。\n", " - 例如，由于 `m` 的第一行包含 `[1, 1]`，它将两次选择 `t` 中索引位置为 `1` 的值，即值为 `2`。\n", " - `m` 的第二行 `[0, 1]` 选择了 `t` 第二行中索引位置为 `0` 和 `1` 的值，即 `3.` 和 `4.`。" ] }, { "cell_type": "code", "execution_count": 42, "id": "4fdN5q1YPAbM", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4fdN5q1YPAbM", "outputId": "e935e8ad-1519-4c4b-dbff-65adae0a15a4" }, "outputs": [ { "data": { "text/plain": [ "tensor([[2., 2.],\n", " [3., 4.]])" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "torch.gather(input=t, dim=-1, index=m)" ] }, { "cell_type": "markdown", "id": "d10eeaf4-f24b-4e79-916a-abedf74fe4a3", "metadata": { "id": "d10eeaf4-f24b-4e79-916a-abedf74fe4a3" }, "source": [ "- 换句话说，`torch.gather` 是一个选择函数。\n", "- 当我们之前计算损失时，我们使用它来获取与正确 token 对应的对数概率，这些 token 来自 50,256 个词汇中的正确选项。\n", "- “正确” tokens 是响应条目中给出的 tokens。\n", "\n", "- 关于上面的 `compute_logprobs` 函数，我们在这里使用 `torch.gather`，因为它比 `cross_entropy` 提供了更多的控制，但本质上思想相似。\n", "- 我们使用的 `selection_mask` 是为了可选地忽略提示和填充 tokens。\n", "- 然后，我们可以如下使用 `compute_logprobs` 函数来计算 `compute_dpo_loss` 损失函数的输入：" ] }, { "cell_type": "code", "execution_count": 43, "id": "dfa7a4db-eba0-47d8-ad6d-7b5e7676e318", "metadata": { "id": "dfa7a4db-eba0-47d8-ad6d-7b5e7676e318" }, "outputs": [], "source": [ "def compute_dpo_loss_batch(batch, policy_model, reference_model, beta):\n", " \"\"\"Compute the DPO loss on an input batch\"\"\"\n", "\n", " # 其中 policy_model(batch[\"chosen\"]) 是 logits\n", " policy_chosen_log_probas = compute_logprobs(\n", " logits=policy_model(batch[\"chosen\"]),\n", " labels=batch[\"chosen\"],\n", " selection_mask=batch[\"chosen_mask\"]\n", " )\n", " policy_rejected_log_probas = compute_logprobs(\n", " logits=policy_model(batch[\"rejected\"]),\n", " labels=batch[\"rejected\"],\n", " selection_mask=batch[\"rejected_mask\"]\n", " )\n", " \n", " with torch.no_grad():\n", " ref_chosen_log_probas = compute_logprobs(\n", " logits=reference_model(batch[\"chosen\"]),\n", " labels=batch[\"chosen\"],\n", " selection_mask=batch[\"chosen_mask\"]\n", " )\n", " ref_rejected_log_probas = compute_logprobs(\n", " logits=reference_model(batch[\"rejected\"]),\n", " labels=batch[\"rejected\"],\n", " selection_mask=batch[\"rejected_mask\"]\n", " )\n", " loss, chosen_rewards, rejected_rewards = compute_dpo_loss(\n", " model_chosen_logprobs=policy_chosen_log_probas,\n", " model_rejected_logprobs=policy_rejected_log_probas,\n", " reference_chosen_logprobs=ref_chosen_log_probas,\n", " reference_rejected_logprobs=ref_rejected_log_probas,\n", " beta=beta\n", " )\n", " return loss, chosen_rewards, rejected_rewards" ] }, { "cell_type": "markdown", "id": "b28caafb-f378-4332-a142-3e0f9ef67fbb", "metadata": { "id": "b28caafb-f378-4332-a142-3e0f9ef67fbb" }, "source": [ "- 上述函数适用于单个批次，例如：" ] }, { "cell_type": "code", "execution_count": 44, "id": "dd74fcc4-4280-41e9-9a22-838e85c84ee4", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "dd74fcc4-4280-41e9-9a22-838e85c84ee4", "outputId": "65a70828-7dd2-4f72-ffec-45aeaf8afad0" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(tensor(0.6931, device='cuda:0'), tensor(0., device='cuda:0'), tensor(0., device='cuda:0'))\n" ] } ], "source": [ "with torch.no_grad():\n", " loss = compute_dpo_loss_batch(batch, policy_model, reference_model, beta=0.1)\n", "print(loss)" ] }, { "cell_type": "markdown", "id": "b17429cd-2a00-41c8-9f16-38b1c9a5179f", "metadata": { "id": "b17429cd-2a00-41c8-9f16-38b1c9a5179f" }, "source": [ "- 我们扩展了这个函数，使其能够在数据加载器中处理指定的 `num_batches`：" ] }, { "cell_type": "code", "execution_count": 45, "id": "682e9ad5-c5de-4d1b-9e93-3918bf5d5302", "metadata": { "id": "682e9ad5-c5de-4d1b-9e93-3918bf5d5302" }, "outputs": [], "source": [ "def compute_dpo_loss_loader(data_loader, policy_model, reference_model, beta, num_batches=None):\n", " \"\"\"Apply compute_dpo_loss_batch to a whole data loader\"\"\"\n", "\n", " total_loss, total_chosen_rewards, total_rejected_rewards = 0., 0., 0.\n", " if len(data_loader) == 0:\n", " return float(\"nan\")\n", "\n", " elif num_batches is None:\n", " num_batches = len(data_loader)\n", " else:\n", " \n", " # 如果指定的批次数量超过了数据加载器中的批次数量，则减少批次数量以匹配数据加载器中的总批次数量\n", " num_batches = min(num_batches, len(data_loader))\n", " for i, batch in enumerate(data_loader):\n", " if i < num_batches:\n", " loss, chosen_rewards, rejected_rewards = compute_dpo_loss_batch(\n", " batch=batch,\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " beta=beta\n", " )\n", " total_loss += loss.item()\n", " total_chosen_rewards += chosen_rewards.item()\n", " total_rejected_rewards += rejected_rewards.item()\n", "\n", " else:\n", " break\n", "\n", " # 计算平均值\n", " total_loss /= num_batches\n", " total_chosen_rewards /= num_batches\n", " total_rejected_rewards /= num_batches\n", " return total_loss, total_chosen_rewards, total_rejected_rewards" ] }, { "cell_type": "markdown", "id": "852e4c09-d285-44d5-be12-d29769950cb6", "metadata": { "id": "852e4c09-d285-44d5-be12-d29769950cb6" }, "source": [ "- 为什么要指定 `num_batches`？这是出于效率考虑（因为每次计算整个数据集的损失会显著减慢训练速度）。\n", "\n", "- 最后，我们定义了一个便捷函数供后续训练函数使用；这个 `evaluate_dpo_loss_loader` 函数计算训练集和验证集数据加载器的 DPO 损失和奖励，以便进行日志记录：" ] }, { "cell_type": "code", "execution_count": 46, "id": "c3d214ec-49ba-4bf0-ac80-f90fa0d832e9", "metadata": { "id": "c3d214ec-49ba-4bf0-ac80-f90fa0d832e9" }, "outputs": [], "source": [ "def evaluate_dpo_loss_loader(policy_model, reference_model, train_loader, val_loader, beta, eval_iter):\n", " \"\"\"Compute the DPO loss for the training and validation dataset\"\"\"\n", "\n", " policy_model.eval()\n", " with torch.no_grad():\n", " train_loss, train_chosen_rewards, train_rejected_rewards = compute_dpo_loss_loader(\n", " data_loader=train_loader,\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " beta=beta,\n", " num_batches=eval_iter\n", " )\n", "\n", " val_loss, val_chosen_rewards, val_rejected_rewards = compute_dpo_loss_loader(\n", " data_loader=val_loader,\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " beta=beta,\n", " num_batches=eval_iter\n", " )\n", "\n", " res = {\n", " \"train_loss\": train_loss,\n", " \"train_chosen_reward\": train_chosen_rewards,\n", " \"train_rejected_reward\": train_rejected_rewards,\n", " \"val_loss\": val_loss,\n", " \"val_chosen_reward\": val_chosen_rewards,\n", " \"val_rejected_reward\": val_rejected_rewards\n", " }\n", "\n", " policy_model.train()\n", " return res" ] }, { "cell_type": "markdown", "id": "6e95ed92-6743-4f13-8b91-0fbf2e540de1", "metadata": { "id": "6e95ed92-6743-4f13-8b91-0fbf2e540de1" }, "source": [ "- 在本节中，我们涵盖了很多内容，简要回顾如下：\n", " - 流程是：通过模型计算 `logits` → 从 logits 计算 `compute_logprobs` → 从对数概率计算 `compute_dpo_loss`\n", " - 我们有 `compute_dpo_loss_batch` 函数，简化了上述过程\n", " - `compute_dpo_loss_loader` 工具函数将 `compute_dpo_loss_batch` 函数应用于数据加载器\n", " - `evaluate_dpo_loss_loader` 函数将 `compute_dpo_loss_batch` 应用于训练集和验证集的数据加载器，以便进行日志记录" ] }, { "cell_type": "markdown", "id": "cb8a8f18-536e-4d83-a0d0-ac518a85f157", "metadata": { "id": "cb8a8f18-536e-4d83-a0d0-ac518a85f157" }, "source": [ " \n", "# 5) 模型训练" ] }, { "cell_type": "markdown", "id": "4b11d63d-3ddc-4070-9b2b-5ca0edb08d0c", "metadata": { "id": "4b11d63d-3ddc-4070-9b2b-5ca0edb08d0c" }, "source": [ "- 在上一节设置了 DPO 损失函数后，我们现在可以开始训练模型了。\n", "- 请注意，这个训练函数与我们在预训练和指令微调时使用的函数相同，唯一的区别是：\n", " - 我们将交叉熵损失替换为新的 DPO 损失函数\n", " - 我们还跟踪奖励值和奖励边际，这些通常用于 RLHF 和 DPO 环境中，以便跟踪训练进展" ] }, { "cell_type": "code", "execution_count": 47, "id": "f90d9325-77b2-417f-88ff-0a5174889413", "metadata": { "id": "f90d9325-77b2-417f-88ff-0a5174889413" }, "outputs": [], "source": [ "from previous_chapters import generate_and_print_sample\n", "\n", "\n", "def train_model_dpo_simple(\n", " policy_model, reference_model, train_loader, val_loader,\n", " optimizer, num_epochs, beta,\n", " eval_freq, eval_iter, start_context, tokenizer\n", "):\n", "\n", " # 初始化列表以跟踪损失和已处理的token\n", " tracking = {\n", " \"train_losses\": [],\n", " \"train_chosen_rewards\": [],\n", " \"train_rejected_rewards\": [],\n", " \"val_losses\": [],\n", " \"val_chosen_rewards\": [],\n", " \"val_rejected_rewards\": [],\n", " \"tokens_seen\": []\n", " }\n", " tokens_seen, global_step = 0, -1\n", "\n", " # 主训练循环\n", " for epoch in range(num_epochs):\n", " policy_model.train() # 将模型设置为训练模式\n", "\n", " for batch_idx, batch in enumerate(train_loader):\n", "\n", " optimizer.zero_grad() # 重置上一批次的损失梯度\n", "\n", " loss, chosen_rewards, rejected_rewards = compute_dpo_loss_batch(\n", " batch=batch,\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " beta=beta\n", " )\n", "\n", " loss.backward() # 计算损失梯度\n", " optimizer.step() # 使用损失梯度更新模型权重\n", "\n", " tokens_seen += batch[\"chosen\"].numel()\n", " global_step += 1\n", "\n", " # 可选的评估步骤\n", " if global_step % eval_freq == 0:\n", " res = evaluate_dpo_loss_loader(\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " train_loader=train_loader,\n", " val_loader=val_loader,\n", " beta=beta,\n", " eval_iter=eval_iter\n", " )\n", " tracking[\"train_losses\"].append(res[\"train_loss\"])\n", " tracking[\"train_chosen_rewards\"].append(res[\"train_chosen_reward\"])\n", " tracking[\"train_rejected_rewards\"].append(res[\"train_rejected_reward\"])\n", " tracking[\"val_losses\"].append(res[\"val_loss\"])\n", " tracking[\"val_chosen_rewards\"].append(res[\"val_chosen_reward\"])\n", " tracking[\"val_rejected_rewards\"].append(res[\"val_rejected_reward\"])\n", " tracking[\"tokens_seen\"].append(tokens_seen)\n", " train_reward_margin = res[\"train_chosen_reward\"] - res[\"train_rejected_reward\"]\n", " val_reward_margin = res[\"val_chosen_reward\"] - res[\"val_rejected_reward\"]\n", "\n", " print(\n", " f\"Ep {epoch+1} (Step {global_step:06d}): \"\n", " f\"Train loss {res['train_loss']:.3f}, Val loss {res['val_loss']:.3f}, \"\n", " f\"Train reward margins {train_reward_margin:.3f}, \"\n", " f\"Val reward margins {val_reward_margin:.3f}\"\n", " )\n", "\n", " # 在每个训练周期后打印示例文本\n", " generate_and_print_sample(\n", " model=model,\n", " tokenizer=tokenizer,\n", " device=loss.device,\n", " start_context=start_context\n", " )\n", "\n", " return tracking" ] }, { "cell_type": "markdown", "id": "820d4904-f819-4d62-bfb4-85cf28863683", "metadata": { "id": "820d4904-f819-4d62-bfb4-85cf28863683" }, "source": [ "- 在开始训练之前，让我们先打印初始的损失和reward：" ] }, { "cell_type": "code", "execution_count": 48, "id": "d53210c5-6d9c-46b0-af22-ee875c2806c5", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "d53210c5-6d9c-46b0-af22-ee875c2806c5", "outputId": "8b1d2b39-16c5-4b99-e920-5b33d3c0f34d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training loss: 0.6931471824645996\n", "Validation loss: 0.6931471824645996\n", "Train reward margin: 0.0\n", "Val reward margin: 0.0\n" ] } ], "source": [ "torch.manual_seed(123) # 由于数据加载器中的随机打乱，为了可重复性，设置随机种子\n", "\n", "res = evaluate_dpo_loss_loader(\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " train_loader=train_loader,\n", " val_loader=val_loader,\n", " beta=0.1,\n", " eval_iter=5\n", ")\n", "\n", "print(\"Training loss:\", res[\"train_loss\"])\n", "print(\"Validation loss:\", res[\"val_loss\"])\n", "\n", "print(\"Train reward margin:\", res[\"train_chosen_reward\"] - res[\"train_rejected_reward\"])\n", "print(\"Val reward margin:\", res[\"val_chosen_reward\"] - res[\"val_rejected_reward\"])" ] }, { "cell_type": "markdown", "id": "4a006e91-df94-43ca-8025-1ba791e37bc4", "metadata": { "id": "4a006e91-df94-43ca-8025-1ba791e37bc4" }, "source": [ "- 用三个例子测试下模型:" ] }, { "cell_type": "code", "execution_count": 49, "id": "q4Ro9DrBa7zH", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "q4Ro9DrBa7zH", "outputId": "b974d4bd-b92a-4a2a-bb7a-5a2a0d1eca11" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Convert the active sentence to passive: 'The chef cooks the meal every day.'\n", "\n", "Correct response:\n", ">> The meal is cooked by the chef every day.\n", "\n", "Model response:\n", ">> The meal is cooked every day by the chef.\n", "\n", "-------------------------------------\n", "\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Classify an input string as either a noun or a verb.\n", "\n", "### Input:\n", "Dance\n", "\n", "Correct response:\n", ">> 'Dance' can be classified as a verb.\n", "\n", "Model response:\n", ">> \"Dance\" can be classified as a verb.\n", "\n", "-------------------------------------\n", "\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Rewrite the sentence using a metaphor.\n", "\n", "### Input:\n", "The book is very interesting.\n", "\n", "Correct response:\n", ">> The book is a page-turner.\n", "\n", "Model response:\n", ">> The book is a treat.\n", "\n", "-------------------------------------\n", "\n" ] } ], "source": [ "torch.manual_seed(123)\n", "\n", "\n", "for entry in val_data[:3]:\n", "\n", " input_text = format_input(entry)\n", "\n", " token_ids = generate(\n", " model=model,\n", " idx=text_to_token_ids(input_text, tokenizer).to(device),\n", " max_new_tokens=256,\n", " context_size=BASE_CONFIG[\"context_length\"],\n", " eos_id=50256\n", " )\n", " generated_text = token_ids_to_text(token_ids, tokenizer)\n", " response_text = (\n", " generated_text[len(input_text):]\n", " .replace(\"### Response:\", \"\")\n", " .strip()\n", ")\n", "\n", " print(input_text)\n", " print(f\"\\nCorrect response:\\n>> {entry['output']}\")\n", " print(f\"\\nModel response:\\n>> {response_text.strip()}\")\n", " print(\"\\n-------------------------------------\\n\")" ] }, { "cell_type": "markdown", "id": "ac2386ae-5c4c-448e-bfbf-4ec0604b171e", "metadata": { "id": "ac2386ae-5c4c-448e-bfbf-4ec0604b171e" }, "source": [ "- **上方展示了原始模型的响应**。 \n", "- **DPO 训练的目标** 是 **引导模型进行轻微的风格调整**， \n", " 例如 **生成与原始响应相似但更礼貌的回答**。 \n", "\n", "- **在正式启动训练之前**，请注意以下几点设置：\n", " - **我们仅优化政策模型（Policy Model）**， \n", " 因此只将其参数传递给 `AdamW` 优化器； \n", " **参考模型（Reference Model）不参与训练**。 \n", " - **训练仅进行 1 个 Epoch**， \n", " **因为 DPO 训练容易“塌陷”（Collapse）**， \n", " **损失可能继续优化，但模型可能开始生成无意义的文本**。 \n", " - **DPO 训练时，建议使用较小的学习率（Learning Rate）**。 \n", " - **β（Beta）值** 可从 **0.1 增加至 0.5**， \n", " **以减少 DPO 训练的影响**； \n", " **我们这里使用 0.1，以确保调整效果更明显**。 \n", " - **训练时长**：\n", " - **在 A100 GPU 上约需 2 分钟**。 \n", " - **在 L4 GPU 上约需 4 分钟**。 \n", " - **在 M3 MacBook Air 上约需 30 分钟**。 \n" ] }, { "cell_type": "code", "execution_count": 50, "id": "54b739be-871e-4c97-bf14-ffd2c58e1311", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "54b739be-871e-4c97-bf14-ffd2c58e1311", "outputId": "d98b08b0-c325-411e-a1a4-05e7403f0345" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Ep 1 (Step 000000): Train loss 0.692, Val loss 0.693, Train reward margins 0.019, Val reward margins 0.009\n", "Ep 1 (Step 000005): Train loss 0.690, Val loss 0.691, Train reward margins 0.070, Val reward margins 0.052\n", "Ep 1 (Step 000010): Train loss 0.687, Val loss 0.688, Train reward margins 0.126, Val reward margins 0.108\n", "Ep 1 (Step 000015): Train loss 0.676, Val loss 0.685, Train reward margins 0.362, Val reward margins 0.173\n", "Ep 1 (Step 000020): Train loss 0.676, Val loss 0.680, Train reward margins 0.351, Val reward margins 0.264\n", "Ep 1 (Step 000025): Train loss 0.666, Val loss 0.676, Train reward margins 0.564, Val reward margins 0.359\n", "Ep 1 (Step 000030): Train loss 0.672, Val loss 0.672, Train reward margins 0.456, Val reward margins 0.441\n", "Ep 1 (Step 000035): Train loss 0.663, Val loss 0.669, Train reward margins 0.658, Val reward margins 0.511\n", "Ep 1 (Step 000040): Train loss 0.666, Val loss 0.666, Train reward margins 0.597, Val reward margins 0.574\n", "Ep 1 (Step 000045): Train loss 0.648, Val loss 0.662, Train reward margins 0.982, Val reward margins 0.660\n", "Ep 1 (Step 000050): Train loss 0.648, Val loss 0.659, Train reward margins 0.993, Val reward margins 0.734\n", "Ep 1 (Step 000055): Train loss 0.647, Val loss 0.656, Train reward margins 1.014, Val reward margins 0.799\n", "Ep 1 (Step 000060): Train loss 0.652, Val loss 0.653, Train reward margins 0.893, Val reward margins 0.870\n", "Ep 1 (Step 000065): Train loss 0.631, Val loss 0.650, Train reward margins 1.361, Val reward margins 0.948\n", "Ep 1 (Step 000070): Train loss 0.618, Val loss 0.646, Train reward margins 1.699, Val reward margins 1.038\n", "Ep 1 (Step 000075): Train loss 0.617, Val loss 0.642, Train reward margins 1.733, Val reward margins 1.121\n", "Ep 1 (Step 000080): Train loss 0.592, Val loss 0.639, Train reward margins 2.333, Val reward margins 1.194\n", "Ep 1 (Step 000085): Train loss 0.610, Val loss 0.636, Train reward margins 1.907, Val reward margins 1.275\n", "Ep 1 (Step 000090): Train loss 0.650, Val loss 0.633, Train reward margins 0.964, Val reward margins 1.353\n", "Ep 1 (Step 000095): Train loss 0.607, Val loss 0.630, Train reward margins 1.962, Val reward margins 1.423\n", "Ep 1 (Step 000100): Train loss 0.600, Val loss 0.627, Train reward margins 2.127, Val reward margins 1.500\n", "Ep 1 (Step 000105): Train loss 0.590, Val loss 0.624, Train reward margins 2.458, Val reward margins 1.564\n", "Ep 1 (Step 000110): Train loss 0.607, Val loss 0.622, Train reward margins 1.976, Val reward margins 1.621\n", "Ep 1 (Step 000115): Train loss 0.621, Val loss 0.620, Train reward margins 1.605, Val reward margins 1.682\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Rewrite the sentence using a metaphor. ### Input: The book is very interesting. ### Response: The book is a treat.<|endoftext|>The following is an instruction that describes a task. Write a response that appropriately completes the request. ### Input: The assignment was written by the student. ### Response\n", "Training completed in 1.69 minutes.\n" ] } ], "source": [ "import time\n", "\n", "start_time = time.time()\n", "\n", "torch.manual_seed(123)\n", "\n", "\n", "optimizer = torch.optim.AdamW(policy_model.parameters(), lr=5e-6, weight_decay=0.01)\n", "\n", "num_epochs = 1\n", "tracking = train_model_dpo_simple(\n", " policy_model=policy_model,\n", " reference_model=reference_model,\n", " train_loader=train_loader,\n", " val_loader=val_loader,\n", " optimizer=optimizer,\n", " num_epochs=num_epochs,\n", " beta=0.1, # 取值在0.1到0.5之间\n", " eval_freq=5,\n", " eval_iter=5,\n", " start_context=format_input(val_data[2]),\n", " tokenizer=tokenizer\n", ")\n", "\n", "end_time = time.time()\n", "execution_time_minutes = (end_time - start_time) / 60\n", "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")" ] }, { "cell_type": "markdown", "id": "eba8ea88-8771-4eb9-855d-2fe1ca2dc2fa", "metadata": { "id": "eba8ea88-8771-4eb9-855d-2fe1ca2dc2fa" }, "source": [ "- **从上面的跟踪结果可以看出，损失（Loss）逐步优化**。 \n", "- **同时，奖励边际（Reward Margins）** —— 即 **偏好响应（Chosen）与非偏好响应（Rejected）奖励之间的差距**， \n", " 也在 **逐步提升**，这是一个 **积极的信号**。 \n", "- **接下来，我们更具体地分析这些结果**。 " ] }, { "cell_type": "markdown", "id": "11e23989-92bd-4ac2-a4bc-65d4c7ac334e", "metadata": { "id": "11e23989-92bd-4ac2-a4bc-65d4c7ac334e" }, "source": [ " \n", "# 6) 结果分析" ] }, { "cell_type": "markdown", "id": "66d7d5fe-c617-45cb-8ea9-ddc7baa22654", "metadata": { "id": "66d7d5fe-c617-45cb-8ea9-ddc7baa22654" }, "source": [ "- **首先，我们通过绘制 DPO 损失曲线（Loss Curve）来分析训练结果：** " ] }, { "cell_type": "code", "execution_count": 51, "id": "8ddcc66f-cd7c-4f46-96ea-af919ea1a199", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 307 }, "id": "8ddcc66f-cd7c-4f46-96ea-af919ea1a199", "outputId": "c7164b26-8d32-41d1-8c6a-ab835d58d4c5" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABs/klEQVR4nO3deVhU5dvA8e/MsO8gsimLKCpuiCCIqGXillmapZmVWtkvxS1b1NdS27TSzErTtFIrTcvSzH3Jpdz3FXFBBRdARXZZ57x/jAySqIDADHh/rutcwpnnnHM/jHDPec6zqBRFURBCCCGEUVIbOgAhhBBC3J0kaiGEEMKISaIWQgghjJgkaiGEEMKISaIWQgghjJgkaiGEEMKISaIWQgghjJgkaiGEEMKISaIWQgghjJgkaiGqsfPnz6NSqTh06JChQxFClJEkaiGMnEqluuc2ceJEQ4cohKhAJoYOQAhxb1euXNF/vWTJEsaPH090dLR+n42NjSHCEkJUErmjFsLIubm56Td7e3tUKpX+excXF6ZNm0bt2rUxNzenefPmrF279q7nys/P5+WXX6Zhw4bExsYC8Oeff9KiRQssLCzw9fXl/fffJy8vT3+MSqXiu+++o2fPnlhZWeHn58eKFSv0r9+4cYN+/fpRs2ZNLC0t8fPzY968eXeNYenSpTRt2hRLS0tq1KhBREQEGRkZ+te/++47/P39sbCwoGHDhnzzzTdFjo+Li6N37944ODjg5OTEU089xfnz5/WvDxgwgB49ejB16lTc3d2pUaMGkZGR5ObmlvhnLoRRUYQQVca8efMUe3t7/ffTpk1T7OzslF9++UU5efKk8s477yimpqbKqVOnFEVRlHPnzimAcvDgQSUrK0vp2bOnEhgYqCQmJiqKoijbtm1T7OzslPnz5ytnz55V1q9fr/j4+CgTJ07UXwNQateurSxatEg5ffq0Mnz4cMXGxka5fv26oiiKEhkZqTRv3lzZu3evcu7cOWXDhg3KihUrio3/8uXLiomJiTJt2jTl3LlzypEjR5SZM2cqaWlpiqIoys8//6y4u7srv//+uxITE6P8/vvvipOTkzJ//nxFURQlJydH8ff3V15++WXlyJEjyokTJ5Tnn39eadCggZKdna0oiqL0799fsbOzU15//XUlKipK+euvvxQrKytlzpw55ftmCFFJJFELUYX8N1F7eHgoH3/8cZEyLVu2VIYMGaIoSmGi/ueff5QOHToobdq0UZKTk/VlO3TooEyaNKnI8T/99JPi7u6u/x5Q3n33Xf336enpCqCsWbNGURRF6d69uzJw4MASxb9//34FUM6fP1/s63Xr1lUWLVpUZN+HH36ohIWF6WNr0KCBotVq9a9nZ2crlpaWyrp16xRF0SVqb29vJS8vT1/m2WefVfr06VOiGIUwNvKMWogqKjU1lcuXLxMeHl5kf3h4OIcPHy6yr2/fvtSuXZu///4bS0tL/f7Dhw+zfft2Pv74Y/2+/Px8srKyyMzMxMrKCoBmzZrpX7e2tsbOzo7ExEQABg8eTK9evThw4ACdOnWiR48etG7dutiYAwIC6NChA02bNqVz58506tSJZ555BkdHRzIyMjh79iyvvPIKgwYN0h+Tl5eHvb29Pt4zZ85ga2tb5LxZWVmcPXtW/33jxo3RaDT6793d3Tl69Og9fppCGC9J1EI8BB5//HF+/vlndu7cyWOPPabfn56ezvvvv8/TTz99xzEWFhb6r01NTYu8plKp0Gq1AHTt2pULFy6wevVqNmzYQIcOHYiMjGTq1Kl3nFOj0bBhwwZ27NjB+vXr+frrrxk3bhy7d+/WfyiYO3cuoaGhdxxXEG9QUBALFy6849w1a9YsUbxCVDWSqIWoouzs7PDw8GD79u088sgj+v3bt28nJCSkSNnBgwfTpEkTnnzySVatWqUv36JFC6Kjo6lXr94DxVKzZk369+9P//79adu2LW+//XaxiRp0STM8PJzw8HDGjx+Pt7c3y5YtY9SoUXh4eBATE0O/fv2KPbZFixYsWbIEFxcX7OzsHihmIaoKSdRCVGFvv/02EyZMoG7dujRv3px58+Zx6NChYu84hw0bRn5+Pk888QRr1qyhTZs2jB8/nieeeAIvLy+eeeYZ1Go1hw8f5tixY3z00UclimH8+PEEBQXRuHFjsrOzWblyJf7+/sWW3b17N5s2baJTp064uLiwe/durl69qi///vvvM3z4cOzt7enSpQvZ2dns27ePGzduMGrUKPr168eUKVN46qmn+OCDD6hduzYXLlzgjz/+4J133qF27dpl/2EKYaQkUQtRhQ0fPpyUlBTefPNNEhMTadSoEStWrMDPz6/Y8iNHjkSr1fL444+zdu1aOnfuzMqVK/nggw/49NNPMTU1pWHDhrz66qsljsHMzIyxY8dy/vx5LC0tadu2LYsXLy62rJ2dHdu2bWP69Omkpqbi7e3N559/TteuXQF49dVXsbKyYsqUKbz99ttYW1vTtGlTRo4cCYCVlRXbtm1j9OjRPP3006SlpVGrVi06dOggd9ii2lIpiqIYOgghhBBCFE8mPBFCCCGMmCRqIYQQwohJohZCCCGMmCRqIYQQwohJohZCCCGMmCRqIYQQwohJoi5nM2fOxMfHBwsLC0JDQ9mzZ0+lXn/btm10794dDw8PVCoVy5cvL/K6oiiMHz8ed3d3LC0tiYiI4PTp00XKJCUl0a9fP+zs7HBwcOCVV14hPT29SJkjR47Qtm1bLCws8PT05LPPPrsjlt9++42GDRtiYWFB06ZNWb16dYnrMXnyZFq2bImtrS0uLi706NGjyBrMoJvfOTIykho1amBjY0OvXr1ISEgoUiY2NpZu3bphZWWFi4sLb7/9dpElHAG2bNlCixYtMDc3p169esyfP/+OeMr6vs6aNYtmzZphZ2eHnZ0dYWFhrFmzpkrVoTiffPIJKpVKP765KtVl4sSJqFSqIlvDhg2rXD0ALl26xAsvvECNGjWwtLSkadOm7Nu3T/96Vfl99/HxueM9UalUREZGAlXrPakQhl0TpHpZvHixYmZmpvzwww/K8ePHlUGDBikODg5KQkJCpcWwevVqZdy4ccoff/yhAMqyZcuKvP7JJ58o9vb2yvLly5XDhw8rTz75pFKnTh3l5s2b+jJdunRRAgIClF27din//POPUq9ePaVv377611NSUhRXV1elX79+yrFjx5RffvlFsbS0VL799lt9me3btysajUb57LPPlBMnTijvvvuuYmpqqhw9erRE9ejcubMyb9485dixY8qhQ4eUxx9/XPHy8lLS09P1ZV5//XXF09NT2bRpk7Jv3z6lVatWSuvWrfWv5+XlKU2aNFEiIiKUgwcPKqtXr1acnZ2VsWPH6svExMQoVlZWyqhRo5QTJ04oX3/9taLRaJS1a9fqyzzI+7pixQpl1apVyqlTp5To6Gjl//7v/xRTU1Pl2LFjVaYO/7Vnzx7Fx8dHadasmTJixAj9/qpSlwkTJiiNGzdWrly5ot+uXr1a5eqRlJSkeHt7KwMGDFB2796txMTEKOvWrVPOnDmjL1NVft8TExOLvB8bNmxQAGXz5s1V6j2pKJKoy1FISIgSGRmp/z4/P1/x8PBQJk+ebJB4/puotVqt4ubmpkyZMkW/Lzk5WTE3N1d++eUXRVEU5cSJEwqg7N27V19mzZo1ikqlUi5duqQoiqJ88803iqOjo379X0VRlNGjRysNGjTQf9+7d2+lW7duReIJDQ1V/ve//5WpLomJiQqgbN26VR+3qamp8ttvv+nLREVFKYCyc+dORVF0H1rUarUSHx+vLzNr1izFzs5OH/s777yjNG7cuMi1+vTpo3Tu3Fn/fXm/r46Ojsp3331XJeuQlpam+Pn5KRs2bFAeeeQRfaKuSnWZMGGCEhAQUOxrVakeo0ePVtq0aXPX16vy7/uIESOUunXrKlqttkq9JxVFmr7LSU5ODvv37yciIkK/T61WExERwc6dOw0YWaFz584RHx9fJEZ7e3tCQ0P1Me7cuRMHBweCg4P1ZSIiIlCr1ezevVtfpl27dpiZmenLdO7cmejoaG7cuKEvc/t1CsqU9WeRkpICgJOTEwD79+8nNze3yDUaNmyIl5dXkbo0bdoUV1fXIjGkpqZy/PjxEsVZnu9rfn4+ixcvJiMjg7CwsCpZh8jISLp163bH9apaXU6fPo2Hhwe+vr7069eP2NjYKlePFStWEBwczLPPPouLiwuBgYHMnTtX/3pV/X3Pycnh559/5uWXX0alUlWp96SiSKIuJ9euXSM/P7/IfxQAV1dX4uPjDRRVUQVx3CvG+Ph4XFxcirxuYmKCk5NTkTLFneP2a9ytTFl+FlqtlpEjRxIeHk6TJk305zczM8PBweGedSlrnKmpqdy8ebNc3tejR49iY2ODubk5r7/+OsuWLaNRo0ZVqg4Aixcv5sCBA0yePPmO16pSXUJDQ5k/fz5r165l1qxZnDt3jrZt25KWllal6hETE8OsWbPw8/Nj3bp1DB48mOHDh7NgwYIisVS13/fly5eTnJzMgAED9OeuKu9JRZFFOYTRi4yM5NixY/z777+GDqVMGjRowKFDh0hJSWHp0qX079+frVu3GjqsUomLi2PEiBFs2LChyDrVVVHBAiAAzZo1IzQ0FG9vb3799VcsLS0NGFnpaLVagoODmTRpEgCBgYEcO3aM2bNn079/fwNHV3bff/89Xbt2xcPDw9ChGA25oy4nzs7OaDSaO3oiJiQk4ObmZqCoiiqI414xurm5kZiYWOT1vLw8kpKSipQp7hy3X+NuZUr7sxg6dCgrV65k8+bNRZYwdHNzIycnh+Tk5HvWpaxx2tnZYWlpWS7vq5mZGfXq1SMoKIjJkycTEBDAl19+WaXqsH//fhITE2nRogUmJiaYmJiwdetWvvrqK0xMTHB1da0ydfkvBwcH6tevz5kzZ6rUe+Lu7k6jRo2K7PP399c341fF3/cLFy6wcePGIqu3VaX3pKJIoi4nZmZmBAUFsWnTJv0+rVbLpk2bCAsLM2BkherUqYObm1uRGFNTU9m9e7c+xrCwMJKTk9m/f7++zN9//41WqyU0NFRfZtu2beTm5urLbNiwgQYNGuDo6Kgvc/t1CsqU9GehKApDhw5l2bJl/P3339SpU6fI60FBQZiamha5RnR0NLGxsUXqcvTo0SJ/iDZs2ICdnZ3+D9z94qyI91Wr1ZKdnV2l6tChQweOHj3KoUOH9FtwcDD9+vXTf11V6vJf6enpnD17Fnd39yr1noSHh98xZPHUqVN4e3sDVev3vcC8efNwcXGhW7du+n1V6T2pMAbtylbNLF68WDE3N1fmz5+vnDhxQnnttdcUBweHIj0RK1paWppy8OBB5eDBgwqgTJs2TTl48KBy4cIFRVF0wzUcHByUP//8Uzly5Ijy1FNPFTtcIzAwUNm9e7fy77//Kn5+fkWGayQnJyuurq7Kiy++qBw7dkxZvHixYmVldcdwDRMTE2Xq1KlKVFSUMmHChFIN1xg8eLBib2+vbNmypciwjczMTH2Z119/XfHy8lL+/vtvZd++fUpYWJgSFhamf71gyEanTp2UQ4cOKWvXrlVq1qxZ7JCNt99+W4mKilJmzpxZ7JCNsr6vY8aMUbZu3aqcO3dOOXLkiDJmzBhFpVIp69evrzJ1uJvbe31Xpbq8+eabypYtW5Rz584p27dvVyIiIhRnZ2clMTGxStVjz549iomJifLxxx8rp0+fVhYuXKhYWVkpP//8s75MVfl9VxRdD2svLy9l9OjRd7xWVd6TiiKJupx9/fXXipeXl2JmZqaEhIQou3btqtTrb968WQHu2Pr3768oim7Ixnvvvae4uroq5ubmSocOHZTo6Ogi57h+/brSt29fxcbGRrGzs1MGDhyopKWlFSlz+PBhpU2bNoq5ublSq1Yt5ZNPPrkjll9//VWpX7++YmZmpjRu3FhZtWpVietRXB0AZd68efoyN2/eVIYMGaI4OjoqVlZWSs+ePZUrV64UOc/58+eVrl27KpaWloqzs7Py5ptvKrm5uXf8zJo3b66YmZkpvr6+Ra5RoKzv68svv6x4e3srZmZmSs2aNZUOHTrok3RVqcPd/DdRV5W69OnTR3F3d1fMzMyUWrVqKX369Cky9riq1ENRFOWvv/5SmjRpopibmysNGzZU5syZU+T1qvL7riiKsm7dOgW4Iz5FqVrvSUVQKYqiGORWXgghhBD3Jc+ohRBCCCMmiVoIIYQwYpKohRBCCCMmiVoIIYQwYpKohRBCCCMmiVoIIYQwYpKoK0B2djYTJ04kOzvb0KE8EKmH8akudaku9YDqU5fqUg+oXnUBkHHUFSA1NRV7e3tSUlKws7MzdDhlJvUwPtWlLtWlHlB96lJd6gHVqy4gd9RCCCGEUZNELYQQQhgxWY+6GHl5eRw8eBBXV1fU6tJ/lklLSwPg0qVLpKamlnd4lUbqYXyqS12qSz2g+tSlutQDqkZdtFotCQkJBAYGYmJy71Qsz6iLsXfvXkJCQgwdhhBCiGpuz549tGzZ8p5l5I66GK6uroDuB+ju7m7gaIQQQlQ3V65cISQkRJ9v7kUSdTEKmrvd3d2pXbu2gaMRQghRXZXk8ap0JhNCCCGMmCRqIYQQwohJohZCCCGMmDyjFkKI2+Tn55Obm2voMEQVZ2pqikajKZdzSaKuaFotXD8NNfygDGOyhRCVQ1EU4uPjSU5ONnQooppwcHDAzc0NlUr1QOeRRF2Bbubks+DPtbx+/HkUcztUtYKgdstbWzBYORk6RCHELQVJ2sXFBSsrqwf+4yoeXoqikJmZSWJiIsADD/OVRF2B9l1IYs/BA/Q3NcMyOxViNuu2Ak6+uqRdK1iXuF2bgImZ4QIW4iGVn5+vT9I1atQwdDiiGrC0tAQgMTERFxeXB2oGl0RdgWpYm+Ma3IMnzrbGIukkzdVnCFSfobnqDPXUlyEpRrcdWaI7QGMOHs3BMwQ6fgjyiV6ISlHwTNrKysrAkYjqpOD/U25uriRqY9XIw47JTzcF4EpKa3aevc7Os9eZHnOd1BtXaa4+S3PVWQLVpwlUn8EhPwPidpOZdgPNY+9jbnLrjf3nc7CuCf7dwdLRgDUSonqT5m5Rnsrr/5Mk6kribm/J0y1q83QL3UxncUmZ7Iy5zq6z1/kt5jpXUm7io4onUHUGJVHFmonrCfJ2pI23Na/v/gx1fpauibwgUacngrkdmFoYsFZCCCEqmiRqA/F0ssLTyYrewZ4oisL565m6O+6YIHaevU52ejY7zl7n6Nk4ckweJ9DkAks3ZfFIg4u0q++My7qxcHIV1GkL9SKgbgeoUVeay4UQD8zHx4eRI0cycuTIEpXfsmUL7du358aNGzg4OFRYXPPnz2fkyJEPXc98SdRGQKVSUcfZmjrO1jwf6oWiKJy9ms7Os9fZcfY6P5x5jtSsPDhyhb+OXAHgb+v9+ObfhNPrdRuAgzfU66BL3HXagbmtAWslhKho92tanTBhAhMnTiz1effu3Yu1tXWJy7du3ZorV65gb29f6muJ+5NEbYRUKhX1XGyp52LLi2E+5OVrORSXzNZTV9kSfZWjl1J4LOMjGqjieER9mMdMjxGkOolp8gXY94NuU5uAZyuo9xjU6whuTeVuW4hq5sqVK/qvlyxZwvjx44mOjtbvs7Gx0X+tKAr5+fn3XfsYoGbNmqWKw8zMDDc3t1IdI0pOZuCoAkw0aoJ9nHizUwP+GtaGfe9GMK13cxo0a8Vv5k/zXNZYmt38loE5bzM/rxOX1O6gzYML/8KmD+DbtvBlAMTtNXRVhBDlyM3NTb/Z29ujUqn03588eRJbW1vWrFlDUFAQ5ubm/Pvvv5w9e5annnoKV1dXbGxsaNmyJRs3bixyXh8fH6ZPn67/XqVS8d1339GzZ0+srKzw8/NjxYoV+te3bNmCSqXSN0nPnz8fBwcH1q1bh7+/PzY2NnTp0qXIB4u8vDyGDx+Og4MDNWrUYPTo0fTv358ePXqU6mcwa9Ys6tati5mZGQ0aNOCnn37Sv6YoChMnTsTLywtzc3M8PDwYPny4/vVvvvkGPz8/LCwscHV15ZlnninVtSuL3FFXQc425vqOaflahSMXC+623Xj/YiATM8FLlUA79RE6mBwhXH0M0+RYVI7ehSc59w8o+eAdDhpTw1VGCCOlKAo3c/MNcm1LU0259RgeM2YMU6dOxdfXF0dHR+Li4nj88cf5+OOPMTc358cff6R79+5ER0fj5eV11/O8//77fPbZZ0yZMoWvv/6afv36ceHCBZycip+4KTMzk6lTp/LTTz+hVqt54YUXeOutt1i4cCEAn376KQsXLmTevHn4+/vz5Zdfsnz5ctq3b1/iui1btowRI0Ywffp0IiIiWLlyJQMHDqR27dq0b9+e33//nS+++ILFixfTuHFj4uPjOXz4MAD79u1j+PDh/PTTT7Ru3ZqkpCT++eefUvxkK48k6ipOo1YR6OVIoJcjIyPqk5SRwz+nr7I1+iprT3vyc3pHLMhmgPd13rJ0LnzDt34K5/+BLp9Aq8GGrIIQRulmbj6Nxq8zyLVPfNAZK7Py+fP8wQcf0LFjR/33Tk5OBAQE6L//8MMPWbZsGStWrGDo0KF3Pc+AAQPo27cvAJMmTeKrr75iz549dOnSpdjyubm5zJ49m7p16wIwdOhQPvjgA/3rX3/9NWPHjqVnz54AzJgxg9WrV5eqblOnTmXAgAEMGTIEgFGjRrFr1y6mTp1K+/btiY2Nxc3NjYiICExNTfHy8iIkJASA2NhYrK2teeKJJ7C1tcXb25vAwMBSXb+ySNN3NeNkbcZTzWsxrU9z9vxfBD+9EgKmlsy+4MGEFcdRFAUUBWrUAytnaPB44cFHl8JvA+H4MshON1wlhBDlJjg4uMj36enpvPXWW/j7++Pg4ICNjQ1RUVHExsbe8zzNmjXTf21tbY2dnZ1+isziWFlZ6ZM06KbRLCifkpJCQkKCPmkCaDQagoKCSlW3qKgowsPDi+wLDw8nKioKgGeffZabN2/i6+vLoEGDWLZsGXl5eQB07NgRb29vfH19efHFF1m4cCGZmZmlun5lkTvqakytVtHWryZfPhfI6z/vZ+HuWLxrWPFau7rQfTp0+xzUt82Wc/Q3OLUWjv+hmyWtXgfwfxIadJGJVsRDx9JUw4kPOhvs2uXlv72333rrLTZs2MDUqVOpV68elpaWPPPMM+Tk5NzzPKamRR+RqVQqtFptqcorilLK6B+Mp6cn0dHRbNy4kQ0bNjBkyBCmTJnC1q1bsbW15cCBA2zZsoX169czfvx4Jk6cyN69eyt0iFlZyB31Q6BzYzfe7dYIgEmrT7L66K0OHer//DF45B0IHwGOdSA/G6JXw/LXYYof/PE/uHK4kiMXwnBUKhVWZiYG2SpyhrTt27czYMAAevbsSdOmTXFzc+P8+fMVdr3i2Nvb4+rqyt69hR1c8/PzOXDgQKnO4+/vz/bt24vs2759O40aNdJ/b2lpSffu3fnqq6/YsmULO3fu5OjRowCYmJgQERHBZ599xpEjRzh//jx///33A9SsYsgd9UPi5XAfYq9nsGDnBd5YcghXOwuCvP9zl1wrSLdFvA8JxyHqL4haAYkn4Mhi3ebTFsIiwa+zLNspRBXk5+fHH3/8Qffu3VGpVLz33nv3vDOuKMOGDWPy5MnUq1ePhg0b8vXXX3Pjxo1SfUh5++236d27N4GBgURERPDXX3/xxx9/6Huxz58/n/z8fEJDQ7GysuLnn3/G0tISb29vVq5cSUxMDO3atcPR0ZHVq1ej1Wpp0KBBRVW5zOQv7UNCpVIxvntjIvxdyM7TMujHfVy4nnG3wuDWBNqPhSE74dW/oUkvUGl0HdB+eQ5mBMOeuZB7s3IrIoR4INOmTcPR0ZHWrVvTvXt3OnfuTIsWLSo9jtGjR9O3b19eeuklwsLCsLGxoXPnzlhYlHxa5B49evDll18ydepUGjduzLfffsu8efN49NFHAd160HPnziU8PJxmzZqxceNG/vrrL2rUqIGDgwN//PEHjz32GP7+/syePZtffvmFxo0bV1CNy06lVPZDgyrg4sWLeHp6EhcXR+3atQ0dTrnKzMmjz7e7OHopBV9na/4Y0hoHqxIurZlyEXZ/C/sXQHYKmFrBG8dlXW1R5WVlZXHu3Dnq1KlTqkQhyo9Wq8Xf35/evXvz4YcfGjqccnGv/1elyTMGv6OeOXMmPj4+WFhYEBoayp49e+5ZPjk5mcjISNzd3TE3N6d+/fpFuvTn5+fz3nvvUadOHSwtLalbty4ffvhhpXdiMFZWZiZ83z+YWg6WxFzL4LUf95OdV8Kxova1odOHMOoEdP0M2o4qmqS3fAKXD1VI3EKI6uXChQvMnTuXU6dOcfToUQYPHsy5c+d4/vnnDR2a0TFool6yZAmjRo1iwoQJHDhwgICAADp37nzXLv85OTl07NiR8+fPs3TpUqKjo5k7dy61atXSl/n000+ZNWsWM2bMICoqik8//ZTPPvuMr7/+urKqZfRc7Cz4YUBLbM1N2HM+iXeWHindBxlzGwj9H7R7u3Bf3F7YMhm+7wiZSeUftBCiWlGr1cyfP5+WLVsSHh7O0aNH2bhxI/7+/oYOzegYtDPZtGnTGDRoEAMHDgRg9uzZrFq1ih9++IExY8bcUf6HH34gKSmJHTt26Lv++/j4FCmzY8cOnnrqKbp166Z//ZdffrnvnfrDpoGbLbNeCGLAvD38eegyXk5WvNnpATpRWDpA095galn0LjvqL6j7GJiVfIJ/IUT15+npeUePbVE8g91R5+TksH//fiIiIgqDUauJiIhg586dxR6zYsUKwsLCiIyMxNXVlSZNmjBp0iTy8wubblu3bs2mTZs4deoUAIcPH+bff/+la9euFVuhKqiNnzOTejYF4Ou/z/Drvriyn8zZD3rNhe5fFu6LPwZLXoBpjWDj+5AW/4ARCyHEw8dgd9TXrl0jPz8fV1fXIvtdXV05efJkscfExMTw999/069fP1avXs2ZM2cYMmQIubm5TJgwAdDNa5uamkrDhg3RaDTk5+fz8ccf069fv7vGkp2dTXZ2tv77tLS0cqhh1dC7pSexSZnM2HyG//vjKB72lrTxcy77CW8fWpF5DZx8ISkG/p0GO2dAs97QejjUNL4hEEIIYYwM3pmsNLRaLS4uLsyZM4egoCD69OnDuHHjmD17tr7Mr7/+ysKFC1m0aBEHDhxgwYIFTJ06lQULFtz1vJMnT8be3l6/3T5Y/mHwZqf6PNXcgzytwuCf9xMdX04fVHwfhaH7oM9C8AyF/Bw4+DPMDIFFfeD8v7rpTIUQQtyVwRK1s7MzGo2GhISEIvsTEhLuuq6pu7s79evXR6MpnFHL39+f+Ph4/fR3b7/9NmPGjOG5556jadOmvPjii7zxxhtMnjz5rrGMHTuWlJQU/XbixIlyqGHVoVKp+OyZZoT4OJGWncfL8/eSmJpVPidXa8D/CXhlPby8Hho+Aah0U5XO7wZzH9PNLa41zCpFQghh7AyWqM3MzAgKCmLTpk36fVqtlk2bNhEWFlbsMeHh4Zw5c6bILDqnTp3C3d0dMzPdWODMzEzU/5kxS6PR3HPmHXNzc+zs7PSbra3tg1StSjI30fDti0H4OltzKfkmryzYR2ZOXvlexCsUnlsIw/ZD8MtgYgGXD8BvA+CrQNg9B3LuMgmLEEI8pAza9D1q1Cjmzp3LggULiIqKYvDgwWRkZOh7gb/00kuMHTtWX37w4MEkJSUxYsQITp06xapVq5g0aRKRkZH6Mt27d+fjjz9m1apVnD9/nmXLljFt2jT9Umri7hytzZg3sCVO1mYcvZTC8F8Okq+tgKbpGnXhiS9g5DF4ZDRYOkHyBVjzNlw7Vf7XE0KIqkwxsK+//lrx8vJSzMzMlJCQEGXXrl361x555BGlf//+Rcrv2LFDCQ0NVczNzRVfX1/l448/VvLy8vSvp6amKiNGjFC8vLwUCwsLxdfXVxk3bpySnZ1d4pji4uIUQImLi3vg+lVF+84nKX7jViveo1cqE/48VvEXzM5QlN1zFGXZkKL7Dy9RlKunK/764qF38+ZN5cSJE8rNmzcNHYpBPPLII8qIESP033t7eytffPHFPY8BlGXLlj3wtcvrPPcyYcIEJSAgoEKvUZx7/b8qTZ4x+KIcQ4cOveti5Vu2bLljX1hYGLt27brr+WxtbZk+fTrTp08vpwgfPkHejnzRuzmRiw4wf8d5ajta0jOwFmlZeaRn55GWlUdaVi7p2bd/n0d6dq7u36w80rIL92m1MPjRurzQyrv4C5pZQcigovvSE+HPoboOaEN2gotMgiDEf3Xv3p3c3FzWrl17x2v//PMP7dq14/Dhw0XWki6JvXv33rE85oOaOHEiy5cv59ChQ0X2X7lyBUdHWUb3XgyeqIVx6tbMnYs3GjJ5zUk+WhXFR6uiHuh87y4/Rm6+loHhdUp2QE66bqKUzGtQs2Hh/ui14B4Adu4PFI8Q1cErr7xCr169uHjx4h3zRc+bN4/g4OBSJ2mAmjVrlleI93W3zsOiUJUaniUq12vtfHmtnS8atW5stLWZBlc7c+q52NDc04G2fs483tSN3sG1eaVNHUZ08OPdbv582qspM59vwYKXQ/hjSGtef6QuAO//dYIFO86X7OJOvvD8Yui/snBsdlYqLB0IXzTSDe+KWgn5uRVQcyGqhieeeIKaNWsyf/78IvvT09P57bffeOWVV7h+/Tp9+/alVq1aWFlZ0bRpU3755Zd7ntfHx6dIq+Tp06dp164dFhYWNGrUiA0bNtxxzOjRo6lfvz5WVlb4+vry3nvvkZur+/2cP38+77//PocPH0alUqFSqfQxq1Qqli9frj/P0aNHeeyxx7C0tKRGjRq89tprpKen618fMGAAPXr0YOrUqbi7u1OjRg0iIyP11yoJrVbLBx98QO3atTE3N6d58+ZFWiVycnIYOnQo7u7uWFhY4O3trR85pCgKEydOxMvLC3Nzczw8PBg+fHiJr10Wckct7kqlUvF/j/szqmN9TDVqfcIurUBPB9Qq+GbLWSasOI5KBS+F+ZTsYNPbVpxJT9TdTcfu1A3vOrUWrF2geV8IfAmc65UpPiHuqSwjETTmoLn15zU/D/KzQaXWTbF7v/OWYrpdExMTXnrpJebPn8+4ceP0azn/9ttv5Ofn07dvX9LT0wkKCmL06NHY2dmxatUqXnzxRerWrUtISMh9r6HVann66adxdXVl9+7dpKSkMHLkyDvK2draMn/+fDw8PDh69CiDBg3C1taWd955hz59+nDs2DHWrl2rXyva3t7+jnNkZGTQuXNnwsLC2Lt3L4mJibz66qsMHTq0yIeRzZs34+7uzubNmzlz5gx9+vShefPmDBo06I5zFufLL7/k888/59tvvyUwMJAffviBJ598kuPHj+Pn58dXX33FihUr+PXXX/Hy8iIuLo64ON3Mjb///jtffPEFixcvpnHjxsTHx3P48OESXbesJFGL+7Iw1dy/0D2oVCre7twArQKzt55l/J/HUalUvHi3Z9Z341wPXl4LV0/BwZ/g8C+QkQjbv9RtXq2hxYvQ6CmZW1yUn0kepT/m2fnQ+NZIk5N/6YYgereBgasKy0xvCpnX7zx2YkqpLvXyyy8zZcoUtm7dql+Hed68efTq1Us/idNbb72lLz9s2DDWrVvHr7/+WqJEvXHjRk6ePMm6devw8ND9LCZNmnTHtMzvvvuu/msfHx/eeustFi9ezDvvvIOlpSU2NjaYmJjcs6l70aJFZGVl8eOPP+qfkc+YMYPu3bvz6aef6meydHR0ZMaMGWg0Gho2bEi3bt3YtGlTiRP11KlTGT16NM899xygW8xp8+bNTJ8+nZkzZxIbG4ufnx9t2rRBpVLh7V34tyo2NhY3NzciIiIwNTXFy8urRD/HByFN36JSqFQqRndpwGvtfAF4b/kxFu2OLdvJata/tdxmlG7Ws/pddHcrsTtg+WCY2gD+GgGX9svMZ6Laa9iwIa1bt+aHH34A4MyZM/zzzz+88sorgG7p3w8//JCmTZvi5OSEjY0N69atIza2ZL9/UVFReHp66pM0UOxcF0uWLCE8PBw3NzdsbGx49913S3yN268VEBBQpCNbeHg4Wq2W6Oho/b7GjRsXmfjK3d39rqsu/ldqaiqXL18mPDy8yP7w8HCionR9cQYMGMChQ4do0KABw4cPZ/369fpyzz77LDdv3sTX15dBgwaxbNky8vLKec6J/5A7alFpVCoVY7s2RKtV+O7fc/zfsqOoVfBciFfZTqgx1c165v8EpF6GQ4t0U5TeOAf75+u2Fv3hya/KsxriYfN/l0t/jMa88OuG3XXnUP3nvmjk0QeL6zavvPIKw4YNY+bMmcybN4+6devyyCOPADBlyhS+/PJLpk+fTtOmTbG2tmbkyJH62RzLw86dO+nXrx/vv/8+nTt3xt7ensWLF/P555+X2zVuV7B6YgGVSnXPSa1Kq0WLFpw7d441a9awceNGevfuTUREBEuXLsXT05Po6Gg2btzIhg0bGDJkiL5F479xlRe5oxaVSqVSMa6bPy/f6v095o+j/Lr3AVbtKmDnAe3egmEHoP9fuiU3TSzQerdhzdErbDt1FZJjYe/3kJZw//MJUcDMuvSb5rZ7II2Jbt/tz6fvdd4y6N27N2q1mkWLFvHjjz/y8ssv659Xb9++naeeeooXXniBgIAAfH199asLloS/vz9xcXFcuXJFv++/Q2R37NiBt7c348aNIzg4GD8/Py5cuFC0umZmRVY6vNu1Dh8+TEZG4fP77du3o1aradCgfBbysbOzw8PD444lNrdv315knQc7Ozv69OnD3LlzWbJkCb///jtJSUkAWFpa0r17d7766iu2bNnCzp07OXq0/D54/ZfcUYtKp1KpeO8Jf7SKwvwd5xn9xxFQQe9gzwc/uVoNddpBnXbsODaaTzbGciT+AAA/N9xJm/NfQ9QKeOnPB7+WEEbCxsaGPn36MHbsWFJTUxkwYID+NT8/P5YuXcqOHTtwdHRk2rRpJCQklHjxoYiICOrXr0///v2ZMmUKqampjBs3rkgZPz8/YmNjWbx4MS1btmTVqlUsW7asSBkfHx/OnTvHoUOHqF27Nra2tpibmxcp069fPyZMmED//v2ZOHEiV69eZdiwYbz44ot3rLT4IN5++20mTJhA3bp1ad68OfPmzePQoUMsXLgQgGnTpuHu7k5gYCBqtZrffvsNNzc3HBwcmD9/Pvn5+YSGhmJlZcXPP/+MpaVlkefY5U3uqIVBqFQqJnRvRP8wbxQFRv9+hKX7L5bLuU9cTuXF73fz/M/RHIm/ieWtznC/ntISa+lPfsMnCwunxcPcDvDvF3D9bLlcXwhDeOWVV7hx4wadO3cu8jz53XffpUWLFnTu3JlHH30UNzc3evToUeLzqtVqli1bxs2bNwkJCeHVV1/l448/LlLmySef5I033mDo0KE0b96cHTt28N577xUp06tXL7p06UL79u2pWbNmsUPErKysWLduHUlJSbRs2ZJnnnmGDh06MGPGjNL9MO5j+PDhjBo1ijfffJOmTZuydu1aVqxYgZ+fH6Drwf7ZZ58RHBxMy5YtOX/+PKtXr0atVuPg4MDcuXMJDw+nWbNmbNy4kb/++osaNWqUa4y3UymK9Lb5r4sXL+Lp6UlcXNwdkwiI8qUoCuP/PM5Puy6gUsHnzwbwdIuy/cwvJ99k6vpolh28hKKAqUbFi618GPZYPVYevcKEP4+hVaB9fWdm9AvC2twE9syF1YU9YnFtAv7dwf9J3WxoqrINSRNVS1ZWFufOnaNOnTpYWFjc/wAhSuBe/69Kk2ek6VsYlEql4oOnGqOg8POuWN787TBqlYoegbVKfI7UrFy+2XyWedvPkZ2n61DyRDN33uncEK8aVgC82MobNzsLhv1ygM2nrvHcnF38MKAlNRv10C3FeWIFnNsGCcd025bJUKOeLmH7dwePQEnaQgiDkDvqYsgddeXTahXe/VM3ZEutgi/6NOep5vdO1jl5Wn7edYGv/z7NjUzdrEQhdZz4v8f9ae7pUOwxB2Nv8MqCfSRl5ODpZMn8gSHUrWmjezEzCaLX6J5hn/1bN894AXtPqN9ZNxTMp23RiVhElSd31KIiyB21qFbUahUfPdUErVZh8d443lhyCLVKRfeAOyebUBSF1Ufj+WzdSS5czwSgbk1rxnb1p4O/i763a3ECvRz5Y3Br+s/bw4XrmfSatYPvXgom2McJrJwgsJ9uy0qF0+t1Sfv0BkiJg73f6TYZ8iWEqESSqIXRUKtVTOrZFK2i8Ou+i4xccgiVCp5oVpis955P4uNVURyKSwbA2cacUR3r0zu4NiaakvWN9HG25o/BrXl5wT4OxyXz/He7+bJPc7o2vW2hDws7aPqMbsvJ1DWLn1oLp9ZBvYjCcpf2w18jocnT0OaNcvgpCCFEUZKohVFRq1V88nQztAos3X+REYt1d9b1XW35dO1JNpzQjYG2MtPwWjtfBrX11XUKK6UaNuYsHtSKYb8cYGNUIkMWHWD8E42KX93LzAoadNFtigLKbRMrnFoH8UfA0afoMdFrwbu1LuELIcQDkEQtjI5areLTXs1QFPj9wEWG/XIQgHytgkatok9LT0Z28MPF7sGeJVqaaZj9QhAT/zrOz7tief+vE1xOvsnYrv6o77YAiUoFqtvmPg95DRzrgP1tz9OvnYFf+oDaFMW7NVdc2hFlF86jYa3KvLCJqBzlObuVEOX1/0kStTBKGrWKz55phqIo/HHwEgAR/q6M6dqAei625XYdE42aD59qgoeDJZ+tjWbuP+e4nJLF588GlGwxEmtn3epdt9GmXiHbzhfL1BhU57bicW4rHsDlXU3xaP8/XTO5LBpiVMzMzFCr1Vy+fJmaNWtiZmZ2z74OQtyLoijk5ORw9epV1Go1ZmZmD3Q+6fVdDOn1bTzytQrLDl7Cp4aVrsNXBVp28CLvLD1Cbr5CSB0n5r4YjL1VyebuzcvXsvtcEmuOXWHd8QSupmXjo7rCY+pDRGgOEqI6gYnq1qdrM1to2gtavAQeLWTYl5HIycnhypUrZGZmGjoUUU1YWVnh7u5ebKIuTZ6RRF0MSdQPr+1nrvH6T/tJy86jnosN8we2pLajVbFls/Py2XHmOmuOXWHDiQT9EDEAWwsTIvxd6dzYjXb1nRn87Rr841cyyPpfauTcNgOba1Ndwm72LFg6VnT1xH0oikJeXt5956QW4n40Gg0mJiZ3bZmRRP2AJFE/3KKupDJw3l7iU7NwsTVn3sCWNPbQLXJ/MyefracSWXMsnr+jEknLLlzeztHKlE6N3OjS1I3wus6YmRT2Qt8Vc53n5uzCRA3/9DbD/eyvcOJPyM/WFQh+BZ6YVqn1FEIYjoyjFuIB+LvbsSyyNQN+2Et0Qhq9Z+9kRIQfB2OT2RJ9lZu5hXdbLrbmdGniRpcmboT4ON11iFgr3xo81tCFv08m8uFxJ77pNxce/wyO/AYHFkCLFwsLXz4EMZuheT+wcang2gohjJ3cURdD7qgFQMrNXF7/aT87Y64X2V/b0ZKut5JzoKfj3XuI/8fJ+FS6fvkPigLLhrQm0OtWU3fBr2BBE9mfkbp1tQOeh56zyqs6QggjUpo8Y/DVs2bOnImPjw8WFhaEhoayZ8+ee5ZPTk4mMjISd3d3zM3NqV+/PqtXry5S5tKlS7zwwgvUqFEDS0tLmjZtyr59+yqyGqIasrc0Zf7LLekb4oW/ux2R7euyclgb/nmnPeO6NSLI26nESRqgoZsdvW4tOPLJmpMotyfo259j1XkUaofonl0XSDgO69+Fi/sLE7sQ4qFg0KbvJUuWMGrUKGbPnk1oaCjTp0+nc+fOREdH4+JyZ5NfTk4OHTt2xMXFhaVLl1KrVi0uXLiAg4ODvsyNGzcIDw+nffv2rFmzhpo1a3L69GkcHaWjjig9cxMNk59uWm7nG9WxPisOX2b3uSQ2RyfyWMNi1tht9qxuu92+ebB3Luz4WjfveKOndFutYN0a3EKIasugTd+hoaG0bNlSv9aoVqvF09OTYcOGMWbMmDvKz549mylTpnDy5ElMTYsfNjNmzBi2b9/OP//8U+a4pOlbVKTJq6P4dlsMDVxtWT2ibckmQTm1Do4s0c14lptRuN/WAxo9CY16gGeoJG0hqogq0fSdk5PD/v37iYgonDdZrVYTERHBzp07iz1mxYoVhIWFERkZiaurK02aNGHSpElFhlKsWLGC4OBgnn32WVxcXAgMDGTu3LkVXh8hSmrIo/WwtzQlOiGNPw5cvP8BoFu565kf4J2z0GchNO2tG4+ddhl2z4Z5XWCaP6x6C87/C1oZXiREdWGwRH3t2jXy8/NxdS3a9Ofq6kp8fHyxx8TExLB06VLy8/NZvXo17733Hp9//jkfffRRkTKzZs3Cz8+PdevWMXjwYIYPH86CBQvuGkt2djapqan6LS0trXwqKUQx7K1MiWxfF4BpG06RlVuKpGpqCf5PQK+58PYZ6LsYAvqCuT2kx+uax+d3g88bwKn1FVQDIURlqlLDs7RaLS4uLsyZMweNRkNQUBCXLl1iypQpTJgwQV8mODiYSZMmARAYGMixY8eYPXs2/fv3L/a8kydP5v3336+0egjxUpgP87ef53JKFgt2nOd/j9Qt/UlMLaBBV92WlwPntsKJ5XByFWRcBUfvwrKXDuj2+bTVLTIihKgyDHZH7ezsjEajISEhocj+hIQE3Nzcij3G3d2d+vXro9EUzsHs7+9PfHw8OTk5+jKNGjUqcpy/vz+xsbF3jWXs2LGkpKTotxMnTpS1WkKUiIWphlGdGgAwc/MZkjNzHuyEJmbg1xGemglvnYYBq6Bmg8LXt38Ji3rDv18U7tNqpQe5EFWAwRK1mZkZQUFBbNq0Sb9Pq9WyadMmwsLCij0mPDycM2fOFFmR5NSpU0XmUg0PDyc6OrrIcadOncLb25u7MTc3x87OTr/Z2pbfog9C3E3PwFo0dLMlNSuPb7acLb8Ta0zBp03RfQ6eYO+lS+YFolfDlwGw6k1dJ7WcDIQQxsegXURHjRrF3LlzWbBgAVFRUQwePJiMjAwGDhwIwEsvvcTYsWP15QcPHkxSUhIjRozg1KlTrFq1ikmTJhEZGakv88Ybb7Br1y4mTZrEmTNnWLRoEXPmzClSRghjoFGrGN2lIQDzd5znUvLNirtYp49g5BGo3bJw35kNkHwB9n6nW5bz0zrwU0/Y+Q1cOy1320IYCYM+o+7Tpw9Xr15l/PjxxMfH07x5c9auXavvYBYbG4v6tuEmnp6erFu3jjfeeINmzZpRq1YtRowYwejRo/VlWrZsybJlyxg7diwffPABderUYfr06fTr16/S6yfE/TzaoCatfJ3YFZPEtPWn+Lx3QMVd7L+LA3T6GPw66xL26Y2QEgtn/9Zt68aCg7fuDtyvkzzbFsKAZArRYsg4alGZDscl89TM7ahUsHp4W/zd7So/CEWBa6fg9AZd4r6wA/Jve26uMdc1p4f+TzdUTAjxQKrEOGohhE6ApwPdmrqjKPDp2pOGCUKl0nU+az0UXvoT3jkHz/0CwS/rnm3nZ8PZTZB8W6fMmzcgMUqayIWoYFVqeJYQ1dVbnRuw7ng8W6KvsuPsNVrXdTZsQOY20PBx3aYocDUaTq2Bht0KyxxfBivf0M2K1vvu8xQIIR6M3FELYQTqOFvTN8QLgE9vX7DDGKhU4NIQ2rwBdh6F+9MTdU3i7rc9V795A5ZHQtRKyMms/FiFqIbkjloIIzG8gx9/HLjI4YsprDp6hSeaedz/IEN6dAyEDQXltpnVTm+AQz/rNhNLqNseGjyum5TF2sCtBEJUUXJHLYSRqGlrzqB2vgBMWRdNbr72PkcYAXMbsLAv/N6lEYQO1j3XzrupG6u9YihM9YMfusLuOZCWcPfzCSHuIIlaCCPyaltfnG3MuHA9k1/23H02PaPl1gS6fqIbs/36v/Do/5Hn0hQULcTugDVvw7SGsKC7bunOjOuGjlgIoyeJWggjYmNuwogOfgB8ufE06dl5Bo6ojFQq8mo25ifzPgRfG0/rrK/4MPcFEu1uJe1z22DlSN2d9ubJho5WCKMmiVoII/NciBc+Nay4npHD3G0xhg6nTHbFXOeJr//lvT+Pk5yZS46NB9/nP85T2e+TO/QQREwEt2a659tOvoUHplyCI79CtqxgJ0QBSdRCGBlTjZq3O+umFp37TwyJaVkGjqjkLiXfJHLRAZ6bs4uT8WnYW5ry4VON2fZOe5xtzLmSksVfsaa6HuSv/wND9xcd8nX0N/hjEPz6kuEqIYSRkUQthBF6vKkbAZ4OZObk89Wm04YO576ycvP5cuNpOny+hVVHrqBWwYutvNny1qO8GOaDlZkJA8N9AJizLaZw+JlzPV2HtAKWDuBUF/y7F+5LS4Clr+iW78x7wFXGHmILdpyn2cR1HLuUYuhQRClJohbCCKlUKsZ21d1V/7Injpir6QaOqHiKorDm6BU6fL6VLzaeIitXS0gdJ1YOa8uHPZrgaG2mL/tCqDdWZhpOxqex9dTV4k8YNACG7YfA2+6oT/wJx5bC4udhmj+sG6ebgEWUysLdF0jNymPZwUuGDkWUkiRqIYxUK98atG9Qk3ytwtT1xpeYouPT6PfdbgYvPMCl5Jt42Fsw4/lAlrzWikYed85Xbm9lynMtdZO6zLnXs3eVCjS3TfHg0wZaRYKNK2Reg50zYGYIfNcRDvwE2cb5IcaYJGXkcCpB93Pad+GGgaMRpSUTnghhxEZ3bciWU1dZfTSe8X8eQ6NWkZevkKdVyMvX6v699XVuvkK+VrcvN19bWE6r+7qmrTm+ztbUdbHB19kG35rWuNtboPrvqlr3kZKZyxcbT/HTrgvkaxXMTNS8/khdBj9SF0szzT2PfbmNDwt2nmfH2escu5RCk1r29ywPgGsj6DIJOn4Ap9fDwZ/g1Dq4uEe3rR0DjXtCi/5QO/jOVcIEe88n6b8+fimFrNx8LEzv/V4J41GmRB0XF4dKpdKv+LFnzx4WLVpEo0aNeO2118o1QCEeZg3d7OjVojZL91/kx50XHuhcJ+PT+Of0tSL7LE011HG2xremNXVr2uj/reNsjbV50T8P+VqFxXtjmboumhuZuQB0aezGuG7+eDqVbAnM2o5WPNHMnT8PXebbbTF83Tew5BXQmBTOP54WD4cWwcGfIemsLnkf/AlqNoTAFyGgL1jXKPm5q7k95woTdZ5W4XBcMqG+8vOpKsqUqJ9//nlee+01XnzxReLj4+nYsSONGzdm4cKFxMfHM378+PKOU4iH1nvdGuFub0F2nhYTtQoTjfrWvypM1Wo0ahWmGt1+/ddqdWFZjQq1SkV8yk1irmZw9moGMdfSib2eyc3cfE5cSeXEldQ7rutmZ4FvTV0S93ayZtnBS/py9V1tmNC9MeH1Sj8t6GvtfPnz0GVWHbnMO50blDjJF2HrBm1H6XqPX9ihS9LHl8PVk7B+HDh6F+2Q9pDbfU43sYy1mYaMnHz2x96QRF2FlClRHzt2jJCQEAB+/fVXmjRpwvbt21m/fj2vv/66JGohypG9lSlvdmpQ7ufNzdcSm5RJzNUMYq6m6/69ls7ZqxkkZeQQn5pFfGoWO84Wzh5mZ2HCGx3r80Irb0w1Zevi0tjDnrZ+zvxz+hrf/3uOiU82LnslVCrwCddtXT+Fo0t1vcPrdykss3sOZFyFFi+Bg2fZr1VFpWblcuKy7gNWv1bezNkWw/7z8py6KilTos7NzcXc3ByAjRs38uSTTwLQsGFDrly5Un7RCSEqjKlGTd2aNtStaQO4FnktOTNHd+d9NZ2Yaxmcu5pBbUdLBj9alxo25g987dfa+fLP6Wss2RvHiA5+RXqHl5mFPbR8RbcV0ObD9i8h9SI4138oE/X+CzfQKuDlZEW3pu66RB17A61WQa2W5/lVQZkSdePGjZk9ezbdunVjw4YNfPjhhwBcvnyZGjWkOUWIqs7ByowgbzOCvB0r5Pxt6jnTyN2OE1dS+WnXBYbfmja13CkKdPoAjv4O/k8U7t/+JcTuhmbP6u6+TS0r5vpGoOD5dEgdJxp52GFhqiY5M5eYaxnUc7G5z9HCGJSp7erTTz/l22+/5dFHH6Vv374EBOjWo12xYoW+SVwIIe5GpVLxv0d0U4cu2HGerNz8+xxRRhoTaNIL+i4qTMaKouuEFr0KfhsAU/xg+RA4u1l3B17NFCTq0DpOmGrUBNR2AGD/haR7HCWMSZkS9aOPPsq1a9e4du0aP/zwg37/a6+9xuzZs8stOCFE9fV4U3dqOVhyPSOH3w9crLwLq1Tw7HwIHwl2tSEnDQ4thJ966CZUWTsWLh3QJfQq7mZOPkcuJgMQWkfX2lnQSrJfxlNXGWVK1Ddv3iQ7OxtHR90bfuHCBaZPn050dDQuLi7lGqAQonoy1ah5pU0dAOZuiyFfW4mJ0bUxdHwfRh6FgWsgaCBYOEB6Auz6Bua2hxktYcunkFQ1F0YBOBh7g9x8BTc7CzyddC0KwT66v9sy8UnVUaZE/dRTT/Hjjz8CkJycTGhoKJ9//jk9evRg1qxZ5RqgEKL66tPSE3tLU85fz2TDifjKD0CtBu/W0H06vHUanvtFN3mKiQVcPw1bJsFXgTC3g27O8Spm923PpwsmtmnhpUvUMbd69wvjV6ZEfeDAAdq2bQvA0qVLcXV15cKFC/z444989dVXpT7fzJkz8fHxwcLCgtDQUPbs2XPP8snJyURGRuLu7o65uTn169dn9erVxZb95JNPUKlUjBw5stRxCSEqlrW5CS+00k0rOnvrbYt1GIKJmW4ylWfnw9tnoMdsqPsYqNSQdgWsaxaWPb0Rrp81WKgldXtHsgIOVmb6TmQH5K66SihTos7MzMTW1haA9evX8/TTT6NWq2nVqhUXLpRu9qQlS5YwatQoJkyYwIEDBwgICKBz584kJiYWWz4nJ4eOHTty/vx5li5dSnR0NHPnzqVWrVp3lN27dy/ffvstzZo1K30lhRCVon9rH8xM1ByKS2avsYzvNbeF5n3hxWUw6iT0+l539w2Qn6dbivPrFrqe40YqOy+fA7G6n2crX6cirwV7S/N3VVKmRF2vXj2WL19OXFwc69ato1OnTgAkJiZiZ3fnZPz3Mm3aNAYNGsTAgQNp1KgRs2fPxsrKqkgntdv98MMPJCUlsXz5csLDw/Hx8eGRRx7R9zwvkJ6eTr9+/Zg7d67+WboQwvi42FrQq4Xug/acbUZ4l2rrCt5hhd9nXgf3Zro77FpBhfu3TYU1o+HcNl0yN7CjF1PIztPiZG12a6x8oRa3ErXcUVcNZUrU48eP56233sLHx4eQkBDCwnT/idevX09gYMnn7s3JyWH//v1EREQUBqRWExERwc6dO4s9ZsWKFYSFhREZGYmrqytNmjRh0qRJ5OcXHVYRGRlJt27dipxbCGGcXm3ri0oFG6MSOZOYZuhw7s3WFV76E944XrjKl6LA/vmwezYs6A5Tbw35Orkacm8aJEz982kfpzsWXim4oz58MZmcPG2lxyZKp0wTnjzzzDO0adOGK1euFLmT7dChAz179izxea5du0Z+fj6urkVnRXJ1deXkyZPFHhMTE8Pff/9Nv379WL16NWfOnGHIkCHk5uYyYcIEABYvXsyBAwfYu3dvieLIzs4mOztb/31ampH/oRCimqlb04aO/q6sP5HA3G3n+PSZKvC4yuS2GdoUBbp+BidXQvRquJmkG/J1aCGYWkG9DtCwO9TvBJaV08JX3PPpAnWcrXGyNiMpI4fjl1MI9JJWR2NW5mUu3dzccHNz4+JF3fjH2rVrV8pkJ1qtFhcXF+bMmYNGoyEoKIhLly4xZcoUJkyYQFxcHCNGjGDDhg1YWFiU6JyTJ0/m/fffr+DIhRD38r9HfFl/IoFlBy/xZqf6uNiV7PfXKKjVhSt75edB7E5d0o5aqZu+NOov3aY21SXtxk/ryprbVkg4efla/TjpUN87E7VKpaKFlyMboxLYf+GGJGojV6amb61WywcffIC9vT3e3t54e3vj4ODAhx9+iFZb8mYUZ2dnNBoNCQlFhz0kJCTg5uZW7DHu7u7Ur18fjaZwLVV/f3/i4+P1TemJiYm0aNECExMTTExM2Lp1K1999RUmJiZ3NJEDjB07lpSUFP124sSJEtdBCFE+grydCPJ2JCdfy7wd5w0dTtlpTKBOW90iIW8cg9e2QLu3dUtwanPh1FpY9hpMqQcpFTPRy4krqaRn52FrYUJDt+L7DcnEJ1VHmRL1uHHjmDFjBp988gkHDx7k4MGDTJo0ia+//pr33nuvxOcxMzMjKCiITZs26fdptVo2bdqkf+79X+Hh4Zw5c6bIB4JTp07h7u6OmZkZHTp04OjRoxw6dEi/BQcH069fPw4dOlQkwRcwNzfHzs5OvxX0aBdCVK7/tdNNK/rzrgukZxu+Q9YDU6nAIxAeexcid8OQXdDuHahRDxx9wL52Ydk9c3Urf+Vl3/V0JVXQ7N3SxwnNXRbeuH3iE4MOixP3Vaam7wULFvDdd9/pV80CaNasGbVq1WLIkCF8/PHHJT7XqFGj6N+/P8HBwYSEhDB9+nQyMjIYOHAgAC+99BK1atVi8uTJAAwePJgZM2YwYsQIhg0bxunTp5k0aRLDhw8HwNbWliZNmhS5hrW1NTVq1LhjvxDCuET4u+Jb05qYqxks3hPLq219DR1S+XLxh8fGQfv/g8zb5trOyYQNEyA3A17dBLWDH+gyu+/xfLpA01r2mGpUXE3L5uKNm2VbF1xUijLdUSclJdGwYcM79jds2JCkpNJN9N6nTx+mTp3K+PHjad68OYcOHWLt2rX6DmaxsbFFls709PRk3bp17N27l2bNmjF8+HBGjBjBmDFjylIVIYQRUatVDLqVnH/49xy5+dW0R7JKBda3rTSYlwVBA8ArrOiQr/XvwYrhELO1xAuGaLUKe8/fP1FbmGpoUssegH2yQIdRUyllaPMIDQ0lNDT0jlnIhg0bxp49e9i923gnASiJixcv4unpSVxcHLVr177/AUKIcpOVm0+bTzdzLT2bL/oE0DPwIf0dzMvRDfPKStZ9b+0CjZ7STXHq1QrUdz7GA4iOT6Pz9G1Ymmo4MrETppq73499tPIE3/17jn6hXnzcs2kFVELcTWnyTJmavj/77DO6devGxo0b9c+Sd+7cSVxc3F2n8hRCiJKwMNUwMNyHKeui+XZrDD2a17pjHPBDQa2B3gvg2B8QtQIyEmHvXN1m4wr+T0LjHrq78NuS9u5z1wFdZ7F7JWnQPaf+7t9z0qHMyJWp6fuRRx7h1KlT9OzZk+TkZJKTk3n66ac5fvw4P/30U3nHKIR4yLwQ6o2VmYaT8WlsO33N0OEYhloDvo/Ck1/pFgzptxSa9wMLe90qX3vnwvxu8HlDWPUmnPsHtPklej5doGCGsuiENFKzciuyNuIBlKnp+24OHz5MixYtih0CVZVI07cQhvfBXyf4Yfs5wuvVYOGrrQwdjvHIy4FzW+H4Mt1Y7awU/UuKjSuPZn3OhXQ1i19rRSvfGvc4kU67zzYTm5TJjy+H0K5+zfuWF+WjNHmmTHfUQghR0V5u44NGrWL7mescu5Ry/wMeFiZm4NcRenwDb525daf9Alg4kG1dmwvpasw0app7OuiGfJ3bds+OaLJAh/GTRC2EMEq1Ha14opk7AN9uizFwNEZKn7RnwttnWN9IN4y1uacDFnmpsHasbu7xeyzJKQt0GD9J1EIIo/XarQlQVh+9QlxSpoGjMXIaU7bE6+YfD6njpBubHdAHvNtAzfqF5Za9Dr/2h0OLIP2qfuKTg7E3yKuuw+GquFL1+n766afv+XpycvKDxCKEEEU09rCnrZ8z/5y+xvf/nmPik40NHZJRK9KRzL4mPDWzaIG8bDjxJ+RmwonlgIoGHi1427wua3KaER0fSuNaMu+3sSnVHbW9vf09N29vb1566aWKilUI8RAquKtesjeOGxk5Bo7GeF28kcml5Jto1Cp9c/Yd1KbQ/y/d3ONuzQAF1eX9RKp+ZaX5u9RZEAx/RuqSeVZqpcYv7q5Ud9Tz5s2rqDiEEKJYbeo508jdjhNXUvl4dRRTnw24/0EPoYLZyJrUssfG/C5/2tVq3fSktYN184+nXoEzGziz/Q/cru3EJucaHPxZt6lNwTsM/DpDixd1w8KEQcgzaiGEUVOpVLz3RCPUKli6/yJL91fMilNV3e4YXaIOLcH4aT07d2jxEgldv6NF9reMMJ0IrYaAU13dSl/ntsHGCUWPSY7VDRETlUYStRDC6IXVrcHICF2HqHeXH+VUQpqBIzI+BStmhfiUIlHfEuDpQJ7KlD/T6hMfNgGGH4BhB6DzZF3ivv1u+rcBMKUunN1cTpGL+5FELYSoEiLb16NNPWeycrUMWXiAzJxqsAxmOUlMyyLmWgYqlW5py9KyMTfB3123brV+OtEadSFsCHT6sLBgToZuDe3sVN1KYAWiVsLObyDp3INUQ9yFJGohRJWgUauY/lxzXGzNOZOYzrvLj8k6yrfsPadLrg3d7LC3Mi3TOQonPrnHSlpm1jDqJLz+L9i63RbAXFg3Fr5qDjNbwcb34eI+0Mpwr/IgiVoIUWU425jzVd9A1Cr448Alftsnz6sB9txaiKNUz6f/o8QTn6jV4PaflbYaPgF12oFKA1ej4N9p8F0H+LwBrBgG0Wt047pFmZRp9SwhhDCUVr41eLNTA6asi+a9P4/RzNOehm52hg7LoEqzEMfdBN9qMj9+OZWbOflYmhW/jGaxQgbptps34PRGiF4NZzbqVvw68KNuM7GEuu2h7mPgHQ41G+qSvrgv+SkJIaqcwY/UpV39mmTn6Z5XZ2Q/vM+rkzNzOBmv61xXlufTBTzsLXCzsyBPq3D4YnLZTmLpCM2ehWfnwdtn4cXlEPI/sPeCvJu6BL76LZgVBhf+LTwuO12aye9BErUQospRq1V80TsANzsLYq5mMG7Z0Yf2efXe87qmat+a1tS0NS/zeVQqFUG3phMtl/WpTcx0d9CPfwYjj8Dr23Vjt30f1fUirxVcWHbzJPisDuz97sGvWw1JohZCVEk1bMz5+vlANGoVyw9dZvHeOEOHZBCFz6fvv6Tl/QR53epQdv4eHcrKQqUCtya6GdFe+hPeOQdmVoWvX9oPWclg4VB038Jn4d/pcHE/5D+8rSbyjFoIUWW19HHirU4N+HTtSSasOE5AbQcaeTxcz6sLnk8/SEeyAgULdByITUarVVCrVQ98zmKp//P8e8AquHIYavgW7ovZAqfX6zYAMxvwDAWvMKjVQrdZPhzzkkuiFkJUaf9r58uec9fZHH2VyEUH+GtYm7tPoVnNpGfn6dfqfpCOZAX83e2wNNWQcjOXs1fT8XO1feBzlojGBGoH/SeYJ8HEAs7/Cxd26O64z27SbQWcfKFWUOHm1hRMLSsn5kokTd9CiCpNrVbxee/muNtbcO5aBmP/eHieV++/cAOtArUdLfFwePAEZapRE+Bprz+3QTn7QVgk9P1F11T++r/Q5VNo8owuQQMkxcDR32DtGPi+IyzqXfQcV09ViyZzSdRCiCrPydqMGc8HYqJW8dfhyyzcHWvokCpFwfPp8ribLhDsrTvXPkMn6tsVjN1u9To88z0MP6hL3i/8Du3fhfpdwdoFPAILj7l5A2a2hE+8IPu2KWez06CKfZAzikQ9c+ZMfHx8sLCwIDQ0lD179tyzfHJyMpGRkbi7u2Nubk79+vVZvXq1/vXJkyfTsmVLbG1tcXFxoUePHkRHR1d0NYQQBhTk7cQ7XRoA8MHKE/om4eqsYH7vVuXQkaxAUEknPjE0KyeoFwGPvA3PL4a3TumSdoGkGDCzBRsXML+tCX9xP5haH355Hv6Zpmtaz8ko8WUN0Vpj8ES9ZMkSRo0axYQJEzhw4AABAQF07tyZxMTEYsvn5OTQsWNHzp8/z9KlS4mOjmbu3LnUqlVLX2br1q1ERkaya9cuNmzYQG5uLp06dSIjo+RvhhCi6hnU1pcIfxdy8rRELjpAWlauoUOqMFm5+RyOK7/n0wVa3Or5HXMtg+vp2eV23gqnUumGhBWoFQRjLsDANYX7FAXij+omYoleBZveh/ndYLInzG4DK0fBoV/g2pli77q3nrpKnzm7iEuq3FnWVIqBH+aEhobSsmVLZsyYAYBWq8XT05Nhw4YxZsyYO8rPnj2bKVOmcPLkSUxNSzan7dWrV3FxcWHr1q20a9fuvuUvXryIp6cncXFx1K5du3QVEkIYVHJmDt2++pdLyTfp1tSdGc8HolJVUO9lA9p59jp95+7Cxdac3f/XoVzr2HHaVk4npjPnxSA6NXa7/wFVSe5NuHIELu4t3FIv3VnO0hFqt9RtTZ8hxcKTTtO3kpCazStt6vDeE40eKIzS5BmD3lHn5OSwf/9+IiIi9PvUajURERHs3Lmz2GNWrFhBWFgYkZGRuLq60qRJEyZNmkR+fv5dr5OSovvU6eRUfp86hRDGycHKjK9vPa9edfQKP+26YOiQKsSe26YNLe8PIgXDtPbHGnnzd1mYWoJXKLQeCr0XwKgT8MYJ6P0jtB6mG/5lYnFrOtT1sPljSI5l/IpjJKRm4+tszVudGlRqyAYdw3Dt2jXy8/NxdXUtst/V1ZWTJ08We0xMTAx///03/fr1Y/Xq1Zw5c4YhQ4aQm5vLhAkT7iiv1WoZOXIk4eHhNGnSpNhzZmdnk51d2MSTliZr3QpRlbXwcmRM14Z8tCqKj1ZGEejpSNPa9vc/sArZc/7WRCe+5fd8ukALL0d+2RPH/vPVMFEXx76Wbmv0lO77vBxIOKZbAeziHtbe8ODPQ6dRq+Dz3gGlmwe9HBj8GXVpabVaXFxcmDNnDkFBQfTp04dx48Yxe/bsYstHRkZy7NgxFi9efNdzTp48GXt7e/3WqNGDNWkIIQzvlTZ16NTIlZx8LUMW7SflZvV5Xp2Tp9UPnyqPiU7+q2CBjiOXUsjOu3trZbVlYqabUCX0NRI7zmDMqvOAbk30QK/Kn2TFoIna2dkZjUZDQkJCkf0JCQm4uRX/XMTd3Z369euj0RR+ovH39yc+Pp6cnJwiZYcOHcrKlSvZvHnzPZ8BjB07lpSUFP124sSJB6iVEMIYqFQqpjwTQG1HS+KSbjJ66ZFqM7766KUUsnK1OFqZUq+mTbmf36eGFTWszcjJ03LsUmq5n7+qUBSFMX8cJTkzl8Yedgx7zM8gcRi06dvMzIygoCA2bdpEjx49AN0d86ZNmxg6dGixx4SHh7No0SK0Wi3qW0uknTp1Cnd3d8zMdD3+FEVh2LBhLFu2jC1btlCnTp17xmFubo65eeFk9qmpD+9/TCGqE3srU2Y+34JnZu9g7fF4Wk3ehKOVGY5WZjhYmeJgZYq9pe5rx9u+drAyxeHW1xamldvMWRIFz6db+jhVyDSfKpWKFt6ObDiRwIELN/RDth42S/bG8ffJRMxM1HzRpzlmJoa5tzX4PHujRo2if//+BAcHExISwvTp08nIyGDgwIEAvPTSS9SqVYvJkycDMHjwYGbMmMGIESMYNmwYp0+fZtKkSQwfPlx/zsjISBYtWsSff/6Jra0t8fHxANjb22NpWf2mlxNC3F2ApwMTn2zM+D+Pk5CaTUJq6YYcWZiq9Um7f2sf+oZ4VVCkJVcRE538V9CtRL3vQhKD8L3/AdVM7PVMPlypa119u1MD6lfWdKrFMHii7tOnD1evXmX8+PHEx8fTvHlz1q5dq+9gFhsbq79zBvD09GTdunW88cYbNGvWjFq1ajFixAhGjx6tLzNr1iwAHn300SLXmjdvHgMGDKjwOgkhjEu/UG86NnIlPiWL5Mxckm/mkpKZw43M3Fvf55Bya/+NzMKv87UKWbla4nOziE/N4qOVJ3gywANrA84lnq9V2Herk1erCuhIViDYu2DJy2QURamWQ9zuJl+r8NZvh8nIySekjhMvt7l3q2xFM3iiBt2z5Ls1dW/ZsuWOfWFhYezateuu56suz6GEEOXHxdYCF1uLEpdXFIX07DxdIs/MZfjig5y7lsHKI5fp09Jwd9VRV1JJy87DxtwEf/eKWymsSS17zDRqrqVnE5uUiXcN6wq7lrH5/t8Y9pxPwtpMw+fPBqCpqFXESqjK9foWQojKoFKpsLUwxdPJiqa17enT0hPA4OteFyxrGezjWKEJxMJUQ5Naug8C+x6WYVpAdHwaU9edAmB890Z4Olnd54iKJ4laCCFKoFeL2pioVRyMTSY63nBzLVTG8+kCBcO0quXEJ8XIydPyxpJD5ORr6dDQhd7BnoYOCZBELYQQJVLT1pwIf13fmcV7DbM6l6Io+h7foeW4EMfdFMz7/bBMfPLVptOcuJKKo5Upk3s1NZrn8pKohRCihPqE6O6wlh28RFZu5U8EciYxnRuZuViYqmlaq+JnWisYlnUqMa1aTRhTnAOxN/hmyxkAJvVsWqr+DBVNErUQQpRQO7+aeNhbkJyZy7rj8ZV+/V237qZbeDlWypjemrbmeNewQlHgYDVu/s7MyePNXw+jVaBnYC26NnU3dEhFSKIWQogS0qhVPHvrueXiPZXfqez2hTgqS5VZn/oBfLLmJOeuZeBmZ8HEJxsbOpw7SKIWQohS6N3SE5UKdsZc5/y1ylvjXvd8uvI6khUoSNT7qmmi3nbqKj/u1K2wNuXZZthblmz55MokiVoIIUqhloMl7fxqArBkX+XdVccmZZKQmo2pRqXv5FUZgr11HwoOxSWTl6+ttOtWhpTMXN5ZegSA/mHetL31vhobSdRCCFFKfW91Klu6/yK5lZS8CsZPB9R2qNT5x/1cbLC1MCEzJ5+TBhyWVhEmrDhGfGoWvs7WjOnqb+hw7koStRBClNJjDV1xtjHjalo2m08mVso1d8dU/vNpALW68A5+3/mkSr12RVp15ArLD11GrYKpBlhjujQkUQshRCmZmajpFaRbOrcyZirLyM5j6yndB4LKTtRw27zfscmVfu2KkJiaxbvLjwIw5NF6lfoooSwkUQshRBn0udX7e0t0IldSblbotb7depZr6Tl417CidV3nCr1WcQo6lO2vBnfUBWtM38jMpZG7HcM7GGaN6dKQRC2EEGXgW9OG0DpOaBX4bd/FCrvOlZSbzPknBoAxXRoaZE3kAE8HNGoVl1OyuJxcsR9KKpp+jWmNYdeYLg3jj1AIIYzUc7c6lS3ZG4dWWzGr9k1ZF01WrpaWPo50aeJWIde4H2tzE/zddesx76/Cw7RuX2P6rc71aeBmuDWmS0MStRBClFHXJu7YWZhwKfkm/565Vu7nP3oxhT8OXALg3W6NDDr3dMEwrW+2nCXmarrB4igrRVF45/dba0z7OPFKG19Dh1RikqiFEKKMLEw19AysBejuqsuToih8tEp399ejuQcBng7lev7S6hvihZ2FCVFXUun21b8s3H0BRamYVoSKsOZYPLtikrAwVTPVCNaYLg1J1EII8QD6tPQCYP2JeK6nZ5fbedefSGD3uSTMTdS83aVhuZ23rBq42bJ2ZDta163Bzdx8xi07xqsL9nE1rfzqXFGycvOZtDoKgP+1q4tXDcOvMV0akqiFEOIBNPKwI6C2Pbn5ir6Z+kHl5Gn5ZM1JAF5tW4daDpblct4H5eFgyc+vhPJuN3/MNGo2nUyky/RtbDyRYOjQ7un7f89x8cZN3Ows+N8jVafJu4AkaiGEeEAFd9W/7I0tl+bgn3dd4Ny1DJxtzBj8aL0HPl95UqtVvNrWlxXDwmnoZsv1jBxe/XEfY/84QkZ2nqHDu0NCahYzN+uWrxzTtSFWZiYGjqj0JFELIcQDerK5B1ZmGmKuZjzw4hXJmTl8uek0AG92aoCNuXEmloZudiyPDGdQ2zqoVPDLnji6ffWP0S2HOWVdNJk5+QR6OfBUcw9Dh1MmkqiFEOIB2Zib8EQz3RrGv+yJfaBzff33GVJu5tLA1ZbetyZVMVYWphrGdWvEwldDcbe34Pz1TJ6ZvZMvNpwyigU8jlxMZul+3Rj38U8Yttf8g5BELYQQ5eC5EF3z9+qjV0i5mVumc5y/lsGPO88DMK6bf5Xpmdy6rjNrR7TjyQAP8rUKX246zTOzd3KuEpcB/S9FUfjgL12v+Z6BtQg08mlC78UoEvXMmTPx8fHBwsKC0NBQ9uzZc8/yycnJREZG4u7ujrm5OfXr12f16tUPdE4hhHgQgZ4O1He1IStXy4pDZetU9smak+TmKzxSvybt6hvnkot3Y29lyld9A/nyuebYWphwKC6Zx7/8h0W7y+e5fWmtPHKFfRduYGmqYbQR9Jp/EAZP1EuWLGHUqFFMmDCBAwcOEBAQQOfOnUlMLH5FmpycHDp27Mj58+dZunQp0dHRzJ07l1q1apX5nEII8aBUKhXP3epUVpaFOnbHXGft8XjUKt3ddFX1VPNarB3Zjla+TtzMzef/lh1l0I/7uFaOQ9fuJys3X99rfvCjdXGzt6i0a1cEgyfqadOmMWjQIAYOHEijRo2YPXs2VlZW/PDDD8WW/+GHH0hKSmL58uWEh4fj4+PDI488QkBAQJnPKYQQ5aFnYC3MNGqOX07l2KWUEh+n1Sp8fGucb98QL+q7Vo2pLe+mloMli15txbjHdcO4NkbphnFtiqqcYVxztsVwKfkmHvYWDGpb9YZj/ZdBE3VOTg779+8nIiJCv0+tVhMREcHOnTuLPWbFihWEhYURGRmJq6srTZo0YdKkSeTn55f5nEIIUR4crc3083GXplPZn4cvceRiCjbmJrzRsX5FhVep1GoVg9r58ufQcBq42nItPYdXFuxj8uqoCm0Kj0/JYtaWswCMedzfqNeZLimDJupr166Rn5+Pq6trkf2urq7Ex8cXe0xMTAxLly4lPz+f1atX89577/H555/z0Ucflfmc2dnZpKam6re0tLRyqJ0Q4mH0XEtdT+0Vhy6TmXP/ccU3c/L5bG00AEPa18XZxrxC46ts/u52/Dk0nFfb1AHg220xTF0fXWHX+2ztSW7m5hPs7Uj3Wz3xqzqDN32XllarxcXFhTlz5hAUFESfPn0YN24cs2fPLvM5J0+ejL29vX5r1KhROUYshHiYtPKtgXcNK9Ky81h15Mp9y3//bwxXUrKo5WDJy+F1KiHCymdhquHdJxrxYY8mAMzcfJZvtpwp9+scjL3BHwd1HfnGd6+6w7H+y6CJ2tnZGY1GQ0JC0ecWCQkJuLkVv5ybu7s79evXR6MpbM7w9/cnPj6enJycMp1z7NixpKSk6LcTJ048YM2EEA8rtVqlH/98v4U6EtOy+OZWM+07XRpgYVr1m2nv5cVW3oztquuB/dnaaBbsOF9u51YUhQ9uLWHZq0VtmtV2KLdzG5pBE7WZmRlBQUFs2rRJv0+r1bJp0ybCwsKKPSY8PJwzZ86g1RYOpj916hTu7u6YmZmV6Zzm5ubY2dnpN1vbqt2RQwhhWM8G1UajVrHvwg1OJ9z9UdoXG06RmZNPc08HngyomrNmldb/HqnL8Md006JOWHGc3/aVz6pjKw5f5mBsMlZmGt7p0qBczmksDN70PWrUKObOncuCBQuIiopi8ODBZGRkMHDgQABeeuklxo4dqy8/ePBgkpKSGDFiBKdOnWLVqlVMmjSJyMjIEp9TCCEqkoudBY81dAHufld9Mj5V/9p7T/hXm2baknijY319M//o34+U6BHBvWTm5OmHY0W2r4erXdUejvVfBp9Etk+fPly9epXx48cTHx9P8+bNWbt2rb4zWGxsLGp14ecJT09P1q1bxxtvvEGzZs2oVasWI0aMYPTo0SU+pxBCVLS+IZ5sOJHA7wcu8naXBpibFDZrK4rCx6ui0CrQrak7Qd5OBoy08qlUKt57wp+M7DyW7Itj5JKDWJlpaH/rw01pfbu18Dn/K22q33N+lVKVVv6uJBcvXsTT05O4uDhq165t6HCEEFVQXr6WNp9uJj41ixnPB/JEs8Km7c3RiQyct1c3xnjUI1VufeTykq9VGLnkEH8dvoy5iZr5A0MIq1ujVOe4nHyTxz7fQlaulpnPt6BbFenpXZo8Y/CmbyGEqI5MNGqeDdb9AV68p7D5Oy9fy8erdJObDAj3eWiTNIBGrWJa7wAi/F3IztPy6oK9pV5969O1J8nK1RLi48TjTYvvMFzVSaIWQogK0jvYE5UK/j1zjbikTEA3veiZxHQcrUyJbG9ca00bgqlGzYznWxBerwYZOfkMmLeXqCupJTp2/4Ub/HnoMipV9RqO9V+SqIUQooJ4OlnRpp4zoOtUlpaVyxcbTgEwMqI+9pamhgzPaFiYapjzYjAtvBxIuZnLi9/v5uzV9Hseo9UWDsd6Nqg2TWrZV0aoBiGJWgghKlDBQh2/7Y/j67/PcD0jB9+a1jwf6mXgyIyLtbkJ8waG0MjdjmvpObzw3W59K0Rxlh+6xOG4ZGzMTXirc/UajvVfkqiFEKICRTRywcnajITUbOZsiwHg/7r6Y6qRP7//ZW9pyk+vhFC3pjVXUrJ44fvdJKZm3VEuIzuPT9cWDsdysa1ew7H+S/6nCCFEBTI30dCrReEyvK3r1qCDf9mGIT0MatiYs/DVVng6WXLheib9vttNUkZOkTLfbj1LQmo2Xk5WvNzGxzCBViJJ1EIIUcH63Gr+Vt1aa7q6dnoqL272Fix6tRWuduacTkyn/w97SM3KBeDijUy+LWiZeLxhkfHp1ZUkaiGEqGD1XGz4pl8Lvn0hiMYe1bfTU3nydLJi4auhOFmbcfRSCq/M36ufgSw7T0srXyc6N66ew7H+SxK1EEJUgsebutPpIUks5aWeiy0/vhyCrYUJe8/foPe3O1l55IpuONYTjR+alglJ1EIIIYxWk1r2zB/YEiszDccu6cZXP9fSk0YedgaOrPJIohZCCGHUgrydmPtSMGYmapyszXizU/UejvVfBl+UQwghhLif8HrO7BjzGADONuYGjqZySaIWQghRJTxsCbqANH0LIYQQRkwStRBCCGHEJFELIYQQRkwStRBCCGHEJFELIYQQRkx6fRdDq9UCcOXKFQNHIoQQojoqyC8F+eZeJFEXIyEhAYCQkBADRyKEEKI6S0hIwMvr3muTqxRFUSopniojLy+PgwcP4urqilr9YE8H0tLSaNSoESdOnMDW1racIhRCCFHZyvPvuVarJSEhgcDAQExM7n3PLIm6gqWmpmJvb09KSgp2dg/P3LRCCFHdGOrvuXQmE0IIIYyYJGohhBDCiEmirmDm5uZMmDABc/OHc45aIYSoLgz191yeUQshhBBGTO6ohRBCCCMmiVoIIYQwYpKohRBCCCMmiboCzZw5Ex8fHywsLAgNDWXPnj2GDkkIIUQpbdu2je7du+Ph4YFKpWL58uWVen1J1BVkyZIljBo1igkTJnDgwAECAgLo3LkziYmJhg5NCCFEKWRkZBAQEMDMmTMNcn3p9V1BQkNDadmyJTNmzAB008V5enoybNgwxowZY+DohBBClIVKpWLZsmX06NGj0q4pd9QVICcnh/379xMREaHfp1ariYiIYOfOnQaMTAghRFUjiboCXLt2jfz8fFxdXYvsd3V1JT4+3kBRCSGEqIokUQshhBBGTBJ1BXB2dkaj0ejXtS6QkJCAm5ubgaISQghRFUmirgBmZmYEBQWxadMm/T6tVsumTZsICwszYGRCCCGqmnuvVi3KbNSoUfTv35/g4GBCQkKYPn06GRkZDBw40NChCSGEKIX09HTOnDmj//7cuXMcOnQIJycnvLy8Kvz6MjyrAs2YMYMpU6YQHx9P8+bN+eqrrwgNDTV0WEIIIUphy5YttG/f/o79/fv3Z/78+RV+fUnUQgghhBGTZ9RCCCGEEZNELYQQQhgxSdRCCCGEEZNELYQQQhgxSdRCCCGEEZNELYQQQhgxSdRCCCGEEZNELYQQQhgxSdRCiEqlUqlYvny5ocMQosqQRC3EQ2TAgAGoVKo7ti5duhg6NCHEXciiHEI8ZLp06cK8efOK7DM3NzdQNEKI+5E7aiEeMubm5ri5uRXZHB0dAV2z9KxZs+jatSuWlpb4+vqydOnSIscfPXqUxx57DEtLS2rUqMFrr71Genp6kTI//PADjRs3xtzcHHd3d4YOHVrk9WvXrtGzZ0+srKzw8/NjxYoV+tdu3LhBv379qFmzJpaWlvj5+d3xwUKIh4kkaiFEEe+99x69evXi8OHD9OvXj+eee46oqCgAMjIy6Ny5M46Ojuzdu5fffvuNjRs3FknEs2bNIjIyktdee42jR4+yYsUK6tWrV+Qa77//Pr179+bIkSM8/vjj9OvXj6SkJP31T5w4wZo1a4iKimLWrFk4OztX3g9ACGOjCCEeGv3791c0Go1ibW1dZPv4448VRVEUQHn99deLHBMaGqoMHjxYURRFmTNnjuLo6Kikp6frX1+1apWiVquV+Ph4RVEUxcPDQxk3btxdYwCUd999V/99enq6Aihr1qxRFEVRunfvrgwcOLB8KixENSDPqIV4yLRv355Zs2YV2efk5KT/OiwsrMhrYWFhHDp0CICoqCgCAgKwtrbWvx4eHo5WqyU6OhqVSsXly5fp0KHDPWNo1qyZ/mtra2vs7OxITEwEYPDgwfTq1YsDBw7QqVMnevToQevWrctUVyGqA0nUQjxkrK2t72iKLi+WlpYlKmdqalrke5VKhVarBaBr165cuHCB1atXs2HDBjp06EBkZCRTp04t93iFqArkGbUQoohdu3bd8b2/vz8A/v7+HD58mIyMDP3r27dvR61W06BBA2xtbfHx8WHTpk0PFEPNmjXp378/P//8M9OnT2fOnDkPdD4hqjK5oxbiIZOdnU18fHyRfSYmJvoOW7/99hvBwcG0adOGhQsXsmfPHr7//nsA+vXrx4QJE+jfvz8TJ07k6tWrDBs2jBdffBFXV1cAJk6cyOuvv46Liwtdu3YlLS2N7du3M2zYsBLFN378eIKCgmjcuDHZ2dmsXLlS/0FBiIeRJGohHjJr167F3d29yL4GDRpw8uRJQNcje/HixQwZMgR3d3d++eUXGjVqBICVlRXr1q1jxIgRtGzZEisrK3r16sW0adP05+rfvz9ZWVl88cUXvPXWWzg7O/PMM8+UOD4zMzPGjh3L+fPnsbS0pG3btixevLgcai5E1aRSFEUxdBBCCOOgUqlYtmwZPXr0MHQoQohb5Bm1EEIIYcQkUQshhBBGTJ5RCyH05EmYEMZH7qiFEEIIIyaJWgghhDBikqiFEEIIIyaJWgghhDBikqiFEEIIIyaJWgghhDBikqiFEEIIIyaJWgghhDBikqiFEEIII/b/ejo2wAhfdUIAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from previous_chapters import plot_losses\n", "\n", "\n", "epochs_tensor = torch.linspace(0, num_epochs, len(tracking[\"train_losses\"]))\n", "plot_losses(\n", " epochs_seen=epochs_tensor,\n", " tokens_seen=tracking[\"tokens_seen\"],\n", " train_losses=tracking[\"train_losses\"],\n", " val_losses=tracking[\"val_losses\"],\n", " label=\"loss\"\n", ")" ] }, { "cell_type": "markdown", "id": "7f8bc233-895f-46d5-8e01-202b991cd60c", "metadata": { "id": "7f8bc233-895f-46d5-8e01-202b991cd60c" }, "source": [ "- **如上所示，损失（Loss）持续下降，这是一个积极的信号**。 \n", "- **从下降趋势来看**，可能会有 **进一步训练的空间**， \n", " **（建议读者可以尝试继续训练）**， \n", " **但需要注意 DPO 可能存在“塌陷”（Collapse）风险**， \n", " **即模型可能会开始生成无意义的响应**。 \n", "- **接下来，我们来看奖励边际（Reward Margins）：** " ] }, { "cell_type": "code", "execution_count": 52, "id": "dmbq6ruuf0Cl", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 307 }, "id": "dmbq6ruuf0Cl", "outputId": "c2886c16-57da-41bd-c9f0-e936da9d9e4d" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABn+ElEQVR4nO3deVhUZfvA8e8My7CDoGyyKyIq4o6IlaaJS+bSYmalWfZaLpkt5luZ1q+stLKyslV7K9PMNFNTcU/FXRQ3XFhVFhXZ95nz+2NkkEQFBAbw/lzXXHLOec4592Fw7nnOeRaVoigKQgghhKiX1MYOQAghhBA3JolaCCGEqMckUQshhBD1mCRqIYQQoh6TRC2EEELUY5KohRBCiHpMErUQQghRj0miFkIIIeoxSdRCCCFEPSaJWohGJD4+HpVKRVRUlLFDEULUEEnUQtQzKpXqpq+ZM2caO0QhRB0yNXYAQojykpOTDT8vXbqUGTNmEBMTY1hnY2NjjLCEEEYiNWoh6hlXV1fDy97eHpVKZVh2dnbm448/xsPDA41GQ4cOHVi3bt0Nj6XVahk7diytW7cmMTERgD///JNOnTphYWGBn58fs2bNoqSkxLCPSqXiu+++Y9iwYVhZWeHv78+qVasM269cucKoUaNo1qwZlpaW+Pv7s3DhwhvG8PvvvxMUFISlpSVOTk707duX3Nxcw/bvvvuOwMBALCwsaN26NV9++WW5/ZOSknjkkUdwcHDA0dGRIUOGEB8fb9g+ZswYhg4dyty5c3Fzc8PJyYkJEyZQXFxc6d+5EPWaIoSotxYuXKjY29sblj/++GPFzs5O+fXXX5WTJ08qr776qmJmZqacOnVKURRFiYuLUwDl0KFDSkFBgTJs2DClY8eOSlpamqIoirJ9+3bFzs5OWbRokXL27Fllw4YNio+PjzJz5kzDOQDFw8NDWbx4sXL69Gll8uTJio2NjXL58mVFURRlwoQJSocOHZR9+/YpcXFxSkREhLJq1aoK479w4YJiamqqfPzxx0pcXJxy5MgR5YsvvlCys7MVRVGUn3/+WXFzc1OWL1+uxMbGKsuXL1ccHR2VRYsWKYqiKEVFRUpgYKAyduxY5ciRI8rx48eVxx57TAkICFAKCwsVRVGU0aNHK3Z2dsr48eOVEydOKH/99ZdiZWWlfPPNNzX7ZghhJJKohajH/p2o3d3dlXfffbdcma5duyrPP/+8oihlifqff/5R+vTpo/Ts2VPJyMgwlO3Tp4/y3nvvldv/p59+Utzc3AzLgPLGG28YlnNychRA+fvvvxVFUZTBgwcrTz31VKXiP3DggAIo8fHxFW5v0aKFsnjx4nLr3nnnHSU0NNQQW0BAgKLT6QzbCwsLFUtLS2X9+vWKougTtbe3t1JSUmIo8/DDDysjRoyoVIxC1HfyjFqIBiIrK4sLFy4QFhZWbn1YWBiHDx8ut27kyJF4eHiwefNmLC0tDesPHz7Mzp07effddw3rtFotBQUF5OXlYWVlBUD79u0N262trbGzsyMtLQ2A5557jgcffJCDBw/Sr18/hg4dSo8ePSqMOTg4mD59+hAUFER4eDj9+vXjoYceokmTJuTm5nL27Fmefvppxo0bZ9inpKQEe3t7Q7xnzpzB1ta23HELCgo4e/asYblt27aYmJgYlt3c3IiOjr7Jb1OIhkMStRCN0MCBA/n555+JjIzk3nvvNazPyclh1qxZDB8+/Lp9LCwsDD+bmZmV26ZSqdDpdAAMGDCAhIQE1q5dS0REBH369GHChAnMnTv3umOamJgQERHBrl272LBhA59//jmvv/46e/bsMXwp+PbbbwkJCbluv9J4O3fuzC+//HLdsZs1a1apeIVo6CRRC9FA2NnZ4e7uzs6dO7nnnnsM63fu3Em3bt3KlX3uuedo164dDzzwAGvWrDGU79SpEzExMbRs2fK2YmnWrBmjR49m9OjR3HXXXbzyyisVJmrQJ82wsDDCwsKYMWMG3t7erFixgqlTp+Lu7k5sbCyjRo2qcN9OnTqxdOlSnJ2dsbOzu62YhWioJFEL0YC88sorvPXWW7Ro0YIOHTqwcOFCoqKiKqxxTpo0Ca1Wy/3338/ff/9Nz549mTFjBvfffz9eXl489NBDqNVqDh8+zNGjR/m///u/SsUwY8YMOnfuTNu2bSksLGT16tUEBgZWWHbPnj1s2rSJfv364ezszJ49e7h48aKh/KxZs5g8eTL29vb079+fwsJC9u/fz5UrV5g6dSqjRo1izpw5DBkyhLfffhsPDw8SEhL4448/ePXVV/Hw8Kj+L1OIBkIStRANyOTJk8nMzOSll14iLS2NNm3asGrVKvz9/SssP2XKFHQ6HQMHDmTdunWEh4ezevVq3n77bT744APMzMxo3bo1zzzzTKVjMDc3Z/r06cTHx2Npacldd93FkiVLKixrZ2fH9u3bmTdvHllZWXh7e/PRRx8xYMAAAJ555hmsrKyYM2cOr7zyCtbW1gQFBTFlyhQArKys2L59O9OmTWP48OFkZ2fTvHlz+vTpIzVsccdQKYqiGDsIIYQQQlRMBjwRQggh6jFJ1EIIIUQ9JolaCCGEqMckUQshhBD1mCRqIYQQoh6TRC2EEELUY5Koq+iLL77Ax8cHCwsLQkJC2Lt3b52ef/v27QwePBh3d3dUKhUrV64st11RFGbMmIGbmxuWlpb07duX06dPlyuTnp7OqFGjsLOzw8HBgaeffpqcnJxyZY4cOcJdd92FhYUFnp6efPjhh9fFsmzZMlq3bo2FhQVBQUGsXbu2Stcye/Zsunbtiq2tLc7OzgwdOrTcvMugH9N5woQJODk5YWNjw4MPPkhqamq5MomJiQwaNAgrKyucnZ155ZVXyk3bCLB161Y6deqERqOhZcuWLFq06Lp4bve9/eqrr2jfvj12dnbY2dkRGhrK33//3SCv5d/ef/99VCqVoX9zQ7uemTNnolKpyr1at27dIK8F4Pz58zz++OM4OTlhaWlJUFAQ+/fvN2xvSJ8DPj4+1703KpWKCRMmAA3vvakVxp0TpGFZsmSJYm5urvzwww/KsWPHlHHjxikODg5KampqncWwdu1a5fXXX1f++OMPBVBWrFhRbvv777+v2NvbKytXrlQOHz6sPPDAA4qvr6+Sn59vKNO/f38lODhY2b17t/LPP/8oLVu2VEaOHGnYnpmZqbi4uCijRo1Sjh49qvz666+KpaWl8vXXXxvK7Ny5UzExMVE+/PBD5fjx48obb7yhmJmZKdHR0ZW+lvDwcGXhwoXK0aNHlaioKGXgwIGKl5eXkpOTYygzfvx4xdPTU9m0aZOyf/9+pXv37kqPHj0M20tKSpR27dopffv2VQ4dOqSsXbtWadq0qTJ9+nRDmdjYWMXKykqZOnWqcvz4ceXzzz9XTExMlHXr1hnK1MR7u2rVKmXNmjXKqVOnlJiYGOW///2vYmZmphw9erTBXcu19u7dq/j4+Cjt27dXXnjhBcP6hnQ9b731ltK2bVslOTnZ8Lp48WKDvJb09HTF29tbGTNmjLJnzx4lNjZWWb9+vXLmzBlDmYb0OZCWllbufYmIiFAAZcuWLYqiNKz3prZIoq6Cbt26KRMmTDAsa7Vaxd3dXZk9e7ZR4vl3otbpdIqrq6syZ84cw7qMjAxFo9Eov/76q6IoinL8+HEFUPbt22co8/fffysqlUo5f/68oiiK8uWXXypNmjQxzPerKIoybdo0JSAgwLD8yCOPKIMGDSoXT0hIiPKf//yn2teTlpamAMq2bdsMsZuZmSnLli0zlDlx4oQCKJGRkYqi6L+4qNVqJSUlxVDmq6++Uuzs7Azxv/rqq0rbtm3LnWvEiBFKeHi4Ybm23tsmTZoo3333XYO9luzsbMXf31+JiIhQ7rnnHkOibmjX89ZbbynBwcEVbmto1zJt2jSlZ8+eN9ze0D8HXnjhBaVFixaKTqdrcO9NbZFb35VUVFTEgQMH6Nu3r2GdWq2mb9++REZGGjGyMnFxcaSkpJSL0d7enpCQEEOMkZGRODg40KVLF0OZvn37olar2bNnj6HM3Xffjbm5uaFMeHg4MTExXLlyxVDm2vOUlrmd30VmZiYAjo6OABw4cIDi4uJy52ndujVeXl7lricoKAgXF5dycWRlZXHs2LFKxVob761Wq2XJkiXk5uYSGhraYK9lwoQJDBo06LpzNsTrOX36NO7u7vj5+TFq1CgSExMb5LWsWrWKLl268PDDD+Ps7EzHjh359ttvDdsb8udAUVERP//8M2PHjkWlUjW496a2SKKupEuXLqHVasv9MQC4uLiQkpJipKjKK43jZjGmpKTg7OxcbrupqSmOjo7lylR0jGvPcaMy1f1d6HQ6pkyZQlhYGO3atTOcw9zcHAcHh5teT3VjzcrKIj8/v0bf2+joaGxsbNBoNIwfP54VK1bQpk2bBnktS5Ys4eDBg8yePfu6bQ3tekJCQli0aBHr1q3jq6++Ii4ujrvuuovs7OwGdy2xsbF89dVX+Pv7s379ep577jkmT57Mjz/+WC6ehvg5sHLlSjIyMhgzZozh+A3pvaktMimHqBcmTJjA0aNH2bFjh7FDuS0BAQFERUWRmZnJ77//zujRo9m2bZuxw6qypKQkXnjhBSIiIsrNU91QlU4CAtC+fXtCQkLw9vbmt99+w9LS0oiRVZ1Op6NLly689957AHTs2JGjR4+yYMECRo8ebeTobs/333/PgAEDcHd3N3Yo9YrUqCupadOmmJiYXNfaMDU1FVdXVyNFVV5pHDeL0dXVlbS0tHLbS0pKSE9PL1emomNce44blanO72LixImsXr2aLVu2lJu20NXVlaKiIjIyMm56PdWN1c7ODktLyxp9b83NzWnZsiWdO3dm9uzZBAcH8+mnnza4azlw4ABpaWl06tQJU1NTTE1N2bZtG5999hmmpqa4uLg0qOv5NwcHB1q1asWZM2ca3Hvj5uZGmzZtyq0LDAw03MpvqJ8DCQkJbNy4sdxMbg3tvaktkqgrydzcnM6dO7Np0ybDOp1Ox6ZNmwgNDTViZGV8fX1xdXUtF2NWVhZ79uwxxBgaGkpGRgYHDhwwlNm8eTM6nY6QkBBDme3bt1NcXGwoExERQUBAAE2aNDGUufY8pWWq8rtQFIWJEyeyYsUKNm/ejK+vb7ntnTt3xszMrNx5YmJiSExMLHc90dHR5T50IiIisLOzM3yY3SrW2nxvdTodhYWFDe5a+vTpQ3R0NFFRUYZXly5dGDVqlOHnhnQ9/5aTk8PZs2dxc3NrcO9NWFjYdd0YT506hbe3N9DwPgdKLVy4EGdnZwYNGmRY19Dem1pj7NZsDcmSJUsUjUajLFq0SDl+/Ljy7LPPKg4ODuVaG9a27Oxs5dChQ8qhQ4cUQPn444+VQ4cOKQkJCYqi6LtlODg4KH/++ady5MgRZciQIRV2y+jYsaOyZ88eZceOHYq/v3+5bhkZGRmKi4uL8sQTTyhHjx5VlixZolhZWV3XLcPU1FSZO3eucuLECeWtt96qcreM5557TrG3t1e2bt1arntGXl6eocz48eMVLy8vZfPmzcr+/fuV0NBQJTQ01LC9tGtGv379lKioKGXdunVKs2bNKuya8corrygnTpxQvvjiiwq7Ztzue/vaa68p27ZtU+Li4pQjR44or732mqJSqZQNGzY0uGupyLWtvhva9bz00kvK1q1blbi4OGXnzp1K3759laZNmyppaWkN7lr27t2rmJqaKu+++65y+vRp5ZdfflGsrKyUn3/+2VCmIX0OKIq+hbWXl5cybdq067Y1pPemtkiirqLPP/9c8fLyUszNzZVu3bopu3fvrtPzb9myRQGue40ePVpRFH3XjDfffFNxcXFRNBqN0qdPHyUmJqbcMS5fvqyMHDlSsbGxUezs7JSnnnpKyc7OLlfm8OHDSs+ePRWNRqM0b95cef/996+L5bffflNatWqlmJubK23btlXWrFlTpWup6DoAZeHChYYy+fn5yvPPP680adJEsbKyUoYNG6YkJyeXO058fLwyYMAAxdLSUmnatKny0ksvKcXFxdf93jp06KCYm5srfn5+5c5R6nbf27Fjxyre3t6Kubm50qxZM6VPnz6GJN3QrqUi/07UDel6RowYobi5uSnm5uZK8+bNlREjRpTrd9yQrkVRFOWvv/5S2rVrp2g0GqV169bKN998U257Q/ocUBRFWb9+vQJcF6OiNLz3pjaoFEVRjFKVF0IIIcQtyTNqIYQQoh6TRC2EEELUY5KohRBCiHpMErUQQghRj0miFkIIIeoxSdRCCCFEPSaJuooKCwuZOXMmhYWFxg6lRjSm62lM1wKN63oa07VA47qexnQt0PiuB0D6UVdRVlYW9vb2ZGZmYmdnZ+xwbltjup7GdC3QuK6nMV0LNK7raUzXAo3vekBq1EIIIUS9JolaCCGEqMfuuPmoS0pKOHToEC4uLqjVVf+ekp2dDcD58+fJysqq6fDqXGO6nsZ0LdC4rqcxXQs0rutpTNcCDed6dDodqampdOzYEVPTm6fiO+4Z9b59++jWrZuxwxBCCCHYu3cvXbt2vWkZo9aoZ8+ezR9//MHJkyextLSkR48efPDBBwQEBNxwn0WLFvHUU0+VW6fRaCgoKKjUOV1cXAD9L8fNza36wQshhBDVlJycTLdu3Qw56WaMmqi3bdvGhAkT6Nq1KyUlJfz3v/+lX79+HD9+HGtr6xvuZ2dnV27idJVKVelzlt7udnNzw8PDo/rBCyGEELepMo9gjZqo161bV2550aJFODs7c+DAAe6+++4b7qdSqXB1da3t8IQQQgijq1etvjMzMwFwdHS8abmcnBy8vb3x9PRkyJAhHDt2rC7CE0IIIepcvUnUOp2OKVOmEBYWRrt27W5YLiAggB9++IE///yTn3/+GZ1OR48ePTh37lyF5QsLC8nKyjK8SlsECiGEEA1BvemeNWHCBI4ePcqOHTtuWi40NJTQ0FDDco8ePQgMDOTrr7/mnXfeua787NmzmTVrVpXj0Wq1FBcXV3k/Ia5lZmaGiYmJscMQQjRg9SJRT5w4kdWrV7N9+/YqN/AyMzOjY8eOnDlzpsLt06dPZ+rUqYbl8+fP06ZNmxseT1EUUlJSyMjIqFIcQtyIg4MDrq6uVWr0KIQoL+5SLnYWpjjZaIwdSp0zaqJWFIVJkyaxYsUKtm7diq+vb5WPodVqiY6OZuDAgRVu12g0aDRlb+ytOsCXJmlnZ2esrKzkw1VUm6Io5OXlkZaWBiDdAYWopsizl3n8+z14NLFkw4t3ozG9s+5SGTVRT5gwgcWLF/Pnn39ia2tLSkoKAPb29lhaWgLw5JNP0rx5c2bPng3A22+/Tffu3WnZsiUZGRnMmTOHhIQEnnnmmduOR6vVGpK0k5PTbR9PiNK/47S0NJydneU2uBBVlJFXxItLo9DqFBIu5/H7gXOMCvE2dlh1yqiNyb766isyMzPp1asXbm5uhtfSpUsNZRITE0lOTjYsX7lyhXHjxhEYGMjAgQPJyspi165dN72dXVmlz6StrKxu+1hClCr9e5I2D0JUjaIoTFt+hJSsAsxN9enqyy1nKSrRGTmyumX0W9+3snXr1nLLn3zyCZ988kktRaQnt7tFTZK/JyGqZ/HeRNYfS8XMRMWv40IY//NBzmfk88fBczzazcvY4dWZetM9SwghhCh1OjWbd1YfB2Ba/9Z09nbkP3f7ATB/yxmKtXdOrVoStbghHx8f5s2bV+nyW7duRaVS1XqL+UWLFuHg4FCr5xBCGE9BsZZJvx6ioFjH3a2aMTZM39B4VIg3TW00nLuSz4qD540cZd2RRN0IqFSqm75mzpxZrePu27ePZ599ttLle/ToQXJyMvb29tU6nxBCALz/90lOpmTT1Macjx4ORq3WPz6yNDe5I2vV9aIftbg91za2W7p0KTNmzCg3aYmNjY3hZ0VR0Gq1t5z/FKBZs2ZVisPc3FzGYBdC3JbNJ1NZtCsegLkPB9PMtny/6VHdvViw7SyJ6XmsPHSeh7t4GiHKuiU16kbA1dXV8LK3tzdMWuLq6srJkyextbXl77//pnPnzmg0Gnbs2MHZs2cZMmQILi4u2NjY0LVrVzZu3FjuuP++9a1Sqfjuu+8YNmwYVlZW+Pv7s2rVKsP2f9/6Lr1FvX79egIDA7GxsaF///7lvliUlJQwefJkHBwccHJyYtq0aYwePZqhQ4dW6Xfw1Vdf0aJFC8zNzQkICOCnn34ybFMUhZkzZ+Ll5YVGo8Hd3Z3Jkycbtn/55Zf4+/tjYWGBi4sLDz30UJXOLcSNrI1OZsG2s+h0t244KyAtq4CXlx0BYGyYL70CnK8rY2VuyrhratUld0CtWhL1LSiKQl5RiVFelWkVX1mvvfYa77//PidOnKB9+/bk5OQwcOBANm3axKFDh+jfvz+DBw8mMTHxpseZNWsWjzzyCEeOHGHgwIGMGjWK9PT0G5bPy8tj7ty5/PTTT2zfvp3ExERefvllw/YPPviAX375hYULF7Jz506ysrJYuXJlla5txYoVvPDCC7z00kscPXqU//znPzz11FNs2bIFgOXLl/PJJ5/w9ddfc/r0aVauXElQUBAA+/fvZ/Lkybz99tvExMSwbt26m87cJkRlpecWMWVJFO//fZIVh+6c56nVpdMpTP3tMOm5RbRxs2PagIAbln2iuzeO1uYkXM7jz6gLdRilccit71vIL9bSZsZ6o5z7+NvhWJnXzFv09ttvc9999xmWHR0dCQ4ONiy/8847rFixglWrVjFx4sQbHmfMmDGMHDkSgPfee4/PPvuMvXv30r9//wrLFxcXs2DBAlq0aAHoh4t9++23Dds///xzpk+fzrBhwwCYP38+a9eurdK1zZ07lzFjxvD8888DMHXqVHbv3s3cuXPp3bs3iYmJuLq60rdvX8zMzPDy8qJbt26Avp++tbU1999/P7a2tnh7e9OxY8cqnV+Iivxx8BxFV2t7c9bHMDDIDUtzGfDmRr79J5YdZy5haWbCZyM73nT0MWuNKc/c5cuH62KYv+UMQzq4Y2rSeOudjffKRDldunQpt5yTk8PLL79MYGAgDg4O2NjYcOLEiVvWqNu3b2/42draGjs7O8MQmRWxsrIyJGnQD6NZWj4zM5PU1FRD0gQwMTGhc+fOVbq2EydOEBYWVm5dWFgYJ06cAODhhx8mPz8fPz8/xo0bx4oVKygpKQHgvvvuw9vbGz8/P5544gl++eUX8vLyqnR+If5NURQW79X/XzJRq0jJKuDbf2KNHFX9deRcBnPW69vVvDW4DS2dbW6xBzwZ6oODlRlxl3L560jjrlVLjfoWLM1MOP52uNHOXVOsra3LLb/88stEREQwd+5cWrZsiaWlJQ899BBFRUU3PY6ZmVm5ZZVKhU5342dEFZWvyVv6leHp6UlMTAwbN24kIiKC559/njlz5rBt2zZsbW05ePAgW7duZcOGDcyYMYOZM2eyb98+6QImqm1vXDqxF3OxNjfhzfvb8Nof0SzYdpZHu3ribGdh7PDqldzCEl5YEkWJTmFgkCsjulaucZiNxpRxd/kxZ30Mn28+wwPBzTFRN87BhaRGfQsqlQorc1OjvGpzRKudO3cyZswYhg0bRlBQEK6ursTHx9fa+Spib2+Pi4sL+/btM6zTarUcPHiwSscJDAxk586d5dbt3Lmz3LCylpaWDB48mM8++4ytW7cSGRlJdHQ0AKampvTt25cPP/yQI0eOEB8fz+bNm2/jysSdrrQ2/UCH5ozo6klHLwfyirR8tOGUkSOrf95adYy4S7m421swe1j7Kn3uPRnqjb2lGbEXc1ndiGvVUqO+Q/n7+/PHH38wePBgVCoVb7755k1rxrVl0qRJzJ49m5YtW9K6dWs+//xzrly5UqX/rK+88gqPPPIIHTt2pG/fvvz111/88ccfhlbsixYtQqvVEhISgpWVFT///DOWlpZ4e3uzevVqYmNjufvuu2nSpAlr165Fp9MREHDjhixC3MyV3CL+jtZPMPRYNy9UKhVvDGrDg1/t4rcDSYzu4UMbdzsjR1k/rDp8gd8PnEOtgnmPdsTeyuzWO13D1sKMZ3r68lHEKT7bdJr727s3ylq11KjvUB9//DFNmjShR48eDB48mPDwcDp16lTncUybNo2RI0fy5JNPEhoaio2NDeHh4VhYVP724NChQ/n000+ZO3cubdu25euvv2bhwoX06tUL0M8H/e233xIWFkb79u3ZuHEjf/31F05OTjg4OPDHH39w7733EhgYyIIFC/j1119p27ZtLV2xaOyWX21E1q65HUEe+sF/Ons3YVB7NxQF3lt7os4f/9RHSel5vP6H/q7WxHv96ebrWK3jjA7zwc7ClLMXc1kTnXzrHRoglXKH/cWcO3cOT09PkpKS8PDwKLetoKCAuLg4fH19q5QoRM3R6XQEBgbyyCOP8M477xg7nBohf1d3DkVR6PvxNs5ezOXdYe3KTceYlJ5Hn4+2UaTVsXBMV3q3vr6P8J2iRKvjka8jOZiYQRfvJix5tvtttdr+dONpPtl4Cn9nG9ZPudswkll9drNc9G9SoxZGlZCQwLfffsupU6eIjo7mueeeIy4ujscee8zYoQlRZfvir3D2Yi5W5iY8EOxebpunoxVPhfkA8O7aE3fEQB038tmm0xxMzMDWwpR5j3a47a5VY8J8sLUw5XRaDn8fTamhKOsPSdTCqNRqNYsWLaJr166EhYURHR3Nxo0bCQwMNHZoQlTZr6WNyILdsbW4/nnr871b4mhtzpm0HH7dl1TX4dULe2IvM3/LGQDeGxaERxOr2z6mvaWZYeKOzzadbnQjwUmiFkbl6enJzp07yczMJCsri127dsnIYKJBysgrMjwjHXmDuZLtLc2Y0tcfgE8iTpFVUFxn8dUHGXlFTFkahU6Bhzt7MPhfdx1ux9gwX2w1psSkZrP+WOOqVUuiFkKIGvDHwfMUleho42ZHe48bzyA3spsXLZpZk55bxJdbztZhhMalKAqvLY8mObMAv6bWzHygZhts2luZGR4tfNrIatWSqIUQ4jYpimK47T0yxOum3QvNTNT8d6D+0c4PO+JISr8zRsJbsi+JdcdSMDNR8dnIjlhrar538NievthoTDmZks2G46k1fnxjkUQthBC36UDCFU6n5WBpZsKQDre+nXtva2fCWjpRpNXx4fqYW5Zv6M6kZTPrr2MAvBremnbNa2fOegcrc8b08AH0z6obS6cmSdRCCHGbSkciGxzshl0Fjcj+TaVS8frANqhU8NfhCxxMvFLbIRrN6iMXeGhBJAXFOu7yb8rTPX1r9XxP9/TF2tyE48lZRDSSWrUkaiGEuA2ZecWsOXLzRmQVaeNux8Od9f1n/2/18UZT+yuVmVfMC0sOMXHxITLyimnX3I6PH+lQ632cm1ibM/pqrfrTRlKrlkQthBC3YcWhcxSW6GjtaksHT4cq7ftSvwAszUw4mJjRqEbV2nH6EuHztvNn1AVM1Com39uSFc+H0cxWUyfnf+YuP6zMTTh2IYtNJ248u19DIYlaGPTq1YspU6YYln18fJg3b95N91GpVKxcufK2z11Tx7mZmTNn0qFDh1o9h7iz6BuR6ftDj7pFI7KKuNhZMP4e/TSwH6w7SUGxtsZjrEv5RVpmrjrG49/vISWrAN+m1vw+PpSp/QIwq8P5oh2tzXky1AdoHLVqSdSNwODBg+nfv3+F2/755x9UKhVHjhyp8nH37dvHs88+e7vhlXOjZJmcnMyAAQNq9FxC1LaDiRnEpGZjYaZmSMfm1TrGuLt9cbHTkJSez4+74ms2wDp0OCmDQZ//w6Kr1/BEd2/WTO5JR68mRoln3F2+WJqZEH0+ky0xDbtWbdREPXv2bLp27YqtrS3Ozs4MHTqUmJhbt4BctmwZrVu3xsLCgqCgINauXVsH0dZfTz/9NBEREZw7d+66bQsXLqRLly60b9++ysdt1qwZVla3P2pQZbi6uqLR1M1tMSFqSmmXrMHt3SvViKwiVuamvBLeGoD5m89wOaewxuKrC8VaHZ9EnGL4V7uIvZiLi52GH8d2452h7bAyN94EjU42Gp4I1Y+1/ummMw26Vm3URL1t2zYmTJjA7t27iYiIoLi4mH79+pGbm3vDfXbt2sXIkSN5+umnOXToEEOHDmXo0KEcPXq0DiOvX+6//36aNWvGokWLyq3Pyclh2bJlPP3001y+fJmRI0fSvHlzrKysCAoK4tdff73pcf996/v06dPcfffdWFhY0KZNGyIiIq7bZ9q0abRq1QorKyv8/Px48803KS7Wj760aNEiZs2axeHDh1GpVKhUKkPM/771HR0dzb333oulpSVOTk48++yz5OTkGLaPGTOGoUOHMnfuXNzc3HBycmLChAmGc1WGTqfj7bffxsPDA41GQ4cOHVi3bp1he1FRERMnTsTNzQ0LCwu8vb2ZPXs2oL/lOXPmTLy8vNBoNLi7uzN58uRKn1s0fJn5xYY5kEeGVL4RWUWGd2xOW3c7sgtL+HTT6ZoIr06cScvhwa928emm02h1CoOD3Vk/5W7uadXM2KEBMO4uPyzM1BxOymDbqYvGDqfajDof9bUfiqD/IHd2dubAgQM3HEby008/pX///rzyyisAvPPOO0RERDB//nwWLFhQe8EW3fjLww2ZaMDk6q9YWwLaQlCpwczy1sc1t670aUxNTXnyySdZtGgRr7/+uuE52bJly9BqtYwcOZKcnBw6d+7MtGnTsLOzY82aNTzxxBO0aNGCbt263fIcOp2O4cOH4+Liwp49e8jMzCz3PLuUra0tixYtwt3dnejoaMaNG4etrS2vvvoqI0aM4OjRo6xbt84wV7S9/fX9KXNzcwkPDyc0NJR9+/aRlpbGM888w8SJE8t9GdmyZQtubm5s2bKFM2fOMGLECDp06MC4ceMq9Xv79NNP+eijj/j666/p2LEjP/zwAw888ADHjh3D39+fzz77jFWrVvHbb7/h5eVFUlISSUn655HLly/nk08+YcmSJbRt25aUlBQOHz5cqfOKxuHPqPMUFOsbkXWsYiOyf1OrVbw+KJDHvt3DL3sSeTLUm5bOtjUTaC3Q6RR+jIzn/b9PUliiw87ClP8bFnTdRCTG1sxWw+Mh3ny3I45PN53mnlbNqtyOoD4waqL+t8zMTAAcHW88L2lkZCRTp04tty48PPyGDZEKCwspLCy7lZSdnV294N6rxh/gw4ug7TD9zyf/gmVjwLsnPLWmrMy8IMi7fP2+MzOrdKqxY8cyZ84ctm3bZpiHeeHChTz44IPY29tjb2/Pyy+/bCg/adIk1q9fz2+//VapRL1x40ZOnjzJ+vXrcXfX/y7ee++9654rv/HGG4affXx8ePnll1myZAmvvvoqlpaW2NjYYGpqiqur6w3PtXjxYgoKCvjf//6HtbX+C8v8+fMZPHgwH3zwAS4uLgA0adKE+fPnY2JiQuvWrRk0aBCbNm2qdKKeO3cu06ZN49FHHwXggw8+YMuWLcybN48vvviCxMRE/P396dmzJyqVCm/vsikLExMTcXV1pW/fvpiZmeHl5VWp36NoHBRFYfGeqyORdat6I7KK9GjRlPvauBBxPJXZa0/y/Ziut33M2nAhI59Xfz/CjjOXALjLvylzHgrG1b5+TuH67D1+/LQ7gUOJGfxz+hJ315PaflXUm8ZkOp2OKVOmEBYWRrt27W5YLiUlxfBBXcrFxYWUlIoHYZ89e7YhUdnb29OmTZsajbu+aN26NT169OCHH34A4MyZM/zzzz88/fTTAGi1Wt555x2CgoJwdHTExsaG9evXk5iYWKnjnzhxAk9PT0OSBggNDb2u3NKlSwkLC8PV1RUbGxveeOONSp/j2nMFBwcbkjRAWFgYOp2uXBuGtm3bYmJiYlh2c3MjLa1yjUaysrK4cOECYWFh5daHhYVx4sQJQH97PSoqioCAACZPnsyGDRsM5R5++GHy8/Px8/Nj3LhxrFixgpKSkipdp2i4opIyOJmSjcZUzdBqNiKryPQBrTFVq9h0Mo2dVxNhfaEoCisPnSd83nZ2nLmEhZmad4a05X9ju9XbJA3gbGthmBe8obYArzc16gkTJnD06FF27NhRo8edPn16uRr4+fPnq5es/3uh6vuYXNM4qvVg/TFU//puNCW66se9gaeffppJkybxxRdfsHDhQlq0aME999wDwJw5c/j000+ZN28eQUFBWFtbM2XKFIqKimrs/JGRkYwaNYpZs2YRHh6Ovb09S5Ys4aOPPqqxc1zLzKx84x2VSoVOV3Nz/Hbq1Im4uDj+/vtvNm7cyCOPPELfvn35/fff8fT0JCYmho0bNxIREcHzzz9vuKPx77hE41Nam76/vTv2ljX3fvs1s+Hx7t4s2hXP/605wepJPTGp5QFCKuNKbhFvrDxq6Osd7OnAJ48E49fMxsiRVc74e/z4ZU8CBxKusP30pXrzDL2y6kWNeuLEiaxevZotW7bg4eFx07Kurq6kppYfFi41NfWGt1I1Gg12dnaGl61tNZ/7mFtX/WVyzfcgE1P9umufT9/suNXwyCOPoFarWbx4Mf/73/8YO3as4Zbczp07GTJkCI8//jjBwcH4+flx6tSpSh87MDCQpKQkkpPLBmXYvXt3uTK7du3C29ub119/nS5duuDv709CQkL5yzU3R6u9eV/RwMBADh8+XK5R4c6dO1Gr1QQEBFQ65puxs7PD3d2dnTt3llu/c+fOcl/k7OzsGDFiBN9++y1Lly5l+fLlpKenA2BpacngwYP57LPP2Lp1K5GRkURH19wXL1E/ZRUU89fVRmSPhXjW+PFf6OOPnYUpJ5KzWH7g+p4cde1yTiEDP/uHNdHJmKpVTL2vFcvHhzaYJA3gbGfB4931teoZfx4lv6hh9Vc3aqJWFIWJEyeyYsUKNm/ejK/vrceADQ0NZdOmTeXWRUREVHgb9k5jY2PDiBEjmD59OsnJyYwZM8awzd/fn4iICHbt2sWJEyf4z3/+c90Xnpvp27cvrVq1YvTo0Rw+fJh//vmH119/vVwZf39/EhMTWbJkCWfPnuWzzz5jxYoV5cr4+PgQFxdHVFQUly5dKtd+oNSoUaOwsLBg9OjRHD16lC1btjBp0iSeeOKJ6x573I5XXnmFDz74gKVLlxITE8Nrr71GVFQUL7zwAgAff/wxv/76KydPnuTUqVMsW7YMV1dXHBwcWLRoEd9//z1Hjx4lNjaWn3/+GUtLy3LPsUXj9OchfSOyVi42dKqFPsJNrM2Z3Ec/Z/XcDTHkFhr3kcq6YykkZxbQ3MGSP57vweQ+/pjW4eAlNWVKX39c7SxIuJzHvI2Vr6TUB0b9bU+YMIGff/6ZxYsXY2trS0pKCikpKeTn5xvKPPnkk0yfPt2w/MILL7Bu3To++ugjTp48ycyZM9m/fz8TJ040xiXUO08//TRXrlwhPDy83PPkN954g06dOhEeHk6vXr1wdXVl6NChlT6uWq1mxYoV5Ofn061bN5555hnefffdcmUeeOABXnzxRSZOnEiHDh3YtWsXb775ZrkyDz74IP3796d37940a9aswi5iVlZWrF+/nvT0dLp27cpDDz1Enz59mD9/ftV+GbcwefJkpk6dyksvvURQUBDr1q1j1apV+PvrPyRtbW358MMP6dKlC127diU+Pp61a9eiVqtxcHDg22+/JSwsjPbt27Nx40b++usvnJycajRGUb8oisIvNdyIrCJPhHrj5WhFWnYhX2+PrZVzVNa+OP0dpAc7e9Dew8GosdwOWwsz/m+ovv3Tt//EcuRchnEDqgKVYsQn6zf6I1+4cKGhNtirVy98fHzKdctZtmwZb7zxBvHx8fj7+/Phhx8ycODASp3z3LlzeHp6kpSUdN1t9oKCAuLi4vD19cXCov42jhANi/xdNR5RSRkM/WInGlM1e/7bBwcr81o719/RyTz3y0EszNRsfbm30Rpshb2/mfMZ+fz8dAg9/ZsaJYaaNOnXQ/x1+AKtXW35a1LPOh3a9Fo3y0X/ZtTGZJX5jrB169br1j388MM8/PDDtRCREELc2K9Xa9ODgtxqNUkD9G/nSlefJuyLv8Lnm0/z7rCgWj1fRc5dyeN8Rj4mahUdvRzq/Py14a3Bbfjn9EVOpmTzzfZYJvRuaeyQbqnhPWgQQggjyC4oZtXhmhmJrDJUKhUT79U/htl8Ms0o3Yr2xetve7drbo+1pt50ErotTW00vDVY32D0002nOXsx5xZ7GJ8kaiGEqIQ/oy6QX6ylpbMNXbzrZqKJbj6OmJmoSM4sIOFyXp2c81p7465cjcM4E2vUlqEdmtMroBlFJTpeW34Ena5+962WRC2EELdQGyORVYaluQkdPfVJMjK2ghEMa9neOP05u/rceLTIhkilUvF/Q9thZW7Cvvgr/LIn4dY7GZEkaiGEuIXo85kcT87C3FTN8BociawyurfQ9ySIPFu3ifpSTiFnL+rHMmhsiRrAo4kVr4brx2V4/++TXMjIv8UexiOJugI1ObqVEPL31PCVTmc5sJ0rTaxrtxHZv4X6XU3UsZfr9Dn1/qvPp1u52NT5NdeVJ0J96OzdhNwiLW+sPFpvhxdtHK0Daoi5uTlqtZoLFy7QrFkzzM3NG+RMK6J+UBSFoqIiLl68iFqtxty8cX7YNXY5hSX8GXW1EVm32m9E9m8dvRwwN1VzMVtfw23pXDcjghmeT/s2vtp0KRO1ig8eDGLgpzvYfDKNVYcvMKRD3d4xqQxJ1NdQq9X4+vqSnJzMhQvVGNtbiApYWVnh5eWFWi03sBqiVVEXyCvS4tfM2ihJy8LMhM5eTYiMvUxk7OU6S9SlLb4b423va7V0tmXivS35OOIUs/46zl3+zXCsZ3cQJFH/i7m5OV5eXpSUlNxyTGohbsXExARTU1O5M9OAld72fqwOG5H9W2gLJyJjL7P77GWe6F77w9RmFxRz7IJ+qt3GXKMuNf6eFqw5kkxMajZv/3WMeY92NHZI5UiiroBKpcLMzExmQRLiDhd9LpPo85mYm6gZ3unmo0fVptAWThABu68+p67tLwwHEzPQKeDpaImbveWtd2jgzE3VfPBQe4Z/uZOVURd4oIM797auuXkFbpfcixNCiBv4dZ++Nt2/natRb4cGezhgaWbC5dwiTqXW/gAdpeN7N/bb3tfq4OnA2DD9xFBvrDhKjpEnQ7mWJGohhKhATmEJfx46DxinEdm1zE3VdLk66Ejk2Uu1fr69VxN1tzsoUQNM7dcKT0dLLmQW8OG6k8YOx0BufQsh7lglWh0XMgqIu5xLwuVc4i/lEX85l/jLuSSl51GsVfBtak13P+MnrO5+Tvxz+hKRsZcZE3brKYGrq7BES9TVmaXuhOfT17IyN+X94e0Z9d0eftqdwOBg93pxV0EStRCiUSvW6jh/JV+fjC/lEn85T5+UL+eRlJ5HyU2Gj7QwUzOlr3+9aAwYenXgkz1x6eh0Cmp17cR05FwmRSU6mtqY49vUulbOUZ+FtWzKI108+G3/OaYtP8LayXdhYWZi1JgkUQshGh2dTmHOhhjWRidz7ko+2pskY3NTNd6OVvg0tcbHyQpvJ2t8nKzxaWqFm70lJrWUEKsqqLk91uYmZOQVcyIli7bu9rVynr3XPJ+uD19QjOH1gW3YEnOR2Iu5zN98hpevjmBmLJKohRCNzo+R8Xy19axh2cJMjbejPvn6OFlfTcb65OxqZ1FrtdOaZGaipquvI1tjLrI7Nr3WE/Wddtv7WvZWZrz9QFue++UgC7adZWCQG22cLeDIEuj0ZJ3HI4laCNGonEzJYvbf+oZAL/drxUOdPXG21TSIZHwroX5ObI25SOTZyzzds+afU2t1CgcS9COS1Ydns0ajLWaAfQKve5/g3YRApi0/wornQjGN/ALaPwqmddsDQFp9CyEajYJiLVOWRFFUoqN3QDMm9G6Jq33DqDFXRtlz6ss3vZ1fXSeSs8gpLMFWY0qgm12NH7/e0mmhILNsOfkI/BDO0xmf4WChJvp8Jj/siofuz0NhVp2HJ4laCNFozFkfw8mUbJyszfnwoeBG94y1jZsdthpTsgtKOH6h5hNG6W3vzj5N6s2z+VqhKHAxBvZ8A0tGwYd+sOGNsu1uwdDEF3WL3rx1n37s748jTpHg8xBYN63zcOXWtxCiUdh+6iLf74gD4MOH2tPMVmPkiGqeqYmabr6ObDqZRmTsJYI8avY59d7GOtCJokB6LCTshLjt+ldOavky5w+V/WxiCpMPgUrFUEVh2fE8dp29zGvLo1k8LqTOvwBKohZCNHjpuUW8vOwwAI9396JPYP0Z/rGmhbZw0ifqs5d59u4WNXZcRVEME3GENPSGZNoSSD0KiZFXX7uvT8ymFuDVHXzvBt97wK1D+e1Xk7FKpWL28CDC520nMvYyS/cl8WgdD4AjiVoI0aApisL0P46Qll1Ii2bWvD6wjbFDqlXdr85PvS/+CiVaHaYmNfMEM/ZSLpdzizA3Vdd4Tb3WFeeDrgQ0tvrlI0vhz+fLlzExh+adwecu8LsHPLqCaeXuung7WfPSfQG8u/YE7649Qe/WzrjYWdTwRdyYJGohRIP22/4k1h9LxcxExaePdsTS3LiDU9S2Nm522FuakZlfTPT5TDp6NamR45be9u7g6YDGtAH9Dte/Dnu+hn7vQPfn9Ou8uoPGHrxC9D979QD3jmBW/eT6VJgPfx25gKejFaZ1/PxeErUQosGKu5TLzFXHAXipXwDtmjewmmA1qNUqQnwd2XA8lcjYyzWWqEsn4qiXt71z0iB2m/4Zc+JueOIPsHPXb7NyAl0xpB4rK+/oB9PiQF1zXzhMTdQsHtcdG03dp01J1EKIBqlYq2PKkkPkF2vp7ufIuLv8jB1SnQlt4aRP1Gcv83yvljVyzL3x9aghWWEOJOyC2K36V9qx8tsTdkHQQ/qfOz4O7YaDwzXzdKtUoKr5uwLGSNJg5ES9fft25syZw4EDB0hOTmbFihUMHTr0huW3bt1K7969r1ufnJyMq6trLUYqhKhvPtt0msPnMrGzMOXjRzo07u5E/1Lan3p//BWKSnSYm97ec+oLGfmcu5KPWgWdvGumhl4l2mI4f+BqYt4G5/bqnzlfy7W9vuGXVyj4hJWtt3Gu01CNwaiJOjc3l+DgYMaOHcvw4cMrvV9MTAx2dmWd8Z2dG/8bJYQosy8+nS+2nAHgveFBuDtYGjmiutXK2RZHa3PSc4s4ci6DLrdZCy5t7d2uuX3d1xr/nADHVkLRv+bZbuIDfr30L5+7wdqpbuOqR4yaqAcMGMCAAQOqvJ+zszMODg41H5AQot7LKihmypIodAoM79Sc+9u7GzukOqdWq+ju58ja6BQiz16+7US9py76TxcXwLE/4Nw+GPSxofsTJYX6JG3pqG+N7ddL313Ksfam8mxoGuTIZB06dMDNzY377ruPnTt33rRsYWEhWVlZhld2dnYdRSmEqA1v/XmM8xn5eDpaMuuBtsYOx2hCr3bTioy9fNvH2lcbiVpRIP9K2bJKBWtegv0/QNqJsvU9X4T/bIdXzsLDi6DzGEnS/9KgGpO5ubmxYMECunTpQmFhId999x29evViz549dOrUqcJ9Zs+ezaxZs+o4UiFEbfgz6jwrDp1HrYJ5Izpga2Fm7JCMpvQ59YGEKxSWaKvdpSo9t4jTafrbzl19bvP5tKJAchQcXwUn/tL3XX5+l36bqQY6P6XvIqWxKdvH5c79slVZDSpRBwQEEBBQNi9ojx49OHv2LJ988gk//fRThftMnz6dqVOnGpbPnz9PmzaNe0AEIRqjc1fyeGPlUQAm3utPZ+960DrZiFo0s6GZrYaL2YUcSswwDIRSVaXPp1s62+BkU41hV3U6feOv0uScmVi2zcQcslPA9mpj3/7vVSvGO12DStQV6datGzt27Ljhdo1Gg0ZT9seXlVX3M58IIW6PVqcw9bfDZBeU0MHTgcn31kyXpIZMpVLR3c+Jvw5fIPLs5eon6urMP60thvgd+sR8cnX54TnNrKBlX2gzBPz7gcUdNAtXLWnwiToqKgo3NzdjhyGEqEVfbz/L3rh0rMxNmDeiQ40Nm9nQhZYm6tjLvFjNY5TWqLvd6vm0TgunI/TJOWZN+efPGnsI6A+Bg6FFHzC3qmY0oiJGTdQ5OTmcOXPGsBwXF0dUVBSOjo54eXkxffp0zp8/z//+9z8A5s2bh6+vL23btqWgoIDvvvuOzZs3s2HDBmNdghCilh05l8HHG04BMPOBtvg0tTZyRPVH6XPqqMQMCoq1WJhV7Tl1bmEJR69Ol9m1ohp1cT6YlXZ9U8HqKZCdrF+0coLWgyBwiL5/s6l5Na9C3IpRE/X+/fvLDWBS+ix59OjRLFq0iOTkZBITy553FBUV8dJLL3H+/HmsrKxo3749GzdurHAQFCFEw5dXVMKUJVGU6BQGtHPl4c4exg6pXvFxssLVzoKUrAIOJlyhR8uqzZV8MPEKWp1CcwdLml/bFz3rAvw2Gq7EwUsx+qE41Wp9Y7C8y/qas1eofjpIUeuM+lvu1asXiqLccPuiRYvKLb/66qu8+uqrtRyVEKK++L81J4i9lIuLnYb3hgXV+TzA9Z3+ObUjK6P0t7+rmqj3xaVjRQFjm8ZBdHrZsJzWznDpFBRkQEo0uHfQr+81rUbjF5VTrUSdlJSESqXCw0P/7Xbv3r0sXryYNm3a8Oyzz9ZogEKIO9OGYyks3qO/o/bRwx1oYi23VisS2sJJn6jPVqE/9eWzcHoDffb/xgTNETTnSiDbC9o9qO/vbGKq79Pc1B/s5S6GsVUrUT/22GM8++yzPPHEE6SkpHDffffRtm1bfvnlF1JSUpgxY0ZNxymEuIOkZRfw2h/RAIy7y5ee/lWrKd5JQv30v5vD5zLIKyrByryCj/WSIkjcBac2wOn1cFnfNigYQAXFdl6YBfSHkoKyZ9It5JFifVGtRH306FG6desGwG+//Ua7du3YuXMnGzZsYPz48ZKohRC3Zfbak6TnFhHoZsfL4QG33uEO5umof758PiOf/fFXuLtVM/2Golx9K+3jf+r/LbpmVEa1KVnOXfksyY9Dmm78PuVJ/TNoUS9VK1EXFxcb+iZv3LiRBx54AIDWrVuTnJxcc9EJIe44Z9KyWRl1HoAPHgyq9ohbd4rS/tTLD54jMvayPlFnXYDPO0NxXllB62b6fs3+/aBFb37efZHv4mMI93VBJUm6XqtWom7bti0LFixg0KBBRERE8M477wBw4cIFnJzu3BlOhBC375ONp1EU6NfGhfYeDsYOp/4rzOYRTSQOJtFEnn1Mv87OHew9QVuoH3gkcAi4dyxXa94XFwNAN1/5zK7vqpWoP/jgA4YNG8acOXMYPXo0wcHBAKxatcpwS1wIIarqZEoWa47o78q9eF8rI0dTjylK2exTF08Rcmga7Uw1dD1/HzmFJfqpKsesAeumZeWuodUp7I/XD1hyy4FOhNFVK1H36tWLS5cukZWVRZMmZYO4P/vss1hZyYg0Qojq+SRCP7DJoCA3At1k6Mly8tIhZq1+7mYHL7j/Y/365p3Arxe/JzbDtLCIfXHp9G7tDDbNbniokylZZF9N6IFutnUTv6i2aiXq/Px8FEUxJOmEhARWrFhBYGAg4eHhNRqgEOLOcPR8JuuPpaJSwQt9/Y0dTv2QexlO/qVvEBa3HXQl+vWWTWDAh/puVCoVPPknx38/Qtb+JCJjL+sT9U2Uju/dybuJDMfaAFQrUQ8ZMoThw4czfvx4MjIyCAkJwczMjEuXLvHxxx/z3HPP1XScQohGrrQ2/UCwO61c7uBaXl46nFwDx1ZA7FZQtGXbXNpBm6HQ5oHrRgULbeHE0v1JlepPvdcwvvdtTmsp6kS1EvXBgwf55JNPAPj9999xcXHh0KFDLF++nBkzZkiiFkJUyaHEK2w6mYZaBS/0uQNr0/kZV29rr4Czm8tqzgCuQdB2mL5BWNMbzxpWOu73sQuZZOYXY29Z8VzdiqKwN+7q82lpSNYgVCtR5+XlYWur/8a7YcMGhg8fjlqtpnv37iQkJNRogEKIxu+TjacBGNbRA79mNkaOpo7lpcNHAaAtKlvn0g7aDoW2w8GpRaUO42JngV9Ta2Iv5bI3Lp372rhUWC7+ch6XcgoxN1HT3sO+Bi5A1LZqPZxo2bIlK1euJCkpifXr19OvXz8A0tLSsLOTBiBCiMrbH5/O9lMXMVGrGn9tuigXjiyDbXPK1lk5glswOLeB3q/DhH3w3E64+5VKJ+lS3a/Wqm92+3tvnH5bsKd9lWfbEsZRrRr1jBkzeOyxx3jxxRe59957CQ0NBfS1644dO9ZogEKIxu2jq1NYPtzZAy+nRthr5NquVJnn4I9nQG0G3Z7RNwoDePwPsLj9Sk6onxOL9yQSGXuzRF1621u6ZTUU1UrUDz30ED179iQ5OdnQhxqgT58+DBs2rMaCE0I0brvOXiIy9jJmJiom3nvj568NTu5l/ZjaJ9eAxg6GfaVf3ywAWt8PzoGg05WVr4EkDdDdT1+jPpGcxZXcogonMtkbr0/iXaX/dINR7WkuXV1dcXV15dy5cwB4eHjIYCdCiEpTFMXQ0ntEV088mjTw2vSlM/oGYTFrIWkPKFcTsaklDJoL5tb65Ud/qbUQmtlq8He24XRaDnviLtO/nVu57SmZBSSl56NWQWdvafHdUFTrGbVOp+Ptt9/G3t4eb29vvL29cXBw4J133kF37bdEIYS4gR1nLrEv/grmpmom9m6Az6Z1WkjcDREzYH5XmN8ZIt6ExEh9knZtD/e8Bk+vB7O6+xJSWqveHZt+3bbSbllt3O2wtai4Vbiof6pVo3799df5/vvvef/99wkLCwNgx44dzJw5k4KCAt59990aDVII0bgoimJ4Nj0qxAtXewsjR1RJJYVwZiOcXAun1kHepbJtajPwvQsCBkKr/uDgaZQQQ1s48dPuhAoblJU2JJPb3g1LtRL1jz/+yHfffWeYNQugffv2NG/enOeff14StRDiprbGXCQqKQMLMzXP9apay+Y6V1IEplef9RbnwdInygYhsbAH/3AIGAAt++iXjay0Rh2Tms3lnEKcbDSGbfviZHzvhqhaiTo9PZ3WrVtft75169akp19/u0UIIUopisLHV59NPxnqg7NtPa1NX4iCNS+BSg3PROjXWTaB4Ef1DcRaDwSvUDCpX7eQHa3Nae1qy8mUbHbHpjOovf45dUZeETGp+jmpu0qL7walWs+og4ODmT9//nXr58+fT/v27W87KCFE47XheCrR5zOxMjfhP3f7GTucMnnpcOl02bKNM5w/AOf367eVGvolDHgffO+ud0m6VGmtOjK27Nb8vquzZfk1s6bpNbVsUf9Vq0b94YcfMmjQIDZu3GjoQx0ZGUlSUhJr166t0QCFEI2HTlfW0ntMD59yt2WNojBb/7z56O/6oTt974YnVui32bnDIz+CRzf9oCQNSGgLJxbtii/3nHrf1YZkIVKbbnCqVaO+5557OHXqFMOGDSMjI4OMjAyGDx/OsWPH+Omnn2o6RiFEI/H30RROpmRjozFl3F1Gqk0XF8CJv+C30TDHH1Y8C6c36MfXzr8C2uKysm2GgJ3bjY9VT3X3dUKlgrMXc0nLKgBg79UZs6QhWcNT7X7U7u7u1zUaO3z4MN9//z3ffPPNbQcmhGhctDqFeRv1temxPX0rHIyj9k5eAnHb4OhyfZIuzCrb5tgCgh6Cdg9Bs1Z1F1Mtsrcyo42bHccuZBEZe5n72rhw9HwmIIm6ITLqRKTbt29n8ODBuLu7o1KpWLly5S332bp1K506dUKj0dCyZUsWLVpU63EKIW7f6iMXOJ2Wg52FKU/39K39E+p0+n7Oa16Gj1vDz8Mh6hd9krZ1h9CJ8OxWmHQAev+30STpUqGG/tSXOZSYQYlOwd3eAo8mlkaOTFRVtWvUNSE3N5fg4GDGjh3L8OHDb1k+Li6OQYMGMX78eH755Rc2bdrEM888g5ubG+Hh4XUQsRCiOkq0OuZdnSFr3F1+N5yCsUYV58H/hkJJvn7Z0lE/I1W7h/SttdVGrafUutAWTny3I47Is5dpdrVlfVdfR1Sl446LBsOoiXrAgAEMGDCg0uUXLFiAr68vH330EQCBgYHs2LGDTz75RBK1ELehoFjLpZxCLuUUcSm7EJ2icHerZjU2u9LKqAvEXcrFwcqMp2qjNl2Up7+tfW4fPPCZfp3GBto/oh+kJOgh8OtVb1tp14auvo6oVfppLVcfuaBfJ7e9G6QqJepb1XozMjJuJ5ZbioyMpG/fvuXWhYeHM2XKlFo9rxANUV5RCZeyi7iYU3g1CRdyKbuo7OdrEnN2Ycl1+7vaWTChdwse6eqJxrT6CbtYq+OzTfra9H/uboGNphbqB8X5sPpF0BVDyH/Apa1+fWnSvgPZWZgR1Nyew+cyib2YC0iL74aqSv9j7O1vPuqOvb09Tz755G0FdDMpKSm4uJSfDN3FxYWsrCzy8/OxtLz+2UthYSGFhYWG5ezs7FqLT4jbpdMpnM/IJ6ewhLyiEnILteQVafU/F2nJKyz/b37pekNZ/b9X8orIK9JW6dzmJmqa2pjT1FZDWlYhKVkFvPnnMRZsi2XivS15qLMHZiZVv128/MA5EtPzaGpjzuge3lXe/zqXz0LUYkiPhYcX6tdZO+kTtJUT2LjcfP87SPcWThw+p29E1sTKjJbONkaOSFRHlRL1woULayuOWjN79mxmzZpl7DCEuCVFUZiw+CB/H02psWNqTNU0tdHQ1FZDMxtzmtlq9MuGlz4xN7XRYGdhanh+WVii5bd9SczfcobzGflM/yOaL7eeYdK9/gzv2BzTSibswhItn28+A8D4e1pgZV7N2nRBJhxboU/QSXvK1t/7BjhdHYI0XIYu/rdQPye+3hYLQBcfeT7dUBn1GXVVubq6kpqaWm5damoqdnZ2FdamAaZPn87UqVMNy+fPn6dNmza1GqcQ1bHuaAp/H01BpQInaw3WGhOszE2xMjfBytwEa3NTrDRX/zXXb7uujMYUS3MTmliZ09TGHBuNabU+nDWmJjwR6sPDXTxZvCeRL7eeJSk9n1d/P8KXW84wuY8/Qzo0x0R982P/tv8c5zPycbbV8Hj3KtamdVqI3QJRv8LJ1VCi7w+MSg0t+0LwSLBrXuVru5N09XHEVK2iRKfIbe8GrEEl6tDQ0OtGPouIiDCMjlYRjUaDRlM2+lFWVtYNywphLNkFxcz86xgAk3q3ZGq/ACNHpGdhZsLYnr6M7ObFz7sTWLDtLPGX85j622HmbznDC338ub+9e4UJu6BYyxdXa9MTeresXMM0nQ7O7YXjf8KxlZB9oWxbs0Do8Ji+gZitaw1dYeNmrTGlb6ALW0+l0SdQHgk0VEZN1Dk5OZw5c8awHBcXR1RUFI6Ojnh5eTF9+nTOnz/P//73PwDGjx/P/PnzefXVVxk7diybN2/mt99+Y82aNca6BCFqxEcbTpGaVYiPkxXP925p7HCuY2luwri7/XgsxIv/RSbw9fazxF7M5YUlUXyx5QxT+raif1tX1Nck7MV7EknJKsDN3oIRXSsx5eOW9+DAj5Bzza1/yyYQ9LA+Qbt1ALl1W2WfjuxAfpEWB6s6HGBG1CijJur9+/fTu3dvw3LpLerRo0ezaNEikpOTSUxMNGz39fVlzZo1vPjii3z66ad4eHjw3XffSdcs0aAdTsrgx8h4AP5vaFCNdYmqDdYaU57r1YLHu3vx4654vtkey6nUHJ7/5SCtXW158b5W9GvjQkGxji+3ngVg4r0V1Ka1xRC/A3zvKevPnJOmT9Iae/20kW0e0N/iNpUJJG6HxtTktlrtC+NTKYqiGDuIunTu3Dk8PT1JSkrCw8PD2OGIO1yJVseQL3Zy7EIWQzu4M+/RjsYOqUoy84v5YUccP+yIM3TxatfcjgAXO5YfPIdHE0s2v9QLc9NrGp/pdPBZB8hIgLEbwCtEvz71GGSeB797JDmLRq8quahxD80jRD33Y2QCxy5kYWdhyuuDGl4jR3tLM168rxX/TOvNxN4tsTY34ej5LJYfPAfAi/d4Yn56LaybDqV1ArUaPEPAuln5Z9AubaFVP0nSQvxLg2pMJkRjciEjn483xAAwfWAgzWwbboJysDLn5fAAxvb05cfNR0jc9xfDLQ7Qc9NBKNYPtkGHx8A1SP/zgA/Awh7UcktWiFuRRC2EkcxcdYzcIi1dvJswokslGlvVZ5fPwql1OJ5ax4sJu0BdAkVXt9l76qeL1NiWlW9g8zsLYUySqIUwgojjqWw4noqpWsW7w4LKtZZuELTFkBgJp9bDqXVw+Uz57U1bXW0QNgTcO0lrbSFugyRqIepYbmEJb/15FIBxd/sR4Gp7iz3qoe/7wYWDZctqM/AJg1b9wb9f2WhhQojbJolaiDr2ScQpLmQW4OloyeR7/Y0dzs2VFELkFxC7FUYtK2vo5d0DMhKhVbj+5dcbLOyMGqoQjZUkaiHq0LELmSzcFQ/A20PaYWlezxpTFeZA+llwC9Yvm5jD3m8gO1nf77llH/36XtPhvnca/ZzOQtQHkqiFqCNancJ/VxxFq1MY1N6N3gHOxg5JP572hSiI3Qxnt+onvNDYwitn9UlYpYKeU/U/l7bYBv1cz0KIOiGJWog68sueBA4nZWCrMeWt+43YZ/pKgn6yi7ObIXYbFGSU366x1deg7a9OeBHybJ2HKIQoI4laiDqQmlXAnHX6PtOv9A/A2c6i7k5ekAlx/5Ql5/TY8ts19uB7F7S4F1r0Bke/uotNCHFLkqiFqANvrz5OdmEJwZ4OjAqp4nSPtyM/A+a0BF1x2TqVCXh20zcAa3EvuHcEE/koEKK+kv+dQtSyLTFprDmSjIlaxXvD2t1yDudqu3gKtn8Ixfnw6C/6dZYO4BwIxXllidmnp7TQFqIBkUQtRC3KL9Ly5kp9n+mnevjQ1t2+5g6eeV6fgJte7eKlUkH0Mn2f5qJcMLfWr39qbflRwYQQDYokaiFq0aebTnPuSj7u9ha8eF+r2zuYouhnmIpZCyfXQHIUBD4AI37Sb2/qD33eAu8wMLUs20+StBANmiRqIWpJTEo23/2jb7g1a0g7rDXV+O+mLYHEXXByrT5BZyRcs1EFRTn6BF46ROddU28/cCFEvSKJWohaoNMp/HdFNCU6hX5tXLivjUvldy7MhjOb9In51Pry3adMLcCvFwQM1I+lbVMP+mILIWqVJGoh0CfWBdvPcjgpg64+jnT3c6KNm121J8tYsi+JAwlXsDY3YeYDbSu3U3osrH8DzkSAtqhsvaWjPikHDNR3nyp99iyEuCNIohZ3vPwiLS8vO8ya6GQA1h9LBcDByowQX0dC/Zzo0bIp/s42qCoxC9TF7ELe//sEAFP7BeDuYFlxQZ0O8i6V1Yo19nB6PehK9H2ZAwZC60HgGSLzNgtxB5NELe5oaVkFjPvffg6fy8TMRMXoUB/OXsxhb1w6GXnFrD+WakjcTW3MCfFzokcLJ0L9nPBtal1h4n53zXGyCkpo627H6NAb9JlOiIQ/xoFdc3h6vX6dtRMM/gzcO4BzG5kaUggBSKIWd7DjF7J45sd9XMgswMHKjK8f70yInxMAxVod0ecziTx7md2xl9kXn86lnCLWHElmzRF9zdvFTkOPFk0J9XMitIUTno5W/HP6IiujLqBWwezhQZiaXJ20IjsV8q+Ac2v9chMfyDwHBVmQlw5Wjvr1HUfV8W9BCFHfqRRFUYwdRF06d+4cnp6eJCUl4eHhYexwhJFsPJ7K5CWHyCvS4tfMmh9Gd8Wn6Y2f/RaWaDmcpE/cu85e4lBiBkVaXbkyzR0sKSzRcSmnkDE9fJgZ7q3vRnVkqX74Tt974MmVZTvEbQePrmB2g1vjQohGqyq5SGrU4o6iKArf74jj3bUnUBQIa+nEl491xt7K7Kb7aUxN6ObrSDdfR17o609BsZaDCVfYdfYykbGXOZyUwfmMfEzQ8oBNDK8XroC5a6E4t+wgxfmgLQaTq+fyvbsWr1QI0VhIohb1QnZBMcVaBUdr81o7R7FWx4w/j/Hr3kQAHgvxYtYDbTEzqfqcyhZmJvRo2ZQeLZtC5jkKT50h41gEtud3YFWcDseuFmziC8GPQtDD4NSiBq9GCHGnkEQtjO5wUgZP/7iPK3nFDApy45m7fGnv4VCj58jMK+a5Xw6w6+xlVCp4Y1Abxob5VKoVd4Xyr8DmdyF2K1w+jQYw9JS2dIR2D0L7EeDRRRqFCSFuS9WrErXgiy++wMfHBwsLC0JCQti7d+8Nyy5atAiVSlXuZWFRh1MGihq1JSaNR7/ZzaWcIrQ6hVWHL/DA/J088nUkEcdT0eluvwlF/KVchn25k11nL2NtbsJ3T3bh6Z6+lU/SJYX6aSJj/i5bZ24Dh3+Fy6dBpYbmXeCul2H0angpBgbNBc+ukqSFELfN6DXqpUuXMnXqVBYsWEBISAjz5s0jPDycmJgYnJ0rHnXJzs6OmJgYw3K1a0XCqH7bn8T0P6LR6hTu8m/KC338+WVPIn8dvsDeuHT2xqXj19SasT19ebCTB5bmVe9LvDv2MuN/PkBGXjHu9hZ8P6YrgW63mDlKpwNtYVkjr9MRsHQUOLXUDzwC+ufM980CG1f9bFSWDlWOTQghKsPorb5DQkLo2rUr8+fPB0Cn0+Hp6cmkSZN47bXXriu/aNEipkyZQkZGRrXOJ62+jU9RFD7ffIaPI04BMLxTcz54sL3hWXFyZj6LdsWzeE8i2QUlADSxMuOJ7t48EepDM1tNpc7z2/4kXl8RTbFWIdjTgW+f7Iyz7Q3uvmSn6kcEO7sZYrdBl7Fw7+v6bfkZ8GWoPiEPmQ+mlTu/EELcSINp9V1UVMSBAweYPn26YZ1araZv375ERkbecL+cnBy8vb3R6XR06tSJ9957j7ZtKx6msbCwkMLCQsNydnZ2zV2AqLISrY43r2nQNaF3C17uF1DuroibvSXTBwQy6V5/ftuXxA874zh3JZ/PNp9hwfZYhnVozjN3+eLvUvGsUDqdwofrY1iw7SwAg4Lc+OiRYCzMrqmR67Rw/iCc3qB/JUeVP0jiNX9/lg4w9bjcxhZCGIVRE/WlS5fQarW4uJSfsMDFxYWTJ09WuE9AQAA//PAD7du3JzMzk7lz59KjRw+OHTtW4beS2bNnM2vWrFqJX1RNfpGWSb8eZOOJNFQqePuBtjwR6nPD8jYaU8b29OXJUG82HE/l239iOZSYwdL9SSzdn0SvgGY809OPsJZOhkSfV1TCi0ujDKOJTbq3JS/2baUfszsvXV9jPr0BzmyEvMvlT+jeEVreBy3uheady2+TJC2EMBKj3vq+cOECzZs3Z9euXYSGhhrWv/rqq2zbto09e/bc8hjFxcUEBgYycuRI3nnnneu2/7tGff78edq0aSO3vutYem4RT/+4j0OJGWhM1Xz6aEf6t3Ot8nEOJKTz7fY41h9PofQvN9DNjmd6+tLN15HnfjnA0fNZmJuo+eChIIZ19ID4HbDpHTi3F5RrBinR2OsnuWgVDi37ykxUQog602BufTdt2hQTExNSU1PLrU9NTcXVtXIf4mZmZnTs2JEzZ85UuF2j0aDRlD1TzMrKqn7AolqS0vN48oe9xF3Kxd7SjO9Hd6GLj2O1jtXZ25HOTziScDmXH3bE8dv+c5xIzuKlZYcBsKKAYVYneWbQ3bTtePWPX20GSbv1Pzu3Af/7wD8cPLuVDT4ihBD1lFG7Z5mbm9O5c2c2bdpkWKfT6di0aVO5GvbNaLVaoqOjcXNzq60wxW04ej6TYV/uIu5SLs0dLFn+XGi1k/S1vJ2smTWkHZHT7+WV8ACcrzYwe9fuDz7RfUjbC8vLCnt0gfvnwZRoeD4S7nsbfMIkSQshGgSjd8+aOnUqo0ePpkuXLnTr1o158+aRm5vLU089BcCTTz5J8+bNmT17NgBvv/023bt3p2XLlmRkZDBnzhwSEhJ45plnjHkZogLbT13kuZ8PkFukJdDNjkVPdcXFrob6vBdkwsk1OBxdzoRe03nmrt4cSLhCpyIT2HBUPytVKbUJdHmqZs4rhBB1zOiJesSIEVy8eJEZM2aQkpJChw4dWLdunaGBWWJiImp1WcX/ypUrjBs3jpSUFJo0aULnzp3ZtWsXbdq0MdYliAosP3COacuPUKJTCGvpxFePd8bO4jZrsEW5+kFHjv6h70qlLdKvd/JH49GFHi2aghIOrcOl8ZcQotEwej/quib9qGuXoih8ufUsc9brB6QZ0sGdOQ8FY25azacsxQX6pHx0OcSsg5L8sm1NA/RDdQY9JONoCyEalAbTmEw0LlqdwsxVx/hpdwIA/7nbj2n9W+u7RlVFSZF+DO2jy/XTRBZd0/e9ia8+Obcbrm8YJjVnIUQjJ4lacDmnkLhLuVhrTLE2N8VKY4KNxhSNqbrSw7MWFGt5Yckh1h9LRaWCGfe34akw36oHc3INrHweCjLK1tl5QLth0Ha4vq+zJGchxB1EEvUdbsfpSzzzv30UFOuu26ZWgbW5KdYaffLW/2xyNZmbYqMxwcrcFGtzE3acucTBxAzMTdXMG9GBgUGVbIV/JQF0JWW3rh1b6JO0tTO0HaqvPXt0A3W9mD9GCCHqnCTqO9jWmDSe/ekARSU6mtrouzflFZWQV6QFQKdAdmEJ2YUllTqenYUp3z7ZhRA/p8oFsOcb+PtVaDsMHl6oX+fcGsauB4+u+tbaQghxh5NEfYfacjKN//x0gCKtjvvauPDFY50MDb50OoW8Yi15hSXkFOoTd25hCblFJeQWaskrKiGnUL899+o2tQqeCPWmpXPF428DkB6n/9fx6i1xr+6AAkU5+hmrSmvNXt1r78KFEKKBkUR9B9p4PJXnfzlIkVZH/7aufP5YR8PMVQBqtQobjSk2GlNue1DNkiI4uRoO/qhvIBb8GAz7Sr/NrT1MOQoOnrd7FiGEaLQkUd9hNhxLYcLigxRrFQYFuTHv0Q7lknSNuXRGn5yjFkPepasrVVCYBYpS1iBMkrQQQtyUJOo7yLqjyUxcfIgSncLgYHc+eSQY05pM0sUFcOIvfYKO/6dsvY0rdHoCOj4BTbxr7nxCCHEHkER9h1hzJJnJSw6h1SkM7eDO3IdrMEmnnYCDP8HhxZB/Rb9OpdZPGdl5DPj3AxP5UxNCiOqQT887wKrDF3hxaRRancLwTs2Z81AwJlUdhKQiJYWwcCCc31+2zq45dHoSOj4O9jLymxBC3C5J1I3cykPnmfpbFDoFHu7swfsPtq9+ktbpIO04uLbTL5tqwNwK1KbQqr8+QbfsK92qhBCiBkmibsSWHzjHy78fRlHg0a6evDcsqOrDeZYqyISv74aMJJh6Amz1k6YwYA5YOYFNs5oLXAghhIEM99RI/bY/yZCkHwvxqnqSLi6AxN1lyxb2+tHCzK0hNbpsvXNrSdJCCFGLpEbdCP26N5Hpf+iT6RPdvXl7SNvKjdmtKJB8GA79DNG/QXE+vBQDVo767cMWgK2b/na3EEKIOiGJupH5eXcCb6w8CsCYHj68NbjNrZN07iWI/l2foK+tLdt5QHpsWaKWqSSFEKLOSaJuRP4XGc+MP48B8HRPX94YFFhxki4pgnN74exmOLsFLhwCrk5LbmIOre/Xt9r26yUNw4QQwsgkUTcSC3fGMeuv4wA8e7cf0we0vj5JH/oFTqyCuH+gOLf8Nrdg/YAk7R4sq0ELIYQwOknUjcB3/8Tyf2tOAPBcrxa8Gh6AKv8KxO+AwMFlw3We3Qyn1ul/tmoKLe6FFr3BrzfYVXJaSiGEEHVKEnUDlFNYwsGEK+yPT2dPnP4FChN7+/NSv1aodCXwSTt9rfn53eAcqN+xw2PgGqRP0C7tZI5nIYRoACRRNwBpWQXsi7/Cvvh09iekc/xCFiZKCUGqWDqrT/Gs2Qna2Bfj2m+H/na3iZl+qsjsZMi7XHagln30LyGEEA2GJOp6RlEUYi/lsi8unX3xV9ifkE7C5TzsyKWT+jT91TG8aRZDB3UsGorKdsxBn5jt3PXLI3/VjxwmhBCiQZNEbWTFWh3HLmSxPz6dvXHp7E+4QnpuEe5coos6hmfUMXQ1j6GV+hzq0pbZpaycwCsUPEP0LbRtXMu2SZIWQohGQRJ1HVIUhXNX8olKyiAqKYPDSRkcvZBJUXEJtuSRiQ0ALU3T2Gg65foDOLbQJ2avEP2/Ti3LGooJIYRolOpFov7iiy+YM2cOKSkpBAcH8/nnn9OtW7cbll+2bBlvvvkm8fHx+Pv788EHHzBw4MA6jLhyMvKKOHwuk6jEDA6f0yfm7NxcVCgUYg5AuHovcy2+5qR1Vw52m0cXH0faudvCZx/qW2J7dtc/b/bqDjbORr4iIYQQdc3oiXrp0qVMnTqVBQsWEBISwrx58wgPDycmJgZn5+sT065duxg5ciSzZ8/m/vvvZ/HixQwdOpSDBw/Srl07I1yBXmGJluMXsjh8TW05/3ISrdWJBKqSGKJOZJoqkRaaC3xu+yJX/IfTwdOBELUZtivn0dXiAl3vuWbkrylH9I3ChBBC3NFUiqIoty5We0JCQujatSvz588HQKfT4enpyaRJk3jttdeuKz9ixAhyc3NZvXq1YV337t3p0KEDCxYsuOX5zp07h6enJ0lJSXh43N58yauPXGBfXDrHE1PQpp6gpZJAoCqR1ip9gm6iyql4x55Toe9b+p+LcuHymavdpWQUMCGEuBNUJRcZtUZdVFTEgQMHmD59umGdWq2mb9++REZGVrhPZGQkU6dOLbcuPDyclStX1maoFVq0M57xF/7LW+oo1KbXf99RVCaomrYCl7ZXX+30/5a2zAb9bFRuwXUYtRBCiIbEqIn60qVLaLVaXFxcyq13cXHh5MmTFe6TkpJSYfmUlJQKyxcWFlJYWGhYzs7Ovs2oyzzQwR2X4mao0xW0lk6o3YJQlSZjl7aomgaAmUWNnU8IIcSdx+jPqGvb7NmzmTVrVq0c+8lQH2g9F8wsMZGGXkIIIWqBUceQbNq0KSYmJqSmppZbn5qaiqura4X7uLq6Vqn89OnTyczMNLyOHz9eM8GXauItrbGFEELUGqMmanNzczp37symTZsM63Q6HZs2bSI0NLTCfUJDQ8uVB4iIiLhheY1Gg52dneFla2tbcxcghBBC1DKj3/qeOnUqo0ePpkuXLnTr1o158+aRm5vLU089BcCTTz5J8+bNmT17NgAvvPAC99xzDx999BGDBg1iyZIl7N+/n2+++caYlyGEEELUCqMn6hEjRnDx4kVmzJhBSkoKHTp0YN26dYYGY4mJiaivmeWpR48eLF68mDfeeIP//ve/+Pv7s3LlSqP2oRZCCCFqi9H7Ude1muxHLYQQQlRHVXKRTEgshBBC1GNGv/Vd13Q6HQDJyclGjkQIIcSdqjQHleakm7njEnVp166bTfohhBBC1IXU1FS8vLxuWuaOe0ZdUlLCoUOHcHFxKddIrTqys7Np06YNx48fl25fQgjRyNXkZ75OpyM1NZWOHTtianrzOvMdl6hrUlZWFvb29mRmZmJnZ2fscIQQQtQiY33mS2MyIYQQoh6TRC2EEELUY5Kob4NGo+Gtt95Co9EYOxQhhBC1zFif+fKMWgghhKjHpEYthBBC1GOSqIUQQoh6TBK1EEIIUY9Jor4NX3zxBT4+PlhYWBASEsLevXuNHZIQQogatn37dgYPHoy7uzsqlYqVK1fW6fklUVfT0qVLmTp1Km+99RYHDx4kODiY8PBw0tLSjB2aEEKIGpSbm0twcDBffPGFUc4vrb6rKSQkhK5duzJ//nxAPxycp6cnkyZN4rXXXjNydEIIIWqDSqVixYoVDB06tM7OKTXqaigqKuLAgQP07dvXsE6tVtO3b18iIyONGJkQQojGRhJ1NVy6dAmtVouLi0u59S4uLqSkpBgpKiGEEI2RJGohhBCiHpNEXQ1NmzbFxMTEMLd1qdTUVFxdXY0UlRBCiMZIEnU1mJub07lzZzZt2mRYp9Pp2LRpE6GhoUaMTAghRGNz89mqxQ1NnTqV0aNH06VLF7p168a8efPIzc3lqaeeMnZoQgghalBOTg5nzpwxLMfFxREVFYWjoyNeXl61fn7pnnUb5s+fz5w5c0hJSaFDhw589tlnhISEGDssIYQQNWjr1q307t37uvWjR49m0aJFtX5+SdRCCCFEPSbPqIUQQoh6TBK1EEIIUY9JohZCCCHqMUnUQgghRD0miVoIIYSoxyRRCyGEEPWYJGohhBCiHpNELYQQQtRjkqiFELVGpVKxcuVKY4chRIMmiVqIRmrMmDGoVKrrXv379zd2aEKIKpBJOYRoxPr378/ChQvLrdNoNEaKRghRHVKjFqIR02g0uLq6lns1adIE0N+W/uqrrxgwYACWlpb4+fnx+++/l9s/Ojqae++9F0tLS5ycnHj22WfJyckpV+aHH36gbdu2aDQa3NzcmDhxYrntly5dYtiwYVhZWeHv78+qVasM265cucKoUaNo1qwZlpaW+Pv7X/fFQog7nSRqIe5gb775Jg8++CCHDx9m1KhRPProo5w4cQKA3NxcwsPDadKkCfv27WPZsmVs3LixXCL+6quvmDBhAs8++yzR0dGsWrWKli1bljvHrFmzeOSRRzhy5AgDBw5k1KhRpKenG85//Phx/v77b06cOMFXX31F06ZN6+4XIERDoAghGqXRo0crJiYmirW1dbnXu+++qyiKogDK+PHjy+0TEhKiPPfcc4qiKMo333yjNGnSRMnJyTFsX7NmjaJWq5WUlBRFURTF3d1def31128YA6C88cYbhuWcnBwFUP7++29FURRl8ODBylNPPVUzFyxEIyXPqIVoxHr37s1XX31Vbp2jo6Ph59DQ0HLbQkNDiYqKAuDEiRMEBwdjbW1t2B4WFoZOpyMmJgaVSsWFCxfo06fPTWNo37694Wdra2vs7OxIS0sD4LnnnuPBBx/k4MGD9OvXj6FDh9KjR49qXasQjZUkaiEaMWtr6+tuRdcUS0vLSpUzMzMrt6xSqdDpdAAMGDCAhIQE1q5dS0REBH369GHChAnMnTu3xuMVoqGSZ9RC3MF279593XJgYCAAgYGBHD58mNzcXMP2nTt3olarCQgIwNbWFh8fHzZt2nRbMTRr1ozRo0fz888/M2/ePL755pvbOp4QjY3UqIVoxAoLC0lJSSm3ztTU1NBga9myZXTp0oWePXvyyy+/sHfvXr7//nsARo0axVtvvcXo0aOZOXMmFy9eZNKkSTzxxBO4uLgAMHPmTMaPH4+zszMDBgwgOzubnTt3MmnSpErFN2PGDDp37kzbtm0pLCxk9erVhi8KQgg9SdRCNGLr1q3Dzc2t3LqAgABOnjwJ6FtkL1myhOeffx43Nzd+/fVX2rRpA4CVlRXr16/nhRdeoGvXrlhZWfHggw/y8ccfG441evRoCgoK+OSTT3j55Zdp2rQpDz30UKXjMzc3Z/r06cTHx2Npacldd93FkiVLauDKhWg8VIqiKMYOQghR91QqFStWrGDo0KHGDkUIcRPyjFoIIYSoxyRRCyGEEPWYPKMW4g4lT72EaBikRi2EEELUY5KohRBCiHpMErUQQghRj0miFkIIIeoxSdRCCCFEPSaJWgghhKjHJFELIYQQ9ZgkaiGEEKIek0QthBBC1GP/D098GJYkfaFAAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "train_reward_margins = [i-j for i,j in zip(tracking[\"train_chosen_rewards\"], tracking[\"train_rejected_rewards\"])]\n", "val_reward_margins = [i-j for i,j in zip(tracking[\"val_chosen_rewards\"], tracking[\"val_rejected_rewards\"])]\n", "\n", "plot_losses(\n", " epochs_seen=epochs_tensor,\n", " tokens_seen=tracking[\"tokens_seen\"],\n", " train_losses=train_reward_margins,\n", " val_losses=val_reward_margins,\n", " label=\"reward margins\"\n", ")" ] }, { "cell_type": "markdown", "id": "69756011-acd6-404c-a5fc-7fe252cf20c8", "metadata": { "id": "69756011-acd6-404c-a5fc-7fe252cf20c8" }, "source": [ "- **可以看到，奖励边际（Reward Margins）在优化过程中逐步提升**， \n", " **这与损失曲线（Loss Curve）相呼应，是一个积极的信号**。 \n", "- **需要注意的是**，虽然 **DPO 损失（Loss）和奖励边际（Reward Margins）** \n", " **是训练过程中重要的衡量指标，但它们不能完全反映优化效果**。 \n", "- **最关键的一步** 是 **对生成的响应进行定性分析（Qualitative Evaluation）**。 \n", "- **在此，我们直接查看模型的响应质量**， \n", " **此外，还可以像第 7 章一样使用 LLM 进行自动评分**， \n", " **进一步量化模型改进情况**。 " ] }, { "cell_type": "code", "execution_count": 53, "id": "5EfUXJGOali8", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5EfUXJGOali8", "outputId": "7ec7db47-d775-4646-f660-0d7f7e7c8503" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Convert the active sentence to passive: 'The chef cooks the meal every day.'\n", "\n", "Correct response:\n", ">> The meal is cooked by the chef every day.\n", "\n", "Reference model response:\n", ">> The meal is cooked every day by the chef.\n", "\n", "Policy model response:\n", ">> The meal is prepared by the chef.\n", "\n", "-------------------------------------\n", "\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Classify an input string as either a noun or a verb.\n", "\n", "### Input:\n", "Dance\n", "\n", "Correct response:\n", ">> 'Dance' can be classified as a verb.\n", "\n", "Reference model response:\n", ">> \"Dance\" can be classified as a verb.\n", "\n", "Policy model response:\n", ">> The input string \"Dance\" could be classified as a verb.\n", "\n", "-------------------------------------\n", "\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Rewrite the sentence using a metaphor.\n", "\n", "### Input:\n", "The book is very interesting.\n", "\n", "Correct response:\n", ">> The book is a page-turner.\n", "\n", "Reference model response:\n", ">> The book is a treat.\n", "\n", "Policy model response:\n", ">> The book is a treat.\n", "\n", "-------------------------------------\n", "\n" ] } ], "source": [ "torch.manual_seed(123)\n", "\n", "\n", "for entry in val_data[:3]:\n", "\n", " input_text = format_input(entry)\n", "\n", " token_ids = generate(\n", " model=reference_model,\n", " idx=text_to_token_ids(input_text, tokenizer).to(device),\n", " max_new_tokens=256,\n", " context_size=BASE_CONFIG[\"context_length\"],\n", " eos_id=50256\n", " )\n", " generated_text = token_ids_to_text(token_ids, tokenizer)\n", " reference_response_text = (\n", " generated_text[len(input_text):]\n", " .replace(\"### Response:\", \"\")\n", " .strip()\n", " )\n", "\n", " token_ids = generate(\n", " model=policy_model,\n", " idx=text_to_token_ids(input_text, tokenizer).to(device),\n", " max_new_tokens=256,\n", " context_size=BASE_CONFIG[\"context_length\"],\n", " eos_id=50256\n", " )\n", " generated_text = token_ids_to_text(token_ids, tokenizer)\n", " policy_response_text = (\n", " generated_text[len(input_text):]\n", " .replace(\"### Response:\", \"\")\n", " .strip()\n", " )\n", "\n", " print(input_text)\n", " print(f\"\\nCorrect response:\\n>> {entry['output']}\")\n", " print(f\"\\nReference model response:\\n>> {reference_response_text.strip()}\")\n", " print(f\"\\nPolicy model response:\\n>> {policy_response_text.strip()}\")\n", " print(\"\\n-------------------------------------\\n\")" ] }, { "cell_type": "markdown", "id": "RmcKVg0JlHVF", "metadata": { "id": "RmcKVg0JlHVF" }, "source": [ "- 从 **参考模型（Reference Model）** 和 **政策模型（Policy Model）** 的响应对比来看， \n", " **优化后的策略模型** **在风格上确实发生了微调**，相较于 **原始参考模型** 更贴近偏好。 \n", "- 例如， \n", " - **原始参考模型** 的响应： \n", " `\"Dance\" can be classified as a verb.` \n", " - **优化后的策略模型** 变为： \n", " `\"The input string 'Dance' could be classified as a verb.\"` \n", " - **优化点**：\n", " - **更礼貌**：使用 `\"could\"` 替代 `\"can\"`， \n", " **使语气更委婉，更符合人类偏好**（减少了过于武断的表述）。 \n" ] }, { "cell_type": "code", "execution_count": 54, "id": "jJSwb2hzQwdP", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "jJSwb2hzQwdP", "outputId": "6e755db4-9524-42a8-a58b-2218bf03e39a" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Rewrite the sentence using a simile.\n", "\n", "### Input:\n", "The car is very fast.\n", "\n", "Correct response:\n", ">> The car is as fast as lightning.\n", "\n", "Reference model response:\n", ">> The car is as fast as a cheetah.\n", "\n", "Policy model response:\n", ">> The car is as fast as a cheetah.\n", "\n", "-------------------------------------\n", "\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "What type of cloud is typically associated with thunderstorms?\n", "\n", "Correct response:\n", ">> The type of cloud typically associated with thunderstorms is cumulonimbus.\n", "\n", "Reference model response:\n", ">> A thunderstorm is a type of storm that typically produces thunder or lightning.\n", "\n", "Policy model response:\n", ">> The type of cloud typically associated with thunderstorms is a cumulus.\n", "\n", "-------------------------------------\n", "\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n", "\n", "### Instruction:\n", "Name the author of 'Pride and Prejudice'.\n", "\n", "Correct response:\n", ">> Jane Austen.\n", "\n", "Reference model response:\n", ">> The author of 'Pride and Prejudice' is Jane Austen.\n", "\n", "Policy model response:\n", ">> The author of 'Pride and Prejudice' is Jane Austen.\n", "\n", "-------------------------------------\n", "\n" ] } ], "source": [ "torch.manual_seed(123)\n", "\n", "\n", "for entry in test_data[:3]:\n", "\n", " input_text = format_input(entry)\n", "\n", " token_ids = generate(\n", " model=reference_model,\n", " idx=text_to_token_ids(input_text, tokenizer).to(device),\n", " max_new_tokens=256,\n", " context_size=BASE_CONFIG[\"context_length\"],\n", " eos_id=50256\n", " )\n", " generated_text = token_ids_to_text(token_ids, tokenizer)\n", " reference_response_text = (\n", " generated_text[len(input_text):]\n", " .replace(\"### Response:\", \"\")\n", " .strip()\n", " )\n", "\n", " token_ids = generate(\n", " model=policy_model,\n", " idx=text_to_token_ids(input_text, tokenizer).to(device),\n", " max_new_tokens=256,\n", " context_size=BASE_CONFIG[\"context_length\"],\n", " eos_id=50256\n", " )\n", " generated_text = token_ids_to_text(token_ids, tokenizer)\n", " policy_response_text = (\n", " generated_text[len(input_text):]\n", " .replace(\"### Response:\", \"\")\n", " .strip()\n", " )\n", "\n", " print(input_text)\n", " print(f\"\\nCorrect response:\\n>> {entry['output']}\")\n", " print(f\"\\nReference model response:\\n>> {reference_response_text.strip()}\")\n", " print(f\"\\nPolicy model response:\\n>> {policy_response_text.strip()}\")\n", " print(\"\\n-------------------------------------\\n\")" ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "A100", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 5 }