{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "xvSGDbExff_I" }, "source": [ "# 微调 LLM：实现抽取式问答\n", "\n", "> 指导文章：[22a. 微调 LLM：实现抽取式问答](../Guide/22a.%20微调%20LLM：实现抽取式问答.md) | 作业文章：[22b. 作业 - Bert 微调抽取式问答](../Guide/22b.%20作业%20-%20Bert%20微调抽取式问答.md)\n", "\n", "**在线链接**：[Kaggle](https://www.kaggle.com/code/aidemos/21a-llm) | [Colab](https://colab.research.google.com/drive/1jgdoO7fKk7Tsn2yi28ytsDQ8VXdthnIm?usp=sharing)\n", "\n", "## 前言\n", "\n", "**预训练 + 微调**是一个非常主流的范式，适用于各种下游任务，如文本分类、命名实体识别、机器翻译等。在这篇文章中，我们将以**抽取式问答任务**为例，再次尝试微调预训练模型。\n", "\n", "首先，了解什么是**抽取式问答**：根据「给定的问题」和「**包含**答案的文本」，从中**抽取**出对应的答案片段，**不需要生成新的词语**。\n", "\n", "**举例说明**：\n", "\n", "- **文本**：`BERT 是由 Google 提出的预训练语言模型，它在多个 NLP 任务上取得了 SOTA 的成绩。`\n", "- **问题**：`谁提出了 BERT？`\n", "- **答案**：`Google`\n", "\n", "> 如果去掉“抽取式”的限定，广义上的“问答”更接近于**生成式问答（Generative Question Answering）**，即答案并非固定的文本片段，模型基于理解进行**生成**，最终的答案不拘泥于特定的文本。\n", ">\n", "> **举例说明**：\n", ">\n", "> - **文本**：同上。\n", "> - **问题**：同上。\n", "> - **答案**：`BERT 是由 Google 提出的预训练语言模型。具体来说，它是由 Jacob Devlin 等研究人员在 2018 年的论文《BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding》中首次介绍的。BERT 在多个 NLP 任务上取得了 SOTA（State-of-the-Art）的成绩，推动了自然语言处理领域的快速发展。`（该答案由 GPT-4o 生成）\n", "\n", "#### Q: 模型怎么完成抽取式问答任务？输出是什么？\n", "\n", "停下来思考一下，是直接生成答案对应的词或句子吗？\n", "\n", "**不是**，输出的是**答案在文本中的起始和结束位置**。通过下图进行理解：\n", "\n", "> ![Extractive-QA-model](../Guide/assets/Extractive-QA-model.png)\n", "\n", "模型的最终输出为两个向量：起始位置得分向量 $\\mathbf{s} \\in \\mathbb{R}^N$ 和结束位置得分向量 $\\mathbf{e} \\in \\mathbb{R}^N$，其中 $N$ 是输入序列的长度。\n", "\n", "对于每个位置 $i$，模型计算其作为答案起始位置和结束位置的得分：\n", "\n", "$$\n", "\\begin{aligned}\n", "s_i &= \\mathbf{w}_{\\text{start}} \\mathbf{h}_i + b_{\\text{start}} \\\\\n", "e_i &= \\mathbf{w}_{\\text{end}} \\mathbf{h}_i + b_{\\text{end}}\n", "\\end{aligned}\n", "$$\n", "\n", "其中, $\\mathbf{h}_i \\in \\mathbb{R}^H$ 是编码器在位置 $i$ 的隐藏状态输出 ($\\mathbf{h}$ 就是 BERT 模型的最终输出), $H$ 是隐藏层的维度。$\\mathbf{w}_{\\text{start}} \\in \\mathbb{R}^H$ 和 $\\mathbf{w}_{\\text{end}} \\in \\mathbb{R}^H$ 是权重向量（对应于 `nn.Linear(H, 1)`，这里写成了常见的数学形式，了解线性层代码的同学可以当做 $\\mathbf{h}\\mathbf{w}^\\top$）, $b_{\\text{start}}$ 和 $b_{\\text{end}}$ 是偏置项。\n", "\n", "然后，对得分向量进行 softmax 操作，得到每个位置作为起始和结束位置的概率分布：\n", "\n", "$$\n", "\\begin{aligned}\n", "P_{\\text{start}}(i) &= \\frac{e^{s_i}}{\\sum_{j=1}^{N} e^{s_j}} \\\\\n", "P_{\\text{end}}(i) &= \\frac{e^{e_i}}{\\sum_{j=1}^{N} e^{e_j}}\n", "\\end{aligned}\n", "$$\n", "\n", "在推理时，选择具有最高概率的起始位置 $\\hat{s}$ 和结束位置 $\\hat{e}$。为了保证答案的合理性，通常要求 $\\hat{s} \\leq \\hat{e}$，并且答案的长度不超过预设的最大长度 $L_{\\text{max}}$。此时的行为称为后处理（Postprocessing），根据实际需求进行。\n", "\n", "最终，答案就是输入序列中从位置 $\\hat{s}$ 到 $\\hat{e}$ 的片段，即：\n", "\n", "$$\n", "\\text{Answer} = \\text{Input}[\\hat{s}:\\hat{e}]\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": { "id": "NYAHsHNbzdKm" }, "source": [ "## 前置准备" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 下载数据集" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "YPrc4Eie9Yo5", "outputId": "1640c875-58d1-4288-e382-e9fa9db39b9f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "数据集已存在，跳过下载\n" ] } ], "source": [ "# 通过以下命令下载数据集\n", "import os\n", "\n", "if all(os.path.exists(f) for f in [\"hw7_train.json\", \"hw7_dev.json\", \"hw7_test.json\"]):\n", " print(\"数据集已存在，跳过下载\")\n", "else:\n", " !wget -nc https://github.com/Hoper-J/HUNG-YI_LEE_Machine-Learning_Homework/raw/refs/heads/master/HW07/ml2023spring-hw7.zip\n", " !unzip -o ml2023spring-hw7.zip" ] }, { "cell_type": "markdown", "metadata": { "id": "TevOvhC03m0h" }, "source": [ "### 安装库 " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "tbxWFX_jpDom", "outputId": "138488c2-96fa-4c41-d450-9d0a1b74f36c", "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.57.6)\n", "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.25.2)\n", "Requirement already satisfied: huggingface-hub<1.0,>=0.34.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.36.2)\n", "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (25.0)\n", "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.3)\n", "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2026.4.4)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.5)\n", "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.22.2)\n", "Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.7.0)\n", "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.67.1)\n", "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (2026.2.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (1.4.3)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (4.15.0)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.4)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.5.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2025.11.12)\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\n", "\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.3\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m26.1.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3 -m pip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (1.13.0)\n", "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate) (1.26.4)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (25.0)\n", "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (7.1.3)\n", "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate) (6.0.3)\n", "Requirement already satisfied: torch>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (2.5.1+cu121)\n", "Requirement already satisfied: huggingface_hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (0.36.2)\n", "Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.10/dist-packages (from accelerate) (0.7.0)\n", "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface_hub>=0.21.0->accelerate) (3.25.2)\n", "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub>=0.21.0->accelerate) (2026.2.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub>=0.21.0->accelerate) (1.4.3)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface_hub>=0.21.0->accelerate) (2.32.5)\n", "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub>=0.21.0->accelerate) (4.67.1)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface_hub>=0.21.0->accelerate) (4.15.0)\n", "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (3.4.2)\n", "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (3.1.6)\n", "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (12.1.105)\n", "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (12.1.105)\n", "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (12.1.105)\n", "Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (9.1.0.70)\n", "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (12.1.3.1)\n", "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (11.0.2.54)\n", "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (10.3.2.106)\n", "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (11.4.5.107)\n", "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (12.1.0.106)\n", "Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (2.21.5)\n", "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (12.1.105)\n", "Requirement already satisfied: triton==3.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (3.1.0)\n", "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.0->accelerate) (1.13.1)\n", "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=2.0.0->accelerate) (12.9.86)\n", "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch>=2.0.0->accelerate) (1.3.0)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=2.0.0->accelerate) (3.0.3)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub>=0.21.0->accelerate) (3.4.4)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub>=0.21.0->accelerate) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub>=0.21.0->accelerate) (2.5.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface_hub>=0.21.0->accelerate) (2025.11.12)\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\n", "\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.3\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m26.1.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3 -m pip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install transformers\n", "%pip install accelerate" ] }, { "cell_type": "markdown", "metadata": { "id": "8dKM4yCh4LI_" }, "source": [ "### 导入库\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "WOTHHtWJoahe" }, "outputs": [], "source": [ "# ========== 标准库模块 ==========\n", "import os\n", "# 设置模型下载镜像\n", "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n", "\n", "import json\n", "import random\n", "\n", "# ========== 第三方库 ==========\n", "import numpy as np\n", "from tqdm.auto import tqdm\n", "\n", "# ========== 深度学习相关库 ==========\n", "import torch\n", "from torch.utils.data import DataLoader, Dataset\n", "from torch.optim.lr_scheduler import LambdaLR\n", "from torch.optim import AdamW\n", "\n", "# Transformers (Hugging Face)\n", "from transformers import (\n", " AutoTokenizer,\n", " AutoModelForQuestionAnswering,\n", " get_linear_schedule_with_warmup\n", ")\n", "\n", "# 加速库\n", "from accelerate import Accelerator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 设置设备和随机数种子" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "# For Mac M1, M2...\n", "# device = \"mps\" if torch.backends.mps.is_available() else (\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "def same_seeds(seed):\n", " \"\"\"\n", " 设置随机种子以确保结果的可复现性。\n", "\n", " 参数:\n", " seed (int): 要设置的随机种子值。\n", " \"\"\"\n", " torch.manual_seed(seed)\n", " if torch.cuda.is_available():\n", " torch.cuda.manual_seed(seed)\n", " torch.cuda.manual_seed_all(seed)\n", " np.random.seed(seed)\n", " random.seed(seed)\n", " torch.backends.cudnn.benchmark = False\n", " torch.backends.cudnn.deterministic = True" ] }, { "cell_type": "markdown", "metadata": { "id": "2YgXHuVLp_6j" }, "source": [ "## 加载模型和分词器\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 220, "referenced_widgets": [ "641665cb9ec6433a81719d57eeaf8298", "ba9c1cd3cfe04add90aa2edf2aa87378", "1e20837bf23548249678da0a07745d58", "c1d24cb7607c44b989774113a2baf675", "c9144705a33848a6b142f9887da6ac49", "ebfa4ba9f18b4945947d326581d3b8e9", "28fc746a3c0548f4a4ff669b5dbb7320", "d26e50974c52452389c5abb001056546", "9eefbb580f0146658d18658f210d3a67", "c813a6f6ce09424582c06a3ac8a7d712", "ff6cd33dca3c44ba9832b2d705fd2de1", "8b264499afca4694849b08e2091f6826", "3eaf82623d964c2abb92b4509ac5ca9d", "aa3ff7831fbf47efa404805886ca4696", "6be63de77ed14eee9c37e93d3a75f743", "5195ed241f09440eaf84a7e0b3dc8155", "33fa92f225be477d8cd176a0a4e8894e", "b80166eeee984661a611a02d1eae4458", "20cd68a7e03f4353aa99fcefc1f16d36", "fb908d9c3e374983aa47478a95411ba7", "304609dbefa74735baa08ce7aadff92d", "250451c12cee4aacb91a32d7da4ab861", "2c37badbe0504073b31abd67a8c0b808", "bfd7d35860c14dd3b8e5eea07032311d", "db2c74f92f6d4a2c8bfdcb91d5bd3d64", "b0af063597ea4de5a8c8bf4f370f782a", "5b1ac1a0f04f4c29b3c1127ec3d95e81", "b91c8359a6444497b3665e66afbe237d", "1181117783fd4bca9d7f5f296507aff3", "941a345ba82a44d5ad597a15c0af0d5c", "23afbee21a644745b24b8ffefed6b286", "a075ee85c2d34ab3b2d011c63dc02286", "052cccb5d69d476f9caac95848e0353a" ] }, "id": "xyBCYGjAp3ym", "outputId": "81f25c6d-cd7f-4be1-e085-f8a54bd1c27d", "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-base-chinese and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] } ], "source": [ "model = AutoModelForQuestionAnswering.from_pretrained(\"bert-base-chinese\").to(device)\n", "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-chinese\")\n", "\n", "# 预训练模型也可以换成其他的\n", "# model = AutoModelForQuestionAnswering.from_pretrained(\"luhua/chinese_pretrain_mrc_macbert_large\").to(device)\n", "# tokenizer = AutoTokenizer.from_pretrained(\"luhua/chinese_pretrain_mrc_macbert_large\")\n", "\n", "# 你可以忽略警告消息（它弹出是因为新的 QA 预测头是随机初始化的）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q: 什么是 `AutoModelForQuestionAnswering`？\n", "\n", "`AutoModelForQuestionAnswering` 是 Hugging Face 提供的自动模型加载类，除了加载指定模型的预训练权重之外，**它会在模型的顶层添加一个用于问答任务的输出层**，用于预测答案的起始位置和结束位置。如果模型本身已经包含用于问答任务的输出层，则会加载相应的权重；如果没有，则会在顶层添加一个**新的输出层**，随机初始化权重（这时候会有警告信息）。\n", "\n", "不妨打印看看这个输出层到底是什么：" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Linear(in_features=768, out_features=2, bias=True)\n" ] } ], "source": [ "print(model.qa_outputs)\n", "# print(model) # 如果感兴趣，也可以打印整个模型" ] }, { "cell_type": "markdown", "metadata": { "id": "3Td-GTmk5OW4" }, "source": [ "## 数据部分\n", "\n", "> 两个繁体中文阅读理解数据集：[DRCD](https://github.com/DRCKnowledgeTeam/DRCD) 和 [ODSQA](https://github.com/Chia-Hsuan-Lee/ODSQA)。\n", "\n", "- **训练集（DRCD + DRCD-backtrans）**：包含 15,329 个段落和 26,918 个问题。一个段落可能对应多个问题。\n", "- **开发集（DRCD + DRCD-backtrans）**：包含 1,255 个段落和 2,863 个问题。用于验证。\n", "- **测试集（DRCD + ODSQA）**：包含 1,606 个段落和 3,504 个问题。测试集的段落没有提供答案，需要模型进行预测。\n", "\n", "所有数据集的格式相同：\n", "\n", "- `id`：问题编号\n", "- `paragraph_id`：段落编号\n", "- `question_text`：问题文本\n", "- `answer_text`：答案文本\n", "- `answer_start`：答案在段落中的起始字符位置\n", "- `answer_end`：答案在段落中的结束字符位置\n", "\n", "![数据集格式](../Guide/assets/4215768313590de87aab01adcad78c90.png)\n", "\n", "### 读取数据" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "NvX7hlepogvu" }, "outputs": [], "source": [ "def read_data(file):\n", " with open(file, 'r', encoding=\"utf-8\") as reader:\n", " data = json.load(reader)\n", " return data[\"questions\"], data[\"paragraphs\"]\n", "\n", "train_questions, train_paragraphs = read_data(\"hw7_train.json\")\n", "dev_questions, dev_paragraphs = read_data(\"hw7_dev.json\")\n", "test_questions, test_paragraphs = read_data(\"hw7_test.json\")" ] }, { "cell_type": "markdown", "metadata": { "id": "Fm0rpTHq0e4N" }, "source": [ "### 分词处理" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "rTZ6B70Hoxie", "outputId": "db13805e-02a9-4cbe-e6d2-66c0e3860c2b", "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Token indices sequence length is longer than the specified maximum sequence length for this model (566 > 512). Running this sequence through the model will result in indexing errors\n" ] } ], "source": [ "# 分别对问题和段落进行分词\n", "# 设置 add_special_tokens=False，因为在自定义数据集 QA_Dataset 的 __getitem__ 中会手动添加特殊标记\n", "\n", "train_questions_tokenized = tokenizer(\n", " [q[\"question_text\"] for q in train_questions], add_special_tokens=False\n", ")\n", "dev_questions_tokenized = tokenizer(\n", " [q[\"question_text\"] for q in dev_questions], add_special_tokens=False\n", ")\n", "test_questions_tokenized = tokenizer(\n", " [q[\"question_text\"] for q in test_questions], add_special_tokens=False\n", ")\n", "\n", "train_paragraphs_tokenized = tokenizer(train_paragraphs, add_special_tokens=False)\n", "dev_paragraphs_tokenized = tokenizer(dev_paragraphs, add_special_tokens=False)\n", "test_paragraphs_tokenized = tokenizer(test_paragraphs, add_special_tokens=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "Ws8c8_4d5UCI" }, "source": [ "### 自定义数据集处理\n", "\n", "以下代码定义了一个 `QA_Dataset` 类，用于处理问答数据。" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "Xjooag-Swnuh" }, "outputs": [], "source": [ "class QA_Dataset(Dataset):\n", " \"\"\"\n", " 自定义的问答数据集类，用于处理问答任务的数据。\n", "\n", " 参数:\n", " split (str): 数据集的类型，'train'、'dev' 或 'test'。\n", " questions (list): 问题列表，每个元素是一个字典，包含问题的详细信息。\n", " tokenized_questions (BatchEncoding): 分词后的问题，由 tokenizer 生成。\n", " tokenized_paragraphs (BatchEncoding): 分词后的段落列表，由 tokenizer 生成。\n", " \n", " 属性:\n", " max_question_len (int): 问题的最大长度（以分词后的 token 数计）。\n", " max_paragraph_len (int): 段落的最大长度（以分词后的 token 数计）。\n", " doc_stride (int): 段落窗口滑动步长。\n", " max_seq_len (int): 输入序列的最大长度。\n", " \"\"\"\n", " \n", " def __init__(self, split, questions, tokenized_questions, tokenized_paragraphs):\n", " self.split = split\n", " self.questions = questions\n", " self.tokenized_questions = tokenized_questions\n", " self.tokenized_paragraphs = tokenized_paragraphs\n", " self.max_question_len = 60\n", " self.max_paragraph_len = 150\n", "\n", " # 设置段落窗口滑动步长为段落最大长度的 10%\n", " self.doc_stride = int(self.max_paragraph_len * 0.1)\n", "\n", " # 输入序列长度 = [CLS] + question + [SEP] + paragraph + [SEP]\n", " self.max_seq_len = 1 + self.max_question_len + 1 + self.max_paragraph_len + 1\n", "\n", " def __len__(self):\n", " \"\"\"\n", " 返回数据集中样本的数量。\n", "\n", " 返回:\n", " int: 数据集的长度\n", " \"\"\"\n", " return len(self.questions)\n", "\n", " def __getitem__(self, idx):\n", " \"\"\"\n", " 获取数据集中指定索引的样本。\n", "\n", " 参数:\n", " idx (int): 样本的索引\n", "\n", " 返回:\n", " 对于训练集：返回一个输入张量和对应的答案位置\n", " (input_ids, token_type_ids, attention_mask, answer_start_token, answer_end_token)\n", " 对于验证/测试集：返回包含多个窗口的输入张量列表\n", " (input_ids_list, token_type_ids_list, attention_mask_list)\n", " \"\"\"\n", " question = self.questions[idx]\n", " tokenized_question = self.tokenized_questions[idx]\n", " tokenized_paragraph = self.tokenized_paragraphs[question[\"paragraph_id\"]]\n", "\n", " ##### 预处理 #####\n", " if self.split == \"train\":\n", " # 将答案在段落文本中的起始/结束位置转换为在分词后段落中的起始/结束位置\n", " answer_start_token = tokenized_paragraph.char_to_token(question[\"answer_start\"])\n", " answer_end_token = tokenized_paragraph.char_to_token(question[\"answer_end\"])\n", "\n", " # 防止模型学习到「答案总是位于中间的位置」，加入随机偏移\n", " mid = (answer_start_token + answer_end_token) // 2\n", " max_offset = self.max_paragraph_len // 2 # 最大偏移量为段落长度的1/2，这是可调的\n", " random_offset = np.random.randint(-max_offset, max_offset) # 在 [-max_offset, +max_offset] 范围内随机选择偏移量\n", " paragraph_start = max(0, min(mid + random_offset - self.max_paragraph_len // 2, len(tokenized_paragraph) - self.max_paragraph_len))\n", " paragraph_end = paragraph_start + self.max_paragraph_len\n", " \n", " # 切片问题/段落，并添加特殊标记（101：CLS，102：SEP）\n", " input_ids_question = [101] + tokenized_question.ids[:self.max_question_len] + [102]\n", " # ... = [tokenizer.cls_token_id] + tokenized_question.ids[: self.max_question_len] + [tokenizer.sep_token_id]\n", " input_ids_paragraph = tokenized_paragraph.ids[paragraph_start : paragraph_end] + [102]\n", " # ... = ... + [tokenizer.sep_token_id]\n", "\n", " # 将答案在分词后段落中的起始/结束位置转换为窗口中的起始/结束位置\n", " answer_start_token += len(input_ids_question) - paragraph_start\n", " answer_end_token += len(input_ids_question) - paragraph_start\n", "\n", " # 填充序列，生成模型的输入\n", " input_ids, token_type_ids, attention_mask = self.padding(input_ids_question, input_ids_paragraph)\n", " \n", " return torch.tensor(input_ids), torch.tensor(token_type_ids), torch.tensor(attention_mask), answer_start_token, answer_end_token\n", "\n", " else:\n", " # 验证集和测试集的处理\n", " input_ids_list, token_type_ids_list, attention_mask_list = [], [], []\n", "\n", " # 段落被分割成多个窗口，每个窗口的起始位置由步长 \"doc_stride\" 分隔\n", " for i in range(0, len(tokenized_paragraph), self.doc_stride):\n", " # 切片问题/段落并添加特殊标记（101：CLS，102：SEP）\n", " input_ids_question = [101] + tokenized_question.ids[:self.max_question_len] + [102]\n", " # ... = [tokenizer.cls_token_id] + tokenized_question.ids[: self.max_question_len] + [tokenizer.sep_token_id]\n", " input_ids_paragraph = tokenized_paragraph.ids[i : i + self.max_paragraph_len] + [102]\n", " # ... = ... + [tokenizer.sep_token_id]\n", "\n", " # 填充序列，生成模型的输入\n", " input_ids, token_type_ids, attention_mask = self.padding(input_ids_question, input_ids_paragraph)\n", "\n", " input_ids_list.append(input_ids)\n", " token_type_ids_list.append(token_type_ids)\n", " attention_mask_list.append(attention_mask)\n", "\n", " return torch.tensor(input_ids_list), torch.tensor(token_type_ids_list), torch.tensor(attention_mask_list)\n", "\n", " def padding(self, input_ids_question, input_ids_paragraph):\n", " \"\"\"\n", " 对输入的序列进行填充，生成统一长度的模型输入。\n", "\n", " 参数:\n", " input_ids_question (list): 问题部分的输入 ID 列表\n", " input_ids_paragraph (list): 段落部分的输入 ID 列表\n", "\n", " 返回:\n", " input_ids (list): 填充后的输入 ID 列表\n", " token_type_ids (list): 区分问题和段落的标记列表\n", " attention_mask (list): 注意力掩码列表，指示哪些位置是有效的输入\n", " \"\"\"\n", " # 计算需要填充的长度\n", " padding_len = self.max_seq_len - len(input_ids_question) - len(input_ids_paragraph)\n", " # 填充输入序列\n", " input_ids = input_ids_question + input_ids_paragraph + [0] * padding_len\n", " # 构造区分问题和段落的 token_type_ids\n", " token_type_ids = [0] * len(input_ids_question) + [1] * len(input_ids_paragraph) + [0] * padding_len\n", " # 构造注意力掩码，有效位置为 1，填充位置为 0\n", " attention_mask = [1] * (len(input_ids_question) + len(input_ids_paragraph)) + [0] * padding_len\n", "\n", " return input_ids, token_type_ids, attention_mask\n", "\n", "train_set = QA_Dataset(\"train\", train_questions, train_questions_tokenized, train_paragraphs_tokenized)\n", "dev_set = QA_Dataset(\"dev\", dev_questions, dev_questions_tokenized, dev_paragraphs_tokenized)\n", "test_set = QA_Dataset(\"test\", test_questions, test_questions_tokenized, test_paragraphs_tokenized)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**解释**：\n", "\n", "- **训练集处理**：定位答案的起始和结束位置，将包含答案的段落部分截取为一个窗口（引入随机偏移，防止模型过拟合于答案总在中间的位置）。然后将问题和段落合并为一个输入序列，并进行填充。\n", "\n", "- **验证/测试集处理**：将段落分成多个窗口，每个窗口之间的步长由 `self.doc_stride` 决定，然后将每个窗口作为模型的输入。验证和测试时不需要答案位置，因此只需生成多个窗口作为输入。\n", "\n", " - `self.doc_stride` 通过控制窗口之间的滑动步长。\n", "\n", " - **训练阶段**不需要使用 `doc_stride`，因为训练时我们已经知道答案的位置，可以直接截取包含答案的窗口。但在**验证和测试**阶段，由于模型并不知道答案的位置，`doc_stride` 保证每个窗口之间有足够的重叠（overlap），减少遗漏答案。" ] }, { "cell_type": "markdown", "metadata": { "id": "5_H1kqhR8CdM" }, "source": [ "## 评估函数" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "SqeA3PLPxOHu" }, "outputs": [], "source": [ "def evaluate(data, output):\n", " \"\"\"\n", " 对模型的输出进行后处理，获取预测的答案文本。\n", "\n", " 参数:\n", " data (tuple): 包含输入数据的元组，(input_ids, token_type_ids, attention_mask)。\n", " output (transformers.modeling_outputs.QuestionAnsweringModelOutput): 模型的输出结果。\n", "\n", " 返回:\n", " answer (str): 模型预测的答案文本。\n", " \"\"\"\n", " answer = ''\n", " max_prob = float('-inf')\n", " num_of_windows = data[0].shape[1]\n", "\n", " for k in range(num_of_windows):\n", " # 通过选择最可能的起始位置/结束位置来获得答案\n", " start_prob, start_index = torch.max(output.start_logits[k], dim=0)\n", " end_prob, end_index = torch.max(output.end_logits[k], dim=0)\n", "\n", " # 确保起始位置索引小于或等于结束位置索引，避免选择错误的起始和结束位置对\n", " if start_index <= end_index:\n", " # 答案的概率计算为 start_prob 和 end_prob 的和\n", " prob = start_prob + end_prob\n", "\n", " # 如果当前窗口的答案具有更高的概率，则更新结果\n", " if prob > max_prob:\n", " max_prob = prob\n", " # 将标记转换为字符（例如，[1920, 7032] --> \"大 金\"）\n", " answer = tokenizer.decode(data[0][0][k][start_index : end_index + 1])\n", " else:\n", " # 如果起始位置索引 > 结束位置索引，则跳过此对（可能是错误情况）\n", " continue\n", " # 移除答案中的空格（例如，\"大 金\" --> \"大金\"）\n", " return answer.replace(' ','')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 训练部分\n", "\n", "### 设置超参数" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# 超参数\n", "num_epoch = 1 # 训练的轮数\n", "validation = True # 是否在每个 epoch 结束后进行验证\n", "logging_step = 100 # 每隔多少步打印一次训练日志\n", "learning_rate = 1e-5 # 学习率\n", "train_batch_size = 8 # 训练时的批次大小\n", "\n", "# 优化器\n", "optimizer = AdamW(model.parameters(), lr=learning_rate)\n", "\n", "# 数据加载器\n", "# 注意：不要更改 dev_loader / test_loader 的批次大小！\n", "# 虽然批次大小=1，但它实际上是由同一对 QA 的多个窗口组成的批次\n", "train_loader = DataLoader(train_set, batch_size=train_batch_size, shuffle=True, pin_memory=True)\n", "dev_loader = DataLoader(dev_set, batch_size=1, shuffle=False, pin_memory=True)\n", "test_loader = DataLoader(test_set, batch_size=1, shuffle=False, pin_memory=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 学习率调度器\n", "\n", "#### 带有 Warmup 的线性衰减\n", "\n", "使用调度器一般可以加速模型的收敛速度，不同的 Warmup 比例对学习率的影响：\n", "![不同 Warmup 比例的学习率曲线](../Guide/assets/20240920020716.png)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# 总训练步数\n", "total_steps = len(train_loader) * num_epoch\n", "num_warmup_steps = int(0.2 * total_steps) # TODO: 调整 warmup 步数的比率\n", "\n", "# [Hugging Face] 应用带有 warmup 的线性学习率衰减\n", "scheduler = get_linear_schedule_with_warmup(\n", " optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=total_steps\n", ")\n", "\n", "# # [PyTorch] 替代方法：应用不带 warmup 的线性学习率衰减\n", "# # lr_lambda 自定义学习率随时间衰减（此处是简单的线性衰减）\n", "# lr_lambda = lambda step: max(0.0, 1.0 - step / total_steps)\n", "# scheduler = LambdaLR(optimizer, lr_lambda=lr_lambda)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 设置 Accelerator\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "#### 梯度累积（可选）####\n", "# 注意：train_batch_size * gradient_accumulation_steps = 有效批次大小\n", "# 如果 CUDA 内存不足，你可以降低 train_batch_size 并提高 gradient_accumulation_steps\n", "# 文档：https://huggingface.co/docs/accelerate/usage_guides/gradient_accumulation\n", "gradient_accumulation_steps = 1\n", "\n", "# 将 \"fp16_training\" 更改为 True 以支持自动混合精度训练（fp16）\n", "fp16_training = True\n", "if fp16_training:\n", " accelerator = Accelerator(mixed_precision=\"fp16\", gradient_accumulation_steps=gradient_accumulation_steps)\n", "else:\n", " accelerator = Accelerator(gradient_accumulation_steps=gradient_accumulation_steps)\n", "\n", "model, optimizer, train_loader, scheduler = accelerator.prepare(model, optimizer, train_loader, scheduler)" ] }, { "cell_type": "markdown", "metadata": { "id": "rzHQit6eMnKG" }, "source": [ "### 开始训练" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "开始训练...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "26f4ba9a3e484d1b8a5cb3d3bc4923c7", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/3365 [00:00