{ "cells": [ { "cell_type": "markdown", "id": "4421e9ed-db25-40cd-b8e1-ce699a2023ab", "metadata": {}, "source": [ "# DPO 微调：根据偏好引导LLM的输出\n", "> [GenAI HW6: LLM Values Alignment](https://colab.research.google.com/drive/1d3zmkqo-ZmxrIOYWSe3vDD0za8tUPguu?usp=sharing#scrollTo=owGIuqdnRI8I) 中文镜像版\n", ">\n", "> 指导文章：[11. DPO 微调示例：根据人类偏好优化 LLM 大语言模型](https://github.com/Hoper-J/LLM-Guide-and-Demos-zh_CN/blob/master/Guide/11.%20DPO%20微调示例：根据人类偏好优化%20LLM%20大语言模型.md)\n", "\n", "**目标**：学习如何使用带标签的偏好数据来对齐模型的行为。\n", "\n", "大白话就是：微调。但这里使用的是 DPO 方法（之后会具体解释什么是 DPO）。\n", "\n", "之前微调唐诗更像是一个传统的方法，使用问题和答案驱动的微调方式，数据标注成本很高。而 DPO 不同，它使用的是人类对文本的偏好对：即人类认为一对文本中哪个更好。\n", "\n", "在线链接：[Kaggle](https://www.kaggle.com/code/aidemos/09-dpo-llm) | [Colab](https://colab.research.google.com/drive/17PgTMSBqSLNFSqgVphmcCulpAkEOWYtT?usp=sharing)\n" ] }, { "cell_type": "markdown", "id": "c6124745-9431-4889-8d6d-c7960c90c409", "metadata": {}, "source": [ "## 安装和导入一些必要的库" ] }, { "cell_type": "code", "execution_count": 1, "id": "e633546a-9dd3-447c-a26f-76bd64376878", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: transformers in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.56.2)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (3.29.0)\n", "Requirement already satisfied: huggingface-hub<1.0,>=0.34.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.36.2)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2.4.4)\n", "Requirement already satisfied: packaging>=20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (26.1)\n", "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (6.0.3)\n", "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2026.4.4)\n", "Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2.33.1)\n", "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.22.2)\n", "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.7.0)\n", "Requirement already satisfied: tqdm>=4.27 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (4.67.3)\n", "Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (2025.3.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (1.4.3)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (4.15.0)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (3.4.7)\n", "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (2026.2.25)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: bitsandbytes in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.49.2)\n", "Requirement already satisfied: torch<3,>=2.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from bitsandbytes) (2.7.1)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from bitsandbytes) (2.4.4)\n", "Requirement already satisfied: packaging>=20.9 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from bitsandbytes) (26.1)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (3.29.0)\n", "Requirement already satisfied: typing-extensions>=4.10.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (4.15.0)\n", "Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (82.0.1)\n", "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (1.14.0)\n", "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (3.6.1)\n", "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (3.1.6)\n", "Requirement already satisfied: fsspec in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (2025.3.0)\n", "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.6.80)\n", "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (9.5.1.17)\n", "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.6.4.1)\n", "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (11.3.0.4)\n", "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (10.3.7.77)\n", "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (11.7.1.2)\n", "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.5.4.2)\n", "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (0.6.3)\n", "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (2.26.2)\n", "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.6.77)\n", "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (12.6.85)\n", "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (1.11.1.6)\n", "Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch<3,>=2.3->bitsandbytes) (3.3.1)\n", "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch<3,>=2.3->bitsandbytes) (1.3.0)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch<3,>=2.3->bitsandbytes) (3.0.3)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: datasets in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.0.0)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (3.29.0)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (2.4.4)\n", "Requirement already satisfied: pyarrow>=15.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (23.0.1)\n", "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (0.3.8)\n", "Requirement already satisfied: pandas in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (2.3.3)\n", "Requirement already satisfied: requests>=2.32.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (2.33.1)\n", "Requirement already satisfied: tqdm>=4.66.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (4.67.3)\n", "Requirement already satisfied: xxhash in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (3.6.0)\n", "Requirement already satisfied: multiprocess<0.70.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (0.70.16)\n", "Requirement already satisfied: fsspec<=2025.3.0,>=2023.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (2025.3.0)\n", "Requirement already satisfied: huggingface-hub>=0.24.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (0.36.2)\n", "Requirement already satisfied: packaging in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (26.1)\n", "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets) (6.0.3)\n", "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (3.13.5)\n", "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (2.6.1)\n", "Requirement already satisfied: aiosignal>=1.4.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (1.4.0)\n", "Requirement already satisfied: attrs>=17.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (26.1.0)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (1.8.0)\n", "Requirement already satisfied: multidict<7.0,>=4.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (6.7.1)\n", "Requirement already satisfied: propcache>=0.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (0.4.1)\n", "Requirement already satisfied: yarl<2.0,>=1.17.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (1.23.0)\n", "Requirement already satisfied: idna>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from yarl<2.0,>=1.17.0->aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (3.11)\n", "Requirement already satisfied: typing-extensions>=4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiosignal>=1.4.0->aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets) (4.15.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub>=0.24.0->datasets) (1.4.3)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets) (3.4.7)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets) (2026.2.25)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets) (2.9.0.post0)\n", "Requirement already satisfied: pytz>=2020.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets) (2026.1.post1)\n", "Requirement already satisfied: tzdata>=2022.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets) (2026.1)\n", "Requirement already satisfied: six>=1.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: peft in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.17.1)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (2.4.4)\n", "Requirement already satisfied: packaging>=20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (26.1)\n", "Requirement already satisfied: psutil in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (7.2.2)\n", "Requirement already satisfied: pyyaml in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (6.0.3)\n", "Requirement already satisfied: torch>=1.13.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (2.7.1)\n", "Requirement already satisfied: transformers in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (4.56.2)\n", "Requirement already satisfied: tqdm in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (4.67.3)\n", "Requirement already satisfied: accelerate>=0.21.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (1.13.0)\n", "Requirement already satisfied: safetensors in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (0.7.0)\n", "Requirement already satisfied: huggingface_hub>=0.25.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft) (0.36.2)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.25.0->peft) (3.29.0)\n", "Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.25.0->peft) (2025.3.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.25.0->peft) (1.4.3)\n", "Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.25.0->peft) (2.33.1)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.25.0->peft) (4.15.0)\n", "Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (82.0.1)\n", "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (1.14.0)\n", "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (3.6.1)\n", "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (3.1.6)\n", "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.6.80)\n", "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (9.5.1.17)\n", "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.6.4.1)\n", "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (11.3.0.4)\n", "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (10.3.7.77)\n", "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (11.7.1.2)\n", "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.5.4.2)\n", "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (0.6.3)\n", "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (2.26.2)\n", "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.6.77)\n", "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (12.6.85)\n", "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (1.11.1.6)\n", "Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft) (3.3.1)\n", "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=1.13.0->peft) (1.3.0)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch>=1.13.0->peft) (3.0.3)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.25.0->peft) (3.4.7)\n", "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.25.0->peft) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.25.0->peft) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.25.0->peft) (2026.2.25)\n", "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers->peft) (2026.4.4)\n", "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers->peft) (0.22.2)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: trl in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.29.1)\n", "Requirement already satisfied: accelerate>=1.4.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from trl) (1.13.0)\n", "Requirement already satisfied: datasets>=3.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from trl) (4.0.0)\n", "Requirement already satisfied: packaging>20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from trl) (26.1)\n", "Requirement already satisfied: transformers>=4.56.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from trl) (4.56.2)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate>=1.4.0->trl) (2.4.4)\n", "Requirement already satisfied: psutil in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate>=1.4.0->trl) (7.2.2)\n", "Requirement already satisfied: pyyaml in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate>=1.4.0->trl) (6.0.3)\n", "Requirement already satisfied: torch>=2.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate>=1.4.0->trl) (2.7.1)\n", "Requirement already satisfied: huggingface_hub>=0.21.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate>=1.4.0->trl) (0.36.2)\n", "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate>=1.4.0->trl) (0.7.0)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (3.29.0)\n", "Requirement already satisfied: pyarrow>=15.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (23.0.1)\n", "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (0.3.8)\n", "Requirement already satisfied: pandas in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (2.3.3)\n", "Requirement already satisfied: requests>=2.32.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (2.33.1)\n", "Requirement already satisfied: tqdm>=4.66.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (4.67.3)\n", "Requirement already satisfied: xxhash in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (3.6.0)\n", "Requirement already satisfied: multiprocess<0.70.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=3.0.0->trl) (0.70.16)\n", "Requirement already satisfied: fsspec<=2025.3.0,>=2023.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (2025.3.0)\n", "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (3.13.5)\n", "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (2.6.1)\n", "Requirement already satisfied: aiosignal>=1.4.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (1.4.0)\n", "Requirement already satisfied: attrs>=17.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (26.1.0)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (1.8.0)\n", "Requirement already satisfied: multidict<7.0,>=4.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (6.7.1)\n", "Requirement already satisfied: propcache>=0.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (0.4.1)\n", "Requirement already satisfied: yarl<2.0,>=1.17.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (1.23.0)\n", "Requirement already satisfied: idna>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from yarl<2.0,>=1.17.0->aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (3.11)\n", "Requirement already satisfied: typing-extensions>=4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiosignal>=1.4.0->aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=3.0.0->trl) (4.15.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate>=1.4.0->trl) (1.4.3)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=3.0.0->trl) (3.4.7)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=3.0.0->trl) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=3.0.0->trl) (2026.2.25)\n", "Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (82.0.1)\n", "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (1.14.0)\n", "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (3.6.1)\n", "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (3.1.6)\n", "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.6.80)\n", "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (9.5.1.17)\n", "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.6.4.1)\n", "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (11.3.0.4)\n", "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (10.3.7.77)\n", "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (11.7.1.2)\n", "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.5.4.2)\n", "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (0.6.3)\n", "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (2.26.2)\n", "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.6.77)\n", "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (12.6.85)\n", "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (1.11.1.6)\n", "Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate>=1.4.0->trl) (3.3.1)\n", "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=2.0.0->accelerate>=1.4.0->trl) (1.3.0)\n", "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.56.2->trl) (2026.4.4)\n", "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.56.2->trl) (0.22.2)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch>=2.0.0->accelerate>=1.4.0->trl) (3.0.3)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets>=3.0.0->trl) (2.9.0.post0)\n", "Requirement already satisfied: pytz>=2020.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets>=3.0.0->trl) (2026.1.post1)\n", "Requirement already satisfied: tzdata>=2022.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets>=3.0.0->trl) (2026.1)\n", "Requirement already satisfied: six>=1.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets>=3.0.0->trl) (1.17.0)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: accelerate in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (1.13.0)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (2.4.4)\n", "Requirement already satisfied: packaging>=20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (26.1)\n", "Requirement already satisfied: psutil in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (7.2.2)\n", "Requirement already satisfied: pyyaml in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (6.0.3)\n", "Requirement already satisfied: torch>=2.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (2.7.1)\n", "Requirement already satisfied: huggingface_hub>=0.21.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (0.36.2)\n", "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate) (0.7.0)\n", "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (3.29.0)\n", "Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (2025.3.0)\n", "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (1.4.3)\n", "Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (2.33.1)\n", "Requirement already satisfied: tqdm>=4.42.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (4.67.3)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.21.0->accelerate) (4.15.0)\n", "Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (82.0.1)\n", "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (1.14.0)\n", "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (3.6.1)\n", "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (3.1.6)\n", "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.6.77)\n", "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.6.80)\n", "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (9.5.1.17)\n", "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.6.4.1)\n", "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (11.3.0.4)\n", "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (10.3.7.77)\n", "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (11.7.1.2)\n", "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.5.4.2)\n", "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (0.6.3)\n", "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (2.26.2)\n", "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.6.77)\n", "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (12.6.85)\n", "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (1.11.1.6)\n", "Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.0.0->accelerate) (3.3.1)\n", "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=2.0.0->accelerate) (1.3.0)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch>=2.0.0->accelerate) (3.0.3)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (3.4.7)\n", "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.21.0->accelerate) (2026.2.25)\n", "Note: you may need to restart the kernel to use updated packages.\n", "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: numpy in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (2.4.4)\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install transformers\n", "%pip install bitsandbytes\n", "%pip install datasets\n", "%pip install peft\n", "%pip install trl\n", "%pip install accelerate\n", "%pip install numpy" ] }, { "cell_type": "code", "execution_count": 2, "id": "7de501d5-c828-4d81-a4ce-31e029b7bc2a", "metadata": {}, "outputs": [], "source": [ "import os\n", "# 设置模型下载镜像（注意，需要在导入 transformers 等模块前进行设置才能起效）\n", "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n", "\n", "import re\n", "import json\n", "\n", "import torch\n", "import pandas as pd\n", "from tqdm.auto import tqdm\n", "\n", "from datasets import Dataset\n", "from peft import LoraConfig\n", "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, GenerationConfig\n", "from trl import DPOConfig, DPOTrainer" ] }, { "cell_type": "markdown", "id": "79ef6669-e845-4d9c-a360-fd4283790bcb", "metadata": { "tags": [] }, "source": [ "## 加载数据集" ] }, { "cell_type": "code", "execution_count": 3, "id": "de383280-d9fa-4b86-9250-ccb58212e946", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "fatal: destination path 'GenAI_hw6_dataset' already exists and is not an empty directory.\n" ] } ], "source": [ "!git clone https://github.com/Baiiiiiiiiii/GenAI_hw6_dataset.git" ] }, { "cell_type": "code", "execution_count": 4, "id": "12734922-1d3d-46d7-acdc-7d484f6dd577", "metadata": { "scrolled": true }, "outputs": [], "source": [ "with open(\"./GenAI_hw6_dataset/labelled_data.json\", 'r') as jsonfile:\n", " full_data = json.load(jsonfile)\n", "\n", "with open(\"./GenAI_hw6_dataset/test_prompt.json\", 'r') as jsonfile:\n", " test_data = json.load(jsonfile)" ] }, { "cell_type": "code", "execution_count": 5, "id": "4887df1d-3fcc-4c39-9d23-7e82958e90d1", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "([{'id': 1,\n", " 'prompt': '日本動漫真人化是否有損原作形象？',\n", " 'support': '真人化能夠呈現更真實的角色形象，提升原作魅力。',\n", " 'oppose': '真人化可能無法完美呈現動畫中的獨特風格，損害原作形象。'},\n", " {'id': 2,\n", " 'prompt': '真人化是否能夠擴大動漫在全球的影響力？',\n", " 'support': '真人化能夠讓更多非動漫迷接觸作品，擴大影響力。',\n", " 'oppose': '真人化可能失去動漫的獨特風格，限制影響力擴大。'},\n", " {'id': 3,\n", " 'prompt': '真人化是否能夠吸引新觀眾？',\n", " 'support': '真人化能夠吸引不熟悉動漫的觀眾，擴大受眾。',\n", " 'oppose': '真人化可能讓原本的動漫迷感到失望，無法吸引新觀眾。'},\n", " {'id': 4,\n", " 'prompt': '真人化是否能夠保留原作故事情節的精髓？',\n", " 'support': '真人化有機會更深入挖掘原作故事，保留精髓。',\n", " 'oppose': '真人化可能因為改編而失去原作故事的深度與精髓。'},\n", " {'id': 5,\n", " 'prompt': '真人化是否能夠提升動漫產業的商業價值？',\n", " 'support': '真人化能夠開拓更多商業機會，提升產業價值。',\n", " 'oppose': '真人化可能讓觀眾對原作失去興趣，影響產業價值。'}],\n", " [{'id': 1, 'prompt': '真人化是否能改善日本漫畫的全球可及性？'},\n", " {'id': 2, 'prompt': '真人化如何影響年輕一代對日本漫畫的看法？'},\n", " {'id': 3, 'prompt': '真人化是否能提升原作漫畫的文學價值？'},\n", " {'id': 4, 'prompt': '真人化是否有助於保護和保存日本漫畫的傳統？'},\n", " {'id': 5, 'prompt': '真人化是否有助於提升日本漫畫行業的經濟效益？'},\n", " {'id': 6, 'prompt': '真人化如何影響日本漫畫原作者的創作動力？'},\n", " {'id': 7, 'prompt': '真人化是否對漫畫原作的忠實粉絲公平？'},\n", " {'id': 8, 'prompt': '真人化是否能夠促進日本漫畫的創新和多樣性？'},\n", " {'id': 9, 'prompt': '真人化是否有助於擴大動漫文化的市場份額？'},\n", " {'id': 10, 'prompt': '真人化是否有助於提高日本漫畫在全球的競爭力？'}])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "full_data[:5], test_data" ] }, { "cell_type": "markdown", "id": "2cfb6314-77d3-45e4-9546-6a4baf3bdc20", "metadata": { "tags": [] }, "source": [ "## 使用 HFD 下载模型\n", "\n", "我们这里使用多线程的方法进行快速下载。\n", "\n", "如果直接运行以下命令报错，根据[a. 使用 HFD 加快 Hugging Face 模型和数据集的下载](https://github.com/Hoper-J/LLM-Guide-and-Demos-zh_CN/blob/master/Guide/a.%20使用%20HFD%20加快%20Hugging%20Face%20模型和数据集的下载.md)进行前置安装。\n", "\n", "当然，你也可以取消我注释的部分，使用官方的命令进行安装，但是会很慢。" ] }, { "cell_type": "code", "execution_count": 6, "id": "fc534866-a4d4-4989-b5b3-04df0bc21f16", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2026-05-16 18:02:52-- https://hf-mirror.com/hfd/hfd.sh\n", "Resolving hf-mirror.com (hf-mirror.com)... 160.16.86.14\n", "Connecting to hf-mirror.com (hf-mirror.com)|160.16.86.14|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 13029 (13K) [application/x-shellscript]\n", "Saving to: ‘hfd.sh.6’\n", "\n", "hfd.sh.6 100%[===================>] 12.72K --.-KB/s in 0.008s \n", "\n", "2026-05-16 18:02:53 (1.53 MB/s) - ‘hfd.sh.6’ saved [13029/13029]\n", "\n" ] } ], "source": [ "!wget https://hf-mirror.com/hfd/hfd.sh\n", "!chmod a+x hfd.sh" ] }, { "cell_type": "code", "execution_count": 7, "id": "4e1b7730-a6bd-4b10-8379-a21f2f35b17d", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0;32mUsing cached metadata: Breeze-7B-Instruct-v0_1/.hfd/repo_metadata.json\u001b[0m\n", "\u001b[0;32mResume from file list: Breeze-7B-Instruct-v0_1/.hfd/aria2c_urls.txt\u001b[0m\n", "\u001b[1;33mStarting download with aria2c to Breeze-7B-Instruct-v0_1...\n", "\u001b[0mNo files to download.\n", "\u001b[0;32mDownload completed successfully. Repo directory: /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/Demos/Breeze-7B-Instruct-v0_1\n", "\u001b[0m" ] } ], "source": [ "import os\n", "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n", "\n", "# 然后执行下载\n", "!./hfd.sh 'MediaTek-Research/Breeze-7B-Instruct-v0_1' --tool aria2c -x 8\n", "# AutoDL 可以用下面这行命令，注意，需要对应更改之后的路径\n", "# !./hfd.sh 'MediaTek-Research/Breeze-7B-Instruct-v0_1' --local-dir /root/autodl-tmp/Breeze-7B-Instruct-v0_1 --tool aria2c -x 8" ] }, { "cell_type": "markdown", "id": "e93db2b4-5f9f-44c6-8515-b989ca7711a6", "metadata": {}, "source": [ "## 加载模型" ] }, { "cell_type": "code", "execution_count": 8, "id": "4526f56b-6555-4016-aaf2-5598fe46ffca", "metadata": { "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9add942ca8c34a53a3704e439a928497", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading checkpoint shards: 0%| | 0/2 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
promptpositionpreferrednon-preferred
0<s>回覆請少於20字 [INST] 日本動漫真人化是否有損原作形象？ [/INST]support真人化能夠呈現更真實的角色形象，提升原作魅力。真人化可能無法完美呈現動畫中的獨特風格，損害原作形象。
1<s>回覆請少於20字 [INST] 真人化是否能夠擴大動漫在全球的影響力？ [/INST]support真人化能夠讓更多非動漫迷接觸作品，擴大影響力。真人化可能失去動漫的獨特風格，限制影響力擴大。
2<s>回覆請少於20字 [INST] 真人化是否能夠吸引新觀眾？ [/INST]support真人化能夠吸引不熟悉動漫的觀眾，擴大受眾。真人化可能讓原本的動漫迷感到失望，無法吸引新觀眾。
3<s>回覆請少於20字 [INST] 真人化是否能夠保留原作故事情節的精髓？ [/INST]support真人化有機會更深入挖掘原作故事，保留精髓。真人化可能因為改編而失去原作故事的深度與精髓。
4<s>回覆請少於20字 [INST] 真人化是否能夠提升動漫產業的商業價值？ [/INST]support真人化能夠開拓更多商業機會，提升產業價值。真人化可能讓觀眾對原作失去興趣，影響產業價值。
5<s>回覆請少於20字 [INST] 真人化是否能夠保持原作的文化特色？ [/INST]oppose真人化可能因為文化差異而失去原作獨有的文化魅力。真人化可以透過場景、服裝等元素保留文化特色。
6<s>回覆請少於20字 [INST] 真人化是否能夠挑戰技術上的新突破？ [/INST]oppose真人化可能因為技術限制而無法達到動畫中的視覺效果。真人化促使技術創新，挑戰視覺效果上的新高度。
7<s>回覆請少於20字 [INST] 真人化是否會受到演員選擇的爭議？ [/INST]oppose演員選擇可能引起爭議，觀眾難以接受角色塑造。演員選擇可因應市場需求，不必受限於動畫形象。
8<s>回覆請少於20字 [INST] 真人化是否能夠提高動漫的社會認同度？ [/INST]oppose真人化可能因為劇情改編而無法贏得社會認同。真人化有機會讓更多人接受動漫，提高社會認同度。
9<s>回覆請少於20字 [INST] 真人化是否能夠保留原作角色的個性特色？ [/INST]oppose真人化可能因演員演技或導演選擇而失去角色的原有特色。真人化可以透過演員表現保留角色的個性特色。
10<s>回覆請少於20字 [INST] 真人化是否會影響原作動畫的收視率？ [/INST]oppose真人化可能因為原作粉絲的反感而影響動畫收視率。真人化可能吸引更廣泛的觀眾群，提高收視率。
11<s>回覆請少於20字 [INST] 真人化是否會受到版權問題的困擾？ [/INST]oppose版權問題可能成為真人化的阻礙，影響製作進程。版權問題可透過合作與協商解決，不一定成為問題。
12<s>回覆請少於20字 [INST] 真人化是否會引起文化觀念差異的爭議？ [/INST]oppose真人化可能因文化觀念差異引起爭議，影響觀眾接受度。透過適度的改編，真人化可減少文化觀念差異的爭議。
13<s>回覆請少於20字 [INST] 真人化是否能夠挑戰影片表現的新風格？ [/INST]oppose真人化可能受制於原作風格，難以創造新的影片風格。真人化有機會探索新的視覺風格，豐富影片表現。
14<s>回覆請少於20字 [INST] 真人化是否能夠推動動漫產業的多元發展？ [/INST]oppose真人化可能使觀眾對原作產生固有印象，難以多元發展。真人化能夠開啟動漫產業的多元化發展方向。
15<s>回覆請少於20字 [INST] 真人化是否會改變原作動畫的風格？ [/INST]oppose真人化可能難以完美呈現原作動畫獨有的視覺風格。真人化可以透過改編風格，呈現不同的視覺風格。
16<s>回覆請少於20字 [INST] 真人化是否能夠擴大原作的世界觀？ [/INST]oppose真人化可能因製作預算或技術限制無法展現原作的豐富世界觀。真人化有機會透過實景呈現，擴大原作的世界觀。
17<s>回覆請少於20字 [INST] 真人化是否能夠吸引更多投資進入動漫產業？ [/IN...oppose真人化可能因觀眾反感而使投資者對動漫產業失去信心。真人化能夠讓更多投資者看到商機，增加投資動機。
18<s>回覆請少於20字 [INST] 真人化是否能夠促進原作動畫的衍生品銷售？ [/IN...oppose真人化可能因觀眾對原作失望而降低衍生品銷售。真人化能夠擴大原作品牌影響力，提高衍生品銷售。
19<s>回覆請少於20字 [INST] 真人化是否能夠提升動漫的社會地位？ [/INST]oppose真人化可能讓觀眾對動漫產生負面印象，降低社會地位。真人化有機會使動漫更受社會重視，提升地位。
20<s>回覆請少於20字 [INST] 真人化是否會破壞原作動畫的獨特性？ [/INST]oppose真人化可能因為改編而使原作失去獨特風格。真人化可以透過新的呈現方式，保留原作的獨特性。
21<s>回覆請少於20字 [INST] 真人化是否能夠促進動漫產業的國際交流？ [/INST]oppose真人化可能因文化差異而難以打開國際市場，限制交流。真人化能夠使動漫更容易被國際觀眾理解，促進交流。
22<s>回覆請少於20字 [INST] 真人化是否會影響原作動畫的經典地位？ [/INST]oppose真人化可能因改編而被視為是對原作經典的不敬。真人化可以為原作注入新活力，提升經典地位。
23<s>回覆請少於20字 [INST] 真人化是否會受到粉絲的反感？ [/INST]oppose部分粉絲可能因真人化而感到失望，產生強烈反感。部分粉絲可能因真人化而感到新鮮，不一定反感。
24<s>回覆請少於20字 [INST] 真人化是否能夠挖掘原作未探索的劇情？ [/INST]oppose真人化可能因為劇情改編而失去原作的深度與探索價值。真人化有機會透過改編劇情，挖掘原作未探索的內容。
25<s>回覆請少於20字 [INST] 真人化是否能夠吸引更多知名導演與演員參與？ [/I...oppose真人化可能因為風險高而難以吸引知名導演與演員參與。真人化能夠吸引更多知名人士參與，提升製作水平。
26<s>回覆請少於20字 [INST] 真人化是否會受到動畫粉絲的期待壓力？ [/INST]oppose真人化可能因無法滿足動畫粉絲的期待而受到極大壓力。真人化可以透過適度的改編滿足觀眾期待，降低壓力。
27<s>回覆請少於20字 [INST] 真人化是否有助於促進動漫產業內的技術創新？ [/I...oppose過分專注於真人化可能會忽視動漫本身的技術創新和發展。真人化的製作需求推動了視覺效果和製作技術的創新。
28<s>回覆請少於20字 [INST] 真人化是否會提高動漫文化的普及和接受度？ [/IN...oppose真人化可能會降低動漫文化的獨特性，減少其對特定群體的吸引力。真人化有助於使動漫文化更加普及，吸引更多不同背景的觀眾。
29<s>回覆請少於20字 [INST] 真人化是否有助於提升動漫角色的深度和多維度？ [/...oppose真人化經常無法徹底展現動漫角色的複雜性和深度。真人演員的演繹能為動漫角色增添更多層次和深度。
30<s>回覆請少於20字 [INST] 真人化能增加原作的知名度嗎？ [/INST]oppose真人化經常失敗，可能損害原作的聲譽。真人化能擴大原作的觀眾群，提高其知名度。
31<s>回覆請少於20字 [INST] 真人化能夠提高原作的藝術價值嗎？ [/INST]oppose真人化往往無法捕捉原作的藝術魅力，反而貶低其價值。真人化是對原作的新詮釋，能增加其藝術層面的深度。
32<s>回覆請少於20字 [INST] 真人化是否利於動漫產業的經濟發展？ [/INST]oppose真人化的商業化導向可能扭曲原作精神，對產業長期發展不利。真人化能創造新的商業機會，刺激動漫產業的經濟增長。
33<s>回覆請少於20字 [INST] 真人化是否有助於動漫文化的國際傳播？ [/INST]oppose真人化可能會錯誤詮釋動漫文化，造成國際上的誤解。真人化作品更容易被國際觀眾接受，有助於推廣動漫文化。
34<s>回覆請少於20字 [INST] 真人化是否能準確再現動漫的奇幻元素？ [/INST]oppose真人化往往無法完美再現動漫的奇幻世界，失去原作魅力。現代技術能夠精準再現動漫的奇幻元素，增強真人化的吸引力。
35<s>回覆請少於20字 [INST] 真人化是否尊重原作的精神和內容？ [/INST]oppose真人化經常為了商業利益而改變原作的核心精神和內容。真人化時，製作團隊通常會努力保持對原作的忠誠和尊重。
36<s>回覆請少於20字 [INST] 真人化對原作的完整性有何影響？ [/INST]oppose真人化往往簡化或改變原作的故事，損害其完整性。真人化為原作提供了新的視角，豐富了其敘事和背景。
37<s>回覆請少於20字 [INST] 真人化是否有助於吸引更廣泛的觀眾群？ [/INST]oppose真人化可能會失去核心動漫粉絲，而無法吸引新觀眾。真人化能吸引不習慣看動漫的觀眾，擴大受眾基礎。
38<s>回覆請少於20字 [INST] 真人化是否有助於提升原創動漫的品牌價值？ [/IN...oppose不成功的真人化可能損害原創動漫的品牌形象。成功的真人化能提升原創動漫的品牌價值和市場認可。
39<s>回覆請少於20字 [INST] 真人化是否有助於促進原創作者的創意表達？ [/IN...oppose真人化可能限制或扭曲原創者的創意，使其不再忠實於原作。真人化提供了一個新平台，讓原創作者能夠以不同方式表達創意。
40<s>回覆請少於20字 [INST] 真人化是否會增加原作的文化影響力？ [/INST]oppose真人化可能會稀釋原作的獨特文化元素，降低其影響力。真人化作品更容易被主流文化接受，增強原作的文化影響力。
41<s>回覆請少於20字 [INST] 真人化是否有助於原作故事的深度和複雜度？ [/IN...oppose真人化經常因時長和預算限制而簡化故事，減少深度和複雜度。真人化可以加深原作的敘事深度，提供更複雜的角色和情節發展。
42<s>回覆請少於20字 [INST] 真人化是否有助於動漫產業的多元化發展？ [/INST]oppose過度專注於真人化可能會忽視動漫其他領域的發展，限制多元化。真人化作品的多樣性能推動整個動漫產業向更多元化方向發展。
43<s>回覆請少於20字 [INST] 真人化是否有助於提高原作的國際競爭力？ [/INST]oppose真人化如果失敗，可能會損害原作在國際市場的聲譽和競爭力。真人化能使原作更容易進入國際市場，提高其全球競爭力。
44<s>回覆請少於20字 [INST] 真人化是否有助於原創作者的經濟利益？ [/INST]oppose真人化可能會未經原創者充分同意就進行，侵犯其經濟和版權利益。真人化的成功能為原創者帶來額外的經濟收益。
45<s>回覆請少於20字 [INST] 真人化是否有助於促進動漫和其他媒介的合作？ [/I...oppose過分依賴真人化可能會忽視動漫本身的獨立性和特色，限制媒介間的真正合作。真人化能開啟與電影、電視等其他媒介的合作機會。
46<s>回覆請少於20字 [INST] 真人化是否有助於拓寬動漫的故事敘述方式？ [/IN...oppose真人化往往受限於現實條件，無法像動漫那樣自由發揮故事敘述。真人化提供了一種全新的敘述方式，豐富動漫的故事表達。
47<s>回覆請少於20字 [INST] 真人化是否有助於提高動漫角色的真實感？ [/INST]oppose真人化往往無法完全捕捉到動漫角色的獨特魅力和特性。真人演員能賦予動漫角色更多人性化和真實感。
48<s>回覆請少於20字 [INST] 真人化是否能夠正確傳達原作的主題和信息？ [/IN...oppose真人化經常因為格式和風格限制而失去傳達原作主題的能力。真人化能夠通過實際演繹更加生動地傳達原作的主題和信息。
49<s>回覆請少於20字 [INST] 真人化是否有助於增進原創動漫的社會認知度？ [/I...oppose真人化可能會將動漫文化簡化，降低其在社會中的地位和認知度。真人化能使原創動漫更容易獲得主流社會的認可和關注。
\n", "" ], "text/plain": [ " prompt position \\\n", "0 回覆請少於20字 [INST] 日本動漫真人化是否有損原作形象？ [/INST] support \n", "1 回覆請少於20字 [INST] 真人化是否能夠擴大動漫在全球的影響力？ [/INST] support \n", "2 回覆請少於20字 [INST] 真人化是否能夠吸引新觀眾？ [/INST] support \n", "3 回覆請少於20字 [INST] 真人化是否能夠保留原作故事情節的精髓？ [/INST] support \n", "4 回覆請少於20字 [INST] 真人化是否能夠提升動漫產業的商業價值？ [/INST] support \n", "5 回覆請少於20字 [INST] 真人化是否能夠保持原作的文化特色？ [/INST] oppose \n", "6 回覆請少於20字 [INST] 真人化是否能夠挑戰技術上的新突破？ [/INST] oppose \n", "7 回覆請少於20字 [INST] 真人化是否會受到演員選擇的爭議？ [/INST] oppose \n", "8 回覆請少於20字 [INST] 真人化是否能夠提高動漫的社會認同度？ [/INST] oppose \n", "9 回覆請少於20字 [INST] 真人化是否能夠保留原作角色的個性特色？ [/INST] oppose \n", "10 回覆請少於20字 [INST] 真人化是否會影響原作動畫的收視率？ [/INST] oppose \n", "11 回覆請少於20字 [INST] 真人化是否會受到版權問題的困擾？ [/INST] oppose \n", "12 回覆請少於20字 [INST] 真人化是否會引起文化觀念差異的爭議？ [/INST] oppose \n", "13 回覆請少於20字 [INST] 真人化是否能夠挑戰影片表現的新風格？ [/INST] oppose \n", "14 回覆請少於20字 [INST] 真人化是否能夠推動動漫產業的多元發展？ [/INST] oppose \n", "15 回覆請少於20字 [INST] 真人化是否會改變原作動畫的風格？ [/INST] oppose \n", "16 回覆請少於20字 [INST] 真人化是否能夠擴大原作的世界觀？ [/INST] oppose \n", "17 回覆請少於20字 [INST] 真人化是否能夠吸引更多投資進入動漫產業？ [/IN... oppose \n", "18 回覆請少於20字 [INST] 真人化是否能夠促進原作動畫的衍生品銷售？ [/IN... oppose \n", "19 回覆請少於20字 [INST] 真人化是否能夠提升動漫的社會地位？ [/INST] oppose \n", "20 回覆請少於20字 [INST] 真人化是否會破壞原作動畫的獨特性？ [/INST] oppose \n", "21 回覆請少於20字 [INST] 真人化是否能夠促進動漫產業的國際交流？ [/INST] oppose \n", "22 回覆請少於20字 [INST] 真人化是否會影響原作動畫的經典地位？ [/INST] oppose \n", "23 回覆請少於20字 [INST] 真人化是否會受到粉絲的反感？ [/INST] oppose \n", "24 回覆請少於20字 [INST] 真人化是否能夠挖掘原作未探索的劇情？ [/INST] oppose \n", "25 回覆請少於20字 [INST] 真人化是否能夠吸引更多知名導演與演員參與？ [/I... oppose \n", "26 回覆請少於20字 [INST] 真人化是否會受到動畫粉絲的期待壓力？ [/INST] oppose \n", "27 回覆請少於20字 [INST] 真人化是否有助於促進動漫產業內的技術創新？ [/I... oppose \n", "28 回覆請少於20字 [INST] 真人化是否會提高動漫文化的普及和接受度？ [/IN... oppose \n", "29 回覆請少於20字 [INST] 真人化是否有助於提升動漫角色的深度和多維度？ [/... oppose \n", "30 回覆請少於20字 [INST] 真人化能增加原作的知名度嗎？ [/INST] oppose \n", "31 回覆請少於20字 [INST] 真人化能夠提高原作的藝術價值嗎？ [/INST] oppose \n", "32 回覆請少於20字 [INST] 真人化是否利於動漫產業的經濟發展？ [/INST] oppose \n", "33 回覆請少於20字 [INST] 真人化是否有助於動漫文化的國際傳播？ [/INST] oppose \n", "34 回覆請少於20字 [INST] 真人化是否能準確再現動漫的奇幻元素？ [/INST] oppose \n", "35 回覆請少於20字 [INST] 真人化是否尊重原作的精神和內容？ [/INST] oppose \n", "36 回覆請少於20字 [INST] 真人化對原作的完整性有何影響？ [/INST] oppose \n", "37 回覆請少於20字 [INST] 真人化是否有助於吸引更廣泛的觀眾群？ [/INST] oppose \n", "38 回覆請少於20字 [INST] 真人化是否有助於提升原創動漫的品牌價值？ [/IN... oppose \n", "39 回覆請少於20字 [INST] 真人化是否有助於促進原創作者的創意表達？ [/IN... oppose \n", "40 回覆請少於20字 [INST] 真人化是否會增加原作的文化影響力？ [/INST] oppose \n", "41 回覆請少於20字 [INST] 真人化是否有助於原作故事的深度和複雜度？ [/IN... oppose \n", "42 回覆請少於20字 [INST] 真人化是否有助於動漫產業的多元化發展？ [/INST] oppose \n", "43 回覆請少於20字 [INST] 真人化是否有助於提高原作的國際競爭力？ [/INST] oppose \n", "44 回覆請少於20字 [INST] 真人化是否有助於原創作者的經濟利益？ [/INST] oppose \n", "45 回覆請少於20字 [INST] 真人化是否有助於促進動漫和其他媒介的合作？ [/I... oppose \n", "46 回覆請少於20字 [INST] 真人化是否有助於拓寬動漫的故事敘述方式？ [/IN... oppose \n", "47 回覆請少於20字 [INST] 真人化是否有助於提高動漫角色的真實感？ [/INST] oppose \n", "48 回覆請少於20字 [INST] 真人化是否能夠正確傳達原作的主題和信息？ [/IN... oppose \n", "49 回覆請少於20字 [INST] 真人化是否有助於增進原創動漫的社會認知度？ [/I... oppose \n", "\n", " preferred non-preferred \n", "0 真人化能夠呈現更真實的角色形象，提升原作魅力。 真人化可能無法完美呈現動畫中的獨特風格，損害原作形象。 \n", "1 真人化能夠讓更多非動漫迷接觸作品，擴大影響力。 真人化可能失去動漫的獨特風格，限制影響力擴大。 \n", "2 真人化能夠吸引不熟悉動漫的觀眾，擴大受眾。 真人化可能讓原本的動漫迷感到失望，無法吸引新觀眾。 \n", "3 真人化有機會更深入挖掘原作故事，保留精髓。 真人化可能因為改編而失去原作故事的深度與精髓。 \n", "4 真人化能夠開拓更多商業機會，提升產業價值。 真人化可能讓觀眾對原作失去興趣，影響產業價值。 \n", "5 真人化可能因為文化差異而失去原作獨有的文化魅力。 真人化可以透過場景、服裝等元素保留文化特色。 \n", "6 真人化可能因為技術限制而無法達到動畫中的視覺效果。 真人化促使技術創新，挑戰視覺效果上的新高度。 \n", "7 演員選擇可能引起爭議，觀眾難以接受角色塑造。 演員選擇可因應市場需求，不必受限於動畫形象。 \n", "8 真人化可能因為劇情改編而無法贏得社會認同。 真人化有機會讓更多人接受動漫，提高社會認同度。 \n", "9 真人化可能因演員演技或導演選擇而失去角色的原有特色。 真人化可以透過演員表現保留角色的個性特色。 \n", "10 真人化可能因為原作粉絲的反感而影響動畫收視率。 真人化可能吸引更廣泛的觀眾群，提高收視率。 \n", "11 版權問題可能成為真人化的阻礙，影響製作進程。 版權問題可透過合作與協商解決，不一定成為問題。 \n", "12 真人化可能因文化觀念差異引起爭議，影響觀眾接受度。 透過適度的改編，真人化可減少文化觀念差異的爭議。 \n", "13 真人化可能受制於原作風格，難以創造新的影片風格。 真人化有機會探索新的視覺風格，豐富影片表現。 \n", "14 真人化可能使觀眾對原作產生固有印象，難以多元發展。 真人化能夠開啟動漫產業的多元化發展方向。 \n", "15 真人化可能難以完美呈現原作動畫獨有的視覺風格。 真人化可以透過改編風格，呈現不同的視覺風格。 \n", "16 真人化可能因製作預算或技術限制無法展現原作的豐富世界觀。 真人化有機會透過實景呈現，擴大原作的世界觀。 \n", "17 真人化可能因觀眾反感而使投資者對動漫產業失去信心。 真人化能夠讓更多投資者看到商機，增加投資動機。 \n", "18 真人化可能因觀眾對原作失望而降低衍生品銷售。 真人化能夠擴大原作品牌影響力，提高衍生品銷售。 \n", "19 真人化可能讓觀眾對動漫產生負面印象，降低社會地位。 真人化有機會使動漫更受社會重視，提升地位。 \n", "20 真人化可能因為改編而使原作失去獨特風格。 真人化可以透過新的呈現方式，保留原作的獨特性。 \n", "21 真人化可能因文化差異而難以打開國際市場，限制交流。 真人化能夠使動漫更容易被國際觀眾理解，促進交流。 \n", "22 真人化可能因改編而被視為是對原作經典的不敬。 真人化可以為原作注入新活力，提升經典地位。 \n", "23 部分粉絲可能因真人化而感到失望，產生強烈反感。 部分粉絲可能因真人化而感到新鮮，不一定反感。 \n", "24 真人化可能因為劇情改編而失去原作的深度與探索價值。 真人化有機會透過改編劇情，挖掘原作未探索的內容。 \n", "25 真人化可能因為風險高而難以吸引知名導演與演員參與。 真人化能夠吸引更多知名人士參與，提升製作水平。 \n", "26 真人化可能因無法滿足動畫粉絲的期待而受到極大壓力。 真人化可以透過適度的改編滿足觀眾期待，降低壓力。 \n", "27 過分專注於真人化可能會忽視動漫本身的技術創新和發展。 真人化的製作需求推動了視覺效果和製作技術的創新。 \n", "28 真人化可能會降低動漫文化的獨特性，減少其對特定群體的吸引力。 真人化有助於使動漫文化更加普及，吸引更多不同背景的觀眾。 \n", "29 真人化經常無法徹底展現動漫角色的複雜性和深度。 真人演員的演繹能為動漫角色增添更多層次和深度。 \n", "30 真人化經常失敗，可能損害原作的聲譽。 真人化能擴大原作的觀眾群，提高其知名度。 \n", "31 真人化往往無法捕捉原作的藝術魅力，反而貶低其價值。 真人化是對原作的新詮釋，能增加其藝術層面的深度。 \n", "32 真人化的商業化導向可能扭曲原作精神，對產業長期發展不利。 真人化能創造新的商業機會，刺激動漫產業的經濟增長。 \n", "33 真人化可能會錯誤詮釋動漫文化，造成國際上的誤解。 真人化作品更容易被國際觀眾接受，有助於推廣動漫文化。 \n", "34 真人化往往無法完美再現動漫的奇幻世界，失去原作魅力。 現代技術能夠精準再現動漫的奇幻元素，增強真人化的吸引力。 \n", "35 真人化經常為了商業利益而改變原作的核心精神和內容。 真人化時，製作團隊通常會努力保持對原作的忠誠和尊重。 \n", "36 真人化往往簡化或改變原作的故事，損害其完整性。 真人化為原作提供了新的視角，豐富了其敘事和背景。 \n", "37 真人化可能會失去核心動漫粉絲，而無法吸引新觀眾。 真人化能吸引不習慣看動漫的觀眾，擴大受眾基礎。 \n", "38 不成功的真人化可能損害原創動漫的品牌形象。 成功的真人化能提升原創動漫的品牌價值和市場認可。 \n", "39 真人化可能限制或扭曲原創者的創意，使其不再忠實於原作。 真人化提供了一個新平台，讓原創作者能夠以不同方式表達創意。 \n", "40 真人化可能會稀釋原作的獨特文化元素，降低其影響力。 真人化作品更容易被主流文化接受，增強原作的文化影響力。 \n", "41 真人化經常因時長和預算限制而簡化故事，減少深度和複雜度。 真人化可以加深原作的敘事深度，提供更複雜的角色和情節發展。 \n", "42 過度專注於真人化可能會忽視動漫其他領域的發展，限制多元化。 真人化作品的多樣性能推動整個動漫產業向更多元化方向發展。 \n", "43 真人化如果失敗，可能會損害原作在國際市場的聲譽和競爭力。 真人化能使原作更容易進入國際市場，提高其全球競爭力。 \n", "44 真人化可能會未經原創者充分同意就進行，侵犯其經濟和版權利益。 真人化的成功能為原創者帶來額外的經濟收益。 \n", "45 過分依賴真人化可能會忽視動漫本身的獨立性和特色，限制媒介間的真正合作。 真人化能開啟與電影、電視等其他媒介的合作機會。 \n", "46 真人化往往受限於現實條件，無法像動漫那樣自由發揮故事敘述。 真人化提供了一種全新的敘述方式，豐富動漫的故事表達。 \n", "47 真人化往往無法完全捕捉到動漫角色的獨特魅力和特性。 真人演員能賦予動漫角色更多人性化和真實感。 \n", "48 真人化經常因為格式和風格限制而失去傳達原作主題的能力。 真人化能夠通過實際演繹更加生動地傳達原作的主題和信息。 \n", "49 真人化可能會將動漫文化簡化，降低其在社會中的地位和認知度。 真人化能使原創動漫更容易獲得主流社會的認可和關注。 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 选择部分数据用于训练\n", "training_data = full_data[:data_size]\n", "\n", "# 定义 support 数据集的大小，用于将一部分数据标记为“支持” (chosen)，另一部分标记为“反对” (rejected)\n", "support_data_size = int(data_size * support_ratio)\n", "\n", "# 为训练数据集准备数据\n", "prompt_list = [data_formulate(data) for data in training_data]\n", "chosen_list = [data['support'] for data in training_data[:support_data_size]] + [data['oppose'] for data in training_data[support_data_size:]]\n", "rejected_list = [data['oppose'] for data in training_data[:support_data_size]] + [data['support'] for data in training_data[support_data_size:]]\n", "position_list = ['support' for _ in range(support_data_size)] + ['oppose' for _ in range(data_size - support_data_size)]\n", "\n", "# 创建训练数据集\n", "train_dataset = Dataset.from_dict({'prompt': prompt_list, 'position': position_list, 'chosen': chosen_list, 'rejected': rejected_list})\n", "pd.DataFrame(train_dataset).rename(columns={\"chosen\": \"preferred\", \"rejected\": \"non-preferred\"})" ] }, { "cell_type": "markdown", "id": "898c4ac3-4f8c-43ac-878d-0b7a5c39e56c", "metadata": { "tags": [] }, "source": [ "## 训练\n", "\n", "现在，我们进入训练阶段。首先，设置训练参数：" ] }, { "cell_type": "code", "execution_count": 12, "id": "873162f1-71eb-4079-85e4-d0ef2c230a89", "metadata": { "tags": [] }, "outputs": [], "source": [ "training_args = DPOConfig(\n", " output_dir='./',\n", " per_device_train_batch_size=1,\n", " num_train_epochs=num_epoch,\n", " gradient_accumulation_steps=8,\n", " gradient_checkpointing=False,\n", " learning_rate=2e-4,\n", " optim=\"paged_adamw_8bit\",\n", " logging_steps = 1,\n", " warmup_ratio = 0.1,\n", " beta=0.1,\n", " report_to = 'none',\n", " \n", " # 显式声明以避免报错\n", " max_length=512,\n", " remove_unused_columns=False,\n", ")" ] }, { "cell_type": "markdown", "id": "a9a6e9c1-4982-4350-a0e9-0b63a5846c04", "metadata": {}, "source": [ "接下来，配置PEFT（Parameter-Efficient Fine-Tuning）：\n" ] }, { "cell_type": "code", "execution_count": 13, "id": "1c5cefa0-8054-46d7-9e21-3fadfe15aa4e", "metadata": { "tags": [] }, "outputs": [], "source": [ "peft_config = LoraConfig(\n", " lora_alpha=16,\n", " lora_dropout=0.1,\n", " r=64,\n", " bias=\"none\",\n", " task_type=\"CAUSAL_LM\",\n", ")" ] }, { "cell_type": "markdown", "id": "b0cf2d33-1b19-4835-a903-38a9a2d340ec", "metadata": {}, "source": [ "然后，初始化DPO训练器：\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "d734c9e9-bf95-400a-b011-1f265706d97f", "metadata": { "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f6010160361e47f59c587fc6ee9823c6", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Adding EOS to train dataset: 0%| | 0/50 [00:00\n", " \n", " \n", " [7/7 00:14, Epoch 1/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
10.694100
20.687200
30.612000
40.582200
50.515900
60.456700
70.363600

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "TrainOutput(global_step=7, training_loss=0.5588084629603794, metrics={'train_runtime': 18.0081, 'train_samples_per_second': 2.777, 'train_steps_per_second': 0.389, 'total_flos': 203355065843712.0, 'train_loss': 0.5588084629603794})" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dpo_trainer.train()" ] }, { "cell_type": "markdown", "id": "f043c0a6-e173-4c6f-bc6c-08403031b7af", "metadata": { "tags": [] }, "source": [ "## 查看微调后的模型输出" ] }, { "cell_type": "code", "execution_count": 16, "id": "769c45e7-338b-401e-abd0-db440471033a", "metadata": { "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f2f4d792b9df4a989fd3f9df3bb86269", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/10 [00:00