{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4fdab2a6-be55-492e-b563-5f62ccc534af",
   "metadata": {},
   "source": [
    "# a. 使用 Transformers 加载量化后的 LLM 大模型（GPTQ & AWQ） \n",
    "\n",
    "> 引导文章：[19a. 从加载到对话：使用 Transformers 本地运行量化 LLM 大模型（GPTQ & AWQ）](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/19a.%20从加载到对话：使用%20Transformers%20本地运行量化%20LLM%20大模型（GPTQ%20%26%20AWQ）.md)。\n",
    "\n",
    "代码文件没有显卡要求，在个人计算机上均可进行对话。\n",
    "\n",
    "**模型文件约为 4 GB**。\n",
    "\n",
    "这里还有一个简单的 [🎡 AI Chat 脚本](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/CodePlayground/chat.py)供你尝试，详见：[CodePlayground](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/CodePlayground/README.md#当前的玩具)，点击 `►` 或对应的文本展开。\n",
    "\n",
    "Llama-cpp-python 关于 GGUF 文件加载的相关链接：[文章 19b](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/19b.%20从加载到对话：使用%20Llama-cpp-python%20本地运行量化%20LLM%20大模型（GGUF）.md) | [代码文件 16b](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Demos/16b.%20使用%20Llama-cpp-python%20加载量化后的%20LLM%20大模型（GGUF）.ipynb)。\n",
    "\n",
    "在线链接：[Kaggle - a](https://www.kaggle.com/code/aidemos/16a-transformers-llm-gptq) | [Colab - a](https://colab.research.google.com/drive/1cmIDjHriW8aQ5mIsV6ZeTqdnqYe6PoOv?usp=sharing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "468c340d-1694-4a80-bc61-3e97e00e9e42",
   "metadata": {},
   "source": [
    "## 手动下载模型（推荐）\n",
    "\n",
    "来试试多线程指定文件下载，对于 Linux，这里给出配置命令，其余系统可以参照[《a. 使用 HFD 加快 Hugging Face 模型和数据集的下载》](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/a.%20使用%20HFD%20加快%20Hugging%20Face%20模型和数据集的下载.md)先进行环境配置。你也可以跳过这部分，后面会介绍自动下载。\n",
    "\n",
    "```bash\n",
    "sudo apt-get update\n",
    "sudo apt-get install git git-lfs wget aria2\n",
    "git lfs install\n",
    "```\n",
    "\n",
    "下载并配置 HFD 脚本：\n",
    "\n",
    "```bash\n",
    "wget https://huggingface.co/hfd/hfd.sh\n",
    "chmod a+x hfd.sh\n",
    "export HF_ENDPOINT=https://hf-mirror.com\n",
    "```\n",
    "\n",
    "使用多线程下载指定模型。\n",
    "\n",
    "### GPTQ\n",
    "\n",
    "命令遵循 `./hfd.sh <model_path> --tool aria2c -x <线程数>`的格式：\n",
    "\n",
    "```bash\n",
    "./hfd.sh neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit --tool aria2c -x 8\n",
    "```\n",
    "\n",
    "### AWQ\n",
    "\n",
    "命令遵循 `./hfd.sh <model_path> --tool aria2c -x <线程数>`的格式：\n",
    "\n",
    "```bash\n",
    "./hfd.sh solidrust/Mistral-7B-Instruct-v0.3-AWQ --tool aria2c -x 8\n",
    "```\n",
    "\n",
    "### GGUF\n",
    "\n",
    "使用多线程下载指定模型，命令遵循 `./hfd.sh <model_path> --include <file_name> --tool aria2c -x <线程数>`的格式：\n",
    "\n",
    "```bash\n",
    "./hfd.sh bartowski/Mistral-7B-Instruct-v0.3-GGUF --include Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --tool aria2c -x 8\n",
    "```\n",
    "\n",
    "下载完成你应该可以看到类似的输出：\n",
    "\n",
    "```\n",
    "Download Results:\n",
    "gid   |stat|avg speed  |path/URI\n",
    "======+====+===========+=======================================================\n",
    "145eba|OK  |   6.8MiB/s|./Mistral-7B-Instruct-v0.3-Q4_K_M.gguf\n",
    "\n",
    "Status Legend:\n",
    "(OK):download completed.\n",
    "Downloaded https://huggingface.co/bartowski/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf successfully.\n",
    "Download completed successfully.\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9dea4a5-9f06-40c1-b443-8d51b2be20d0",
   "metadata": {},
   "source": [
    "## Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be3f9a99-176d-429c-af1d-63a2cb46aee9",
   "metadata": {},
   "source": [
    "### 环境配置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a085c448-afe2-451b-897c-68ed7ad131d9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: numpy in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (2.2.6)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: pandas in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (2.3.3)\n",
      "Requirement already satisfied: numpy>=1.26.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas) (2.2.6)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas) (2026.1.post1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas) (2026.1)\n",
      "Requirement already satisfied: six>=1.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: transformers in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.56.2)\n",
      "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (3.29.0)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.34.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.36.2)\n",
      "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2.2.6)\n",
      "Requirement already satisfied: packaging>=20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (26.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (6.0.3)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2026.4.4)\n",
      "Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (2.33.1)\n",
      "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.22.2)\n",
      "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (0.7.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers) (4.67.3)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (2025.3.0)\n",
      "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (1.4.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers) (4.15.0)\n",
      "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (3.4.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (3.11)\n",
      "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (2.6.3)\n",
      "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->transformers) (2026.2.25)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: optimum in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (1.27.0)\n",
      "Requirement already satisfied: transformers>=4.29 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from optimum) (4.56.2)\n",
      "Requirement already satisfied: torch>=1.11 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from optimum) (2.7.1)\n",
      "Requirement already satisfied: packaging in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from optimum) (26.1)\n",
      "Requirement already satisfied: numpy in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from optimum) (2.2.6)\n",
      "Requirement already satisfied: huggingface_hub>=0.8.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from optimum) (0.36.2)\n",
      "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (3.29.0)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (2025.3.0)\n",
      "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (1.4.3)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (6.0.3)\n",
      "Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (2.33.1)\n",
      "Requirement already satisfied: tqdm>=4.42.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (4.67.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface_hub>=0.8.0->optimum) (4.15.0)\n",
      "Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (79.0.1)\n",
      "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (1.14.0)\n",
      "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (3.6.1)\n",
      "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (3.1.6)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.6.77)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.6.77)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.6.80)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (9.5.1.17)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.6.4.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (11.3.0.4)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (10.3.7.77)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (11.7.1.2)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.5.4.2)\n",
      "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (0.6.3)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (2.26.2)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.6.77)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (12.6.85)\n",
      "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (1.11.1.6)\n",
      "Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.11->optimum) (3.3.1)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=1.11->optimum) (1.3.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.29->optimum) (2026.4.4)\n",
      "Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.29->optimum) (0.22.2)\n",
      "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.29->optimum) (0.7.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch>=1.11->optimum) (3.0.3)\n",
      "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.8.0->optimum) (3.4.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.8.0->optimum) (3.11)\n",
      "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.8.0->optimum) (2.6.3)\n",
      "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests->huggingface_hub>=0.8.0->optimum) (2026.2.25)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install numpy\n",
    "%pip install pandas\n",
    "%pip install transformers\n",
    "%pip install optimum"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f273e7ad-9ee7-43ed-827d-654d2cb95506",
   "metadata": {},
   "source": [
    "### 关于 autogptq 的迁移\n",
    "\n",
    "需要注意的是，AutoGPTQ 的开发团队在 [24 年 12 月](https://github.com/AutoGPTQ/AutoGPTQ/discussions/758)明确建议所有用户迁移到 GPTQModel。且仓库的 [setup.py](https://github.com/AutoGPTQ/AutoGPTQ/blob/9f7d37072917ab3a7545835f23e808294a542153/setup.py#L11) 文件中包含弃用警告：\"AutoGPTQ 已停止开发，请过渡到 GPTQModel...计划在近期完全从 HuggingFace 框架中弃用 AutoGPTQ\"。\n",
    "\n",
    "所以这里新增一个小模块进行说明。\n",
    "\n",
    "在用法方面，既可以使用[官方仓库](https://github.com/ModelCloud/GPTQModel)所叙述的 GPTQModel.load(...)，也可以直接使用 transformers，详见 [GPTQ - Hugging Face](https://huggingface.co/docs/transformers/quantization/gptq?install=GPTQmodel)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5619732b-590d-4119-a968-64983be6c19f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using pip 26.0.1 from /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages/pip (python 3.12)\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: gptqmodel==4.2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.2.5)\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: logbar in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.4.3)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -v gptqmodel==4.2.5 --no-build-isolation --no-deps # 如果没有预安装，时间可能较长\n",
    "%pip install logbar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16e82aff-a3a9-4ea5-a510-f935628ac045",
   "metadata": {},
   "source": [
    "### GPTQ\n",
    "\n",
    "#### 导入库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4edf71d3-8ea9-4cb7-b5f9-25bd1c2bc8b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARN  Python GIL is enabled: Multi-gpu quant acceleration for MoE models is sub-optimal and multi-core accelerated cpu packing is also disabled. We recommend Python >= 3.13.3t with Pytorch > 2.8 for mult-gpu quantization and multi-cpu packing with env `PYTHON_GIL=0`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARN  Feature `utils/Perplexity` requires python GIL or Python >= 3.13.3T (T for Threading-Free edition of Python) plus Torch 2.8. Feature is currently skipped/disabled.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  ENV: Auto setting PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' for memory saving.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  ENV: Auto setting CUDA_DEVICE_ORDER=PCI_BUS_ID for correctness.          \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "# 设置模型下载镜像（注意，需要在导入 transformers 等模块前进行设置才能起效）\n",
    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
    "\n",
    "import torch\n",
    "from gptqmodel import GPTQModel\n",
    "from transformers import AutoTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c74c5e0-8d78-4800-af07-290302b054fd",
   "metadata": {},
   "source": [
    "下面介绍两种导入模型的方法，实际执行时本地/自动导入二选一。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d0d16b3-1d23-486f-bd32-a60d245d7d87",
   "metadata": {},
   "source": [
    "#### 设置模型路径\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "45027d9a-0f4a-4bf0-8416-9f20fb010825",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 如果你已经配置过了，可以直接在 Notebook 中执行下面的命令下载。\n",
    "# !./hfd.sh neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit --tool aria2c -x 8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e3ae2a7-fd89-4b8a-b7d9-b47a44c613be",
   "metadata": {},
   "source": [
    "如果已经在本地下载了模型，可以通过指定模型路径来加载模型。以下示例假设模型位于当前目录的 `Mistral-7B-Instruct-v0.3-GPTQ-4bit` 文件夹下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "61f7636b-e20d-4a38-880d-8915dd0b0216",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# # 指定本地模型的路径\n",
    "# model_path = \"./Mistral-7B-Instruct-v0.3-GPTQ-4bit\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f97d93f-4c5a-4f6d-b90c-9640e74a3a17",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "如果没有本地模型，设置远程路径（`id` + `/` + `model_name`），导入的时候会自动从 Hugging Face 下载模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "13d0d8bc-3948-4f99-be76-41c8858a7eec",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 指定远程模型的路径\n",
    "model_path = \"neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f5510a5-4d7a-40a5-8859-54bbf2557a4e",
   "metadata": {},
   "source": [
    "#### 加载模型\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d3f05f48-b9a7-48f5-996c-153227b93822",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from_quantized: adapter: None\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "44b5930ea2f24b42935794a80a65ad51",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 9 files:   0%|          | 0/9 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`torch_dtype` is deprecated! Use `dtype` instead!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Loader: Auto dtype (native float16): `torch.float16`                     \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  QuantizeConfig: Ignoring unknown parameter in the quantization configuration: model_name_or_path.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  QuantizeConfig: Ignoring unknown parameter in the quantization configuration: model_file_base_name.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  QuantizeConfig: `checkpoint_format` is missing from the quantization configuration and is automatically inferred to FORMAT.GPTQ\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Estimated Quantization BPW (bits per weight): 4.2875 bpw, based on [bits: 4, group_size: 128]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`torch_dtype` is deprecated! Use `dtype` instead!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO   Kernel: Auto-selection: adding candidate `TritonV2QuantLinear`          \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO   Kernel: Auto-selection: adding candidate `TorchQuantLinear`             \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Kernel: candidates -> `[TritonV2QuantLinear, TorchQuantLinear]`          \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Kernel: selected -> `TritonV2QuantLinear`.                               \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:accelerate.utils.modeling:Some weights of the model checkpoint at /root/.cache/huggingface/hub/models--neuralmagic--Mistral-7B-Instruct-v0.3-GPTQ-4bit/snapshots/ed07c8f1d2c87648508d9e153199d9b761cad5a8/model.safetensors were not used when initializing MistralForCausalLM: {'model.layers.17.self_attn.o_proj.bias', 'model.layers.28.mlp.down_proj.bias', 'model.layers.15.self_attn.k_proj.bias', 'model.layers.5.self_attn.o_proj.bias', 'model.layers.16.mlp.gate_proj.bias', 'model.layers.16.self_attn.k_proj.bias', 'model.layers.26.self_attn.v_proj.bias', 'model.layers.28.self_attn.o_proj.bias', 'model.layers.31.self_attn.v_proj.bias', 'model.layers.28.mlp.gate_proj.bias', 'model.layers.19.self_attn.k_proj.bias', 'model.layers.22.self_attn.v_proj.bias', 'model.layers.8.mlp.up_proj.bias', 'model.layers.24.self_attn.o_proj.bias', 'model.layers.24.mlp.down_proj.bias', 'model.layers.1.self_attn.k_proj.bias', 'model.layers.12.self_attn.k_proj.bias', 'model.layers.21.mlp.gate_proj.bias', 'model.layers.22.self_attn.k_proj.bias', 'model.layers.6.self_attn.q_proj.bias', 'model.layers.25.self_attn.k_proj.bias', 'model.layers.7.self_attn.q_proj.bias', 'model.layers.21.self_attn.q_proj.bias', 'model.layers.7.mlp.up_proj.bias', 'model.layers.13.mlp.up_proj.bias', 'model.layers.13.self_attn.q_proj.bias', 'model.layers.15.mlp.gate_proj.bias', 'model.layers.2.self_attn.q_proj.bias', 'model.layers.22.mlp.down_proj.bias', 'model.layers.5.self_attn.q_proj.bias', 'model.layers.17.self_attn.q_proj.bias', 'model.layers.18.self_attn.v_proj.bias', 'model.layers.21.mlp.up_proj.bias', 'model.layers.18.mlp.down_proj.bias', 'model.layers.0.self_attn.q_proj.bias', 'model.layers.10.self_attn.q_proj.bias', 'model.layers.27.mlp.down_proj.bias', 'model.layers.14.mlp.gate_proj.bias', 'model.layers.31.mlp.up_proj.bias', 'model.layers.17.self_attn.k_proj.bias', 'model.layers.18.mlp.gate_proj.bias', 'model.layers.31.self_attn.q_proj.bias', 'model.layers.3.mlp.up_proj.bias', 'model.layers.8.self_attn.v_proj.bias', 'model.layers.21.self_attn.o_proj.bias', 'model.layers.1.mlp.down_proj.bias', 'model.layers.23.mlp.gate_proj.bias', 'model.layers.24.mlp.gate_proj.bias', 'model.layers.25.self_attn.v_proj.bias', 'model.layers.9.self_attn.k_proj.bias', 'model.layers.9.mlp.down_proj.bias', 'model.layers.20.self_attn.q_proj.bias', 'model.layers.17.mlp.up_proj.bias', 'model.layers.19.mlp.up_proj.bias', 'model.layers.31.mlp.down_proj.bias', 'model.layers.15.self_attn.v_proj.bias', 'model.layers.1.mlp.gate_proj.bias', 'model.layers.2.self_attn.v_proj.bias', 'model.layers.29.mlp.up_proj.bias', 'model.layers.25.mlp.gate_proj.bias', 'model.layers.29.mlp.down_proj.bias', 'model.layers.3.self_attn.k_proj.bias', 'model.layers.31.self_attn.k_proj.bias', 'model.layers.6.self_attn.v_proj.bias', 'model.layers.2.mlp.down_proj.bias', 'model.layers.0.mlp.gate_proj.bias', 'model.layers.2.self_attn.k_proj.bias', 'model.layers.28.self_attn.q_proj.bias', 'model.layers.22.mlp.gate_proj.bias', 'model.layers.4.mlp.up_proj.bias', 'model.layers.11.self_attn.k_proj.bias', 'model.layers.20.mlp.down_proj.bias', 'model.layers.10.mlp.up_proj.bias', 'model.layers.2.self_attn.o_proj.bias', 'model.layers.6.mlp.up_proj.bias', 'model.layers.14.self_attn.k_proj.bias', 'model.layers.28.self_attn.v_proj.bias', 'model.layers.0.self_attn.k_proj.bias', 'model.layers.9.self_attn.v_proj.bias', 'model.layers.26.mlp.down_proj.bias', 'model.layers.3.mlp.down_proj.bias', 'model.layers.14.self_attn.q_proj.bias', 'model.layers.2.mlp.gate_proj.bias', 'model.layers.5.mlp.gate_proj.bias', 'model.layers.20.self_attn.v_proj.bias', 'model.layers.8.mlp.gate_proj.bias', 'model.layers.11.self_attn.q_proj.bias', 'model.layers.23.mlp.up_proj.bias', 'model.layers.11.mlp.gate_proj.bias', 'model.layers.3.self_attn.o_proj.bias', 'model.layers.13.self_attn.k_proj.bias', 'model.layers.27.self_attn.v_proj.bias', 'model.layers.1.self_attn.o_proj.bias', 'model.layers.27.mlp.gate_proj.bias', 'model.layers.30.self_attn.v_proj.bias', 'model.layers.18.self_attn.q_proj.bias', 'model.layers.29.self_attn.o_proj.bias', 'model.layers.31.self_attn.o_proj.bias', 'model.layers.6.self_attn.o_proj.bias', 'model.layers.29.self_attn.k_proj.bias', 'model.layers.8.self_attn.o_proj.bias', 'model.layers.23.self_attn.k_proj.bias', 'model.layers.13.self_attn.v_proj.bias', 'model.layers.21.self_attn.v_proj.bias', 'model.layers.23.self_attn.o_proj.bias', 'model.layers.4.self_attn.o_proj.bias', 'model.layers.7.mlp.gate_proj.bias', 'model.layers.6.mlp.gate_proj.bias', 'model.layers.19.self_attn.v_proj.bias', 'model.layers.8.self_attn.q_proj.bias', 'model.layers.4.self_attn.k_proj.bias', 'model.layers.15.mlp.down_proj.bias', 'model.layers.20.mlp.up_proj.bias', 'model.layers.25.mlp.up_proj.bias', 'model.layers.26.mlp.up_proj.bias', 'model.layers.4.self_attn.v_proj.bias', 'model.layers.27.self_attn.q_proj.bias', 'model.layers.14.self_attn.o_proj.bias', 'model.layers.0.mlp.up_proj.bias', 'model.layers.2.mlp.up_proj.bias', 'model.layers.11.self_attn.v_proj.bias', 'model.layers.10.self_attn.o_proj.bias', 'model.layers.29.self_attn.v_proj.bias', 'model.layers.25.mlp.down_proj.bias', 'model.layers.27.self_attn.k_proj.bias', 'model.layers.30.self_attn.k_proj.bias', 'model.layers.26.self_attn.o_proj.bias', 'model.layers.23.self_attn.q_proj.bias', 'model.layers.14.mlp.up_proj.bias', 'model.layers.13.mlp.gate_proj.bias', 'model.layers.11.mlp.down_proj.bias', 'model.layers.18.mlp.up_proj.bias', 'model.layers.7.mlp.down_proj.bias', 'model.layers.28.self_attn.k_proj.bias', 'model.layers.4.mlp.gate_proj.bias', 'model.layers.10.self_attn.k_proj.bias', 'model.layers.24.self_attn.v_proj.bias', 'model.layers.18.self_attn.k_proj.bias', 'model.layers.25.self_attn.q_proj.bias', 'model.layers.6.mlp.down_proj.bias', 'model.layers.6.self_attn.k_proj.bias', 'model.layers.30.self_attn.q_proj.bias', 'model.layers.17.self_attn.v_proj.bias', 'model.layers.20.self_attn.o_proj.bias', 'model.layers.5.self_attn.v_proj.bias', 'model.layers.17.mlp.gate_proj.bias', 'model.layers.8.mlp.down_proj.bias', 'model.layers.24.self_attn.k_proj.bias', 'model.layers.19.mlp.down_proj.bias', 'model.layers.4.self_attn.q_proj.bias', 'model.layers.9.self_attn.q_proj.bias', 'model.layers.9.self_attn.o_proj.bias', 'model.layers.24.self_attn.q_proj.bias', 'model.layers.30.mlp.down_proj.bias', 'model.layers.18.self_attn.o_proj.bias', 'model.layers.19.mlp.gate_proj.bias', 'model.layers.19.self_attn.q_proj.bias', 'model.layers.22.self_attn.o_proj.bias', 'model.layers.29.self_attn.q_proj.bias', 'model.layers.27.mlp.up_proj.bias', 'model.layers.20.mlp.gate_proj.bias', 'model.layers.3.mlp.gate_proj.bias', 'model.layers.5.mlp.up_proj.bias', 'model.layers.16.self_attn.o_proj.bias', 'model.layers.25.self_attn.o_proj.bias', 'model.layers.30.mlp.up_proj.bias', 'model.layers.24.mlp.up_proj.bias', 'model.layers.14.mlp.down_proj.bias', 'model.layers.15.mlp.up_proj.bias', 'model.layers.0.mlp.down_proj.bias', 'model.layers.26.self_attn.q_proj.bias', 'model.layers.16.self_attn.q_proj.bias', 'model.layers.3.self_attn.q_proj.bias', 'model.layers.8.self_attn.k_proj.bias', 'model.layers.28.mlp.up_proj.bias', 'model.layers.1.mlp.up_proj.bias', 'model.layers.10.mlp.gate_proj.bias', 'model.layers.0.self_attn.o_proj.bias', 'model.layers.12.self_attn.q_proj.bias', 'model.layers.29.mlp.gate_proj.bias', 'model.layers.7.self_attn.k_proj.bias', 'model.layers.20.self_attn.k_proj.bias', 'model.layers.19.self_attn.o_proj.bias', 'model.layers.23.mlp.down_proj.bias', 'model.layers.5.mlp.down_proj.bias', 'model.layers.1.self_attn.v_proj.bias', 'model.layers.30.mlp.gate_proj.bias', 'model.layers.12.mlp.up_proj.bias', 'model.layers.4.mlp.down_proj.bias', 'model.layers.9.mlp.gate_proj.bias', 'model.layers.16.mlp.down_proj.bias', 'model.layers.21.mlp.down_proj.bias', 'model.layers.11.mlp.up_proj.bias', 'model.layers.7.self_attn.o_proj.bias', 'model.layers.15.self_attn.q_proj.bias', 'model.layers.9.mlp.up_proj.bias', 'model.layers.23.self_attn.v_proj.bias', 'model.layers.12.self_attn.v_proj.bias', 'model.layers.17.mlp.down_proj.bias', 'model.layers.30.self_attn.o_proj.bias', 'model.layers.14.self_attn.v_proj.bias', 'model.layers.0.self_attn.v_proj.bias', 'model.layers.13.mlp.down_proj.bias', 'model.layers.5.self_attn.k_proj.bias', 'model.layers.12.mlp.down_proj.bias', 'model.layers.11.self_attn.o_proj.bias', 'model.layers.21.self_attn.k_proj.bias', 'model.layers.1.self_attn.q_proj.bias', 'model.layers.22.mlp.up_proj.bias', 'model.layers.27.self_attn.o_proj.bias', 'model.layers.10.mlp.down_proj.bias', 'model.layers.10.self_attn.v_proj.bias', 'model.layers.31.mlp.gate_proj.bias', 'model.layers.7.self_attn.v_proj.bias', 'model.layers.26.mlp.gate_proj.bias', 'model.layers.16.self_attn.v_proj.bias', 'model.layers.13.self_attn.o_proj.bias', 'model.layers.12.mlp.gate_proj.bias', 'model.layers.15.self_attn.o_proj.bias', 'model.layers.16.mlp.up_proj.bias', 'model.layers.22.self_attn.q_proj.bias', 'model.layers.3.self_attn.v_proj.bias', 'model.layers.26.self_attn.k_proj.bias', 'model.layers.12.self_attn.o_proj.bias'}. This may or may not be an issue - make sure that the checkpoint does not have unnecessary parameters, or that the model definition correctly corresponds to the checkpoint.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Format: Converting `checkpoint_format` from `FORMAT.GPTQ` to internal `FORMAT.GPTQ_V2`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Format: Converting GPTQ v1 to v2                                         \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Format: Conversion complete: 0.029140233993530273s                       \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO   Kernel: Auto-selection: adding candidate `TritonV2QuantLinear`          \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Optimize: `TritonV2QuantLinear` compilation triggered.                   \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tokenicer.tokenicer:Tokenicer: Auto fixed pad_token_id=0 (token='<unk>').\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Model: Loaded `generation_config`: GenerationConfig {\n",
      "  \"bos_token_id\": 1,\n",
      "  \"eos_token_id\": 2\n",
      "}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Model: `generation_config.json` not found. Skipped checking.             \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Kernel: loaded -> `[TritonV2QuantLinear]`                                \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 加载分词器\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_path)\n",
    "\n",
    "# 下载并加载模型\n",
    "model = GPTQModel.load(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44f31a6f-94e4-48bb-a8fd-78eac7b2f3af",
   "metadata": {},
   "source": [
    "#### 推理测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b25eaaa6-59e6-4a6f-a0ed-aa9db261cb7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, World!\n",
      "\n",
      "I’m a software engineer, currently working at Google. I’ve been working in the tech industry for about 10 years now, and I’ve been fortunate enough to work on a variety of interesting\n"
     ]
    }
   ],
   "source": [
    "# 输入文本\n",
    "input_text = \"Hello, World!\"\n",
    "\n",
    "# 将输入文本编码为模型可接受的格式（包含attention_mask）\n",
    "inputs = tokenizer(\n",
    "    input_text, \n",
    "    return_tensors=\"pt\",\n",
    "    return_attention_mask=True  # 明确返回attention_mask\n",
    ").to(model.device)\n",
    "\n",
    "# 生成输出\n",
    "with torch.no_grad():\n",
    "    output_ids = model.generate(\n",
    "        **inputs,  # 传递整个inputs字典，包含input_ids和attention_mask\n",
    "        max_length=50,\n",
    "        pad_token_id=tokenizer.pad_token_id\n",
    "    )\n",
    "\n",
    "# 解码生成的输出\n",
    "output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)\n",
    "\n",
    "# 打印生成的文本\n",
    "print(output_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40c3f269-22f0-485f-b5fd-2d0ebf3a662c",
   "metadata": {},
   "source": [
    "### 关于 autoawq 的迁移\n",
    "\n",
    "目前 AutoAWQ 已经弃用，可使用另一个项目 vLLM 进行替代：https://github.com/vllm-project/llm-compressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8046b56d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 释放之前 GPTQ 演示占用的显存\n",
    "import gc\n",
    "\n",
    "del model\n",
    "del tokenizer\n",
    "gc.collect()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2d1945e5-4d79-4a47-97b0-08dd23ddd2c0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: vllm==0.10.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.10.1)\n",
      "Requirement already satisfied: regex in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2026.4.4)\n",
      "Requirement already satisfied: cachetools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (7.1.2)\n",
      "Requirement already satisfied: psutil in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (7.2.2)\n",
      "Requirement already satisfied: sentencepiece in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.2.1)\n",
      "Requirement already satisfied: numpy in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2.2.6)\n",
      "Requirement already satisfied: requests>=2.26.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2.33.1)\n",
      "Requirement already satisfied: tqdm in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (4.67.3)\n",
      "Requirement already satisfied: blake3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.0.8)\n",
      "Requirement already satisfied: py-cpuinfo in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (9.0.0)\n",
      "Requirement already satisfied: transformers>=4.55.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (4.56.2)\n",
      "Requirement already satisfied: tokenizers>=0.21.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.22.2)\n",
      "Requirement already satisfied: protobuf in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (7.34.1)\n",
      "Requirement already satisfied: fastapi>=0.115.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (0.136.0)\n",
      "Requirement already satisfied: aiohttp in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (3.13.5)\n",
      "Requirement already satisfied: openai>=1.99.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2.32.0)\n",
      "Requirement already satisfied: pydantic>=2.10 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2.12.3)\n",
      "Requirement already satisfied: prometheus_client>=0.18.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.25.0)\n",
      "Requirement already satisfied: pillow in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (12.2.0)\n",
      "Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (7.1.0)\n",
      "Requirement already satisfied: tiktoken>=0.6.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.12.0)\n",
      "Requirement already satisfied: lm-format-enforcer<0.11,>=0.10.11 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.10.12)\n",
      "Requirement already satisfied: llguidance<0.8.0,>=0.7.11 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.7.30)\n",
      "Requirement already satisfied: outlines_core==0.2.10 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.2.10)\n",
      "Requirement already satisfied: diskcache==5.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (5.6.3)\n",
      "Requirement already satisfied: lark==1.2.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.2.2)\n",
      "Requirement already satisfied: xgrammar==0.1.21 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.1.21)\n",
      "Requirement already satisfied: typing_extensions>=4.10 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (4.15.0)\n",
      "Requirement already satisfied: filelock>=3.16.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (3.29.0)\n",
      "Requirement already satisfied: partial-json-parser in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.2.1.1.post7)\n",
      "Requirement already satisfied: pyzmq>=25.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (27.1.0)\n",
      "Requirement already satisfied: msgspec in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.21.1)\n",
      "Requirement already satisfied: gguf>=0.13.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.19.0)\n",
      "Requirement already satisfied: mistral_common>=1.8.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (1.11.2)\n",
      "Requirement already satisfied: opencv-python-headless>=4.11.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (4.13.0.92)\n",
      "Requirement already satisfied: pyyaml in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (6.0.3)\n",
      "Requirement already satisfied: six>=1.16.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.17.0)\n",
      "Requirement already satisfied: setuptools<80,>=77.0.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (79.0.1)\n",
      "Requirement already satisfied: einops in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.8.2)\n",
      "Requirement already satisfied: compressed-tensors==0.10.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.10.2)\n",
      "Requirement already satisfied: depyf==0.19.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.19.0)\n",
      "Requirement already satisfied: cloudpickle in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (3.1.2)\n",
      "Requirement already satisfied: watchfiles in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.1.1)\n",
      "Requirement already satisfied: python-json-logger in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (4.1.0)\n",
      "Requirement already satisfied: scipy in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.17.1)\n",
      "Requirement already satisfied: ninja in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.13.0)\n",
      "Requirement already satisfied: pybase64 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.4.3)\n",
      "Requirement already satisfied: cbor2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (6.1.1)\n",
      "Requirement already satisfied: setproctitle in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (1.3.7)\n",
      "Requirement already satisfied: openai-harmony>=0.0.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.0.8)\n",
      "Requirement already satisfied: numba==0.61.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.61.2)\n",
      "Requirement already satisfied: ray>=2.48.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from ray[cgraph]>=2.48.0->vllm==0.10.1) (2.55.1)\n",
      "Requirement already satisfied: torch==2.7.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2.7.1)\n",
      "Requirement already satisfied: torchaudio==2.7.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (2.7.1)\n",
      "Requirement already satisfied: torchvision==0.22.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.22.1)\n",
      "Requirement already satisfied: xformers==0.0.31 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from vllm==0.10.1) (0.0.31)\n",
      "Requirement already satisfied: astor in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from depyf==0.19.0->vllm==0.10.1) (0.8.1)\n",
      "Requirement already satisfied: dill in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from depyf==0.19.0->vllm==0.10.1) (0.3.8)\n",
      "Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from numba==0.61.2->vllm==0.10.1) (0.44.0)\n",
      "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (1.14.0)\n",
      "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (3.6.1)\n",
      "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (3.1.6)\n",
      "Requirement already satisfied: fsspec in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (2025.3.0)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.6.77)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.6.77)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.6.80)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (9.5.1.17)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.6.4.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (11.3.0.4)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (10.3.7.77)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (11.7.1.2)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.5.4.2)\n",
      "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (0.6.3)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (2.26.2)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.6.77)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (12.6.85)\n",
      "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (1.11.1.6)\n",
      "Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch==2.7.1->vllm==0.10.1) (3.3.1)\n",
      "Requirement already satisfied: interegular>=0.3.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from lm-format-enforcer<0.11,>=0.10.11->vllm==0.10.1) (0.3.3)\n",
      "Requirement already satisfied: packaging in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from lm-format-enforcer<0.11,>=0.10.11->vllm==0.10.1) (26.1)\n",
      "Requirement already satisfied: starlette>=0.46.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.52.1)\n",
      "Requirement already satisfied: typing-inspection>=0.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.4.2)\n",
      "Requirement already satisfied: annotated-doc>=0.0.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.0.4)\n",
      "Requirement already satisfied: fastapi-cli>=0.0.8 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.0.24)\n",
      "Requirement already satisfied: fastar>=0.9.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (0.11.0)\n",
      "Requirement already satisfied: httpx<1.0.0,>=0.23.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (0.28.1)\n",
      "Requirement already satisfied: python-multipart>=0.0.18 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (0.0.26)\n",
      "Requirement already satisfied: email-validator>=2.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (2.3.0)\n",
      "Requirement already satisfied: uvicorn>=0.12.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.44.0)\n",
      "Requirement already satisfied: pydantic-settings>=2.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (2.13.1)\n",
      "Requirement already satisfied: pydantic-extra-types>=2.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi[standard]>=0.115.0->vllm==0.10.1) (2.11.1)\n",
      "Requirement already satisfied: anyio in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (4.13.0)\n",
      "Requirement already satisfied: certifi in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (2026.2.25)\n",
      "Requirement already satisfied: httpcore==1.* in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (1.0.9)\n",
      "Requirement already satisfied: idna in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (3.11)\n",
      "Requirement already satisfied: h11>=0.16 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpcore==1.*->httpx<1.0.0,>=0.23.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.16.0)\n",
      "Requirement already satisfied: dnspython>=2.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from email-validator>=2.0.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (2.8.0)\n",
      "Requirement already satisfied: typer>=0.16.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.24.1)\n",
      "Requirement already satisfied: rich-toolkit>=0.14.8 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.19.9)\n",
      "Requirement already satisfied: fastapi-cloud-cli>=0.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.17.1)\n",
      "Requirement already satisfied: rignore>=0.5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.7.6)\n",
      "Requirement already satisfied: sentry-sdk>=2.20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (2.60.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch==2.7.1->vllm==0.10.1) (3.0.3)\n",
      "Requirement already satisfied: jsonschema>=4.21.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (4.26.0)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pydantic>=2.10->vllm==0.10.1) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.41.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pydantic>=2.10->vllm==0.10.1) (2.41.4)\n",
      "Requirement already satisfied: attrs>=22.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (26.1.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (2025.9.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (0.37.0)\n",
      "Requirement already satisfied: rpds-py>=0.25.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (0.30.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from openai>=1.99.1->vllm==0.10.1) (1.9.0)\n",
      "Requirement already satisfied: jiter<1,>=0.10.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from openai>=1.99.1->vllm==0.10.1) (0.14.0)\n",
      "Requirement already satisfied: sniffio in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from openai>=1.99.1->vllm==0.10.1) (1.3.1)\n",
      "Requirement already satisfied: pycountry>=23 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pydantic-extra-types[pycountry]>=2.10.5->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (26.2.16)\n",
      "Requirement already satisfied: python-dotenv>=0.21.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pydantic-settings>=2.0.0->fastapi[standard]>=0.115.0->vllm==0.10.1) (1.2.2)\n",
      "Requirement already satisfied: click>=7.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from ray>=2.48.0->ray[cgraph]>=2.48.0->vllm==0.10.1) (8.3.2)\n",
      "Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from ray>=2.48.0->ray[cgraph]>=2.48.0->vllm==0.10.1) (1.1.2)\n",
      "Requirement already satisfied: cupy-cuda12x in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from ray[cgraph]>=2.48.0->vllm==0.10.1) (14.0.1)\n",
      "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.26.0->vllm==0.10.1) (3.4.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.26.0->vllm==0.10.1) (2.6.3)\n",
      "Requirement already satisfied: rich>=13.7.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (15.0.0)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (4.0.0)\n",
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (2.20.0)\n",
      "Requirement already satisfied: mdurl~=0.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.1.2)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch==2.7.1->vllm==0.10.1) (1.3.0)\n",
      "Requirement already satisfied: huggingface-hub<2.0,>=0.16.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tokenizers>=0.21.1->vllm==0.10.1) (0.36.2)\n",
      "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.21.1->vllm==0.10.1) (1.4.3)\n",
      "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.55.0->vllm==0.10.1) (0.7.0)\n",
      "Requirement already satisfied: shellingham>=1.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from typer>=0.16.0->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (1.5.4)\n",
      "Requirement already satisfied: httptools>=0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.7.1)\n",
      "Requirement already satisfied: uvloop>=0.15.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (0.22.1)\n",
      "Requirement already satisfied: websockets>=10.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm==0.10.1) (15.0.1)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp->vllm==0.10.1) (2.6.1)\n",
      "Requirement already satisfied: aiosignal>=1.4.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp->vllm==0.10.1) (1.4.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp->vllm==0.10.1) (1.8.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp->vllm==0.10.1) (6.7.1)\n",
      "Requirement already satisfied: propcache>=0.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp->vllm==0.10.1) (0.4.1)\n",
      "Requirement already satisfied: yarl<2.0,>=1.17.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp->vllm==0.10.1) (1.23.0)\n",
      "Requirement already satisfied: cuda-pathfinder==1.*,>=1.3.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from cupy-cuda12x->ray[cgraph]>=2.48.0->vllm==0.10.1) (1.5.4)\n",
      "Requirement already satisfied: soundfile>=0.12.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (0.13.1)\n",
      "Requirement already satisfied: cffi>=1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from soundfile>=0.12.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (2.0.0)\n",
      "Requirement already satisfied: pycparser in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from cffi>=1.0->soundfile>=0.12.1->mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (3.0)\n",
      "Requirement already satisfied: soxr>=0.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from mistral_common>=1.8.2->mistral_common[audio,image]>=1.8.2->vllm==0.10.1) (1.0.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install vllm==0.10.1\n",
    "# 如果还想安装 autoawq，取消下面的注释\n",
    "# %pip install autoawq 'autoawq[kernels]'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c093ae9a-0049-4eb7-995d-78958aaa4004",
   "metadata": {},
   "source": [
    "### AWQ\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58a036ce-ce60-4efe-aa9b-86ef1c1d9d61",
   "metadata": {},
   "source": [
    "#### 导入库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "da97c440-7083-4a09-9203-4d4a4bec7102",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 05-17 23:55:59 [__init__.py:241] Automatically detected platform cuda.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# 设置模型下载镜像（注意，需要在导入 transformers 等模块前进行设置才能起效）\n",
    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
    "# vllm：关掉绕过镜像的 Xet（直连美国 CDN 超时）和可能抖动的hf_transfer\n",
    "os.environ['HF_HUB_ENABLE_HF_TRANSFER'] = '0'\n",
    "os.environ['HF_HUB_DISABLE_XET'] = '1'\n",
    "\n",
    "import torch\n",
    "from vllm import LLM, SamplingParams"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c534d8-3792-41a5-97ee-e7859e2aa094",
   "metadata": {},
   "source": [
    "下面介绍两种导入模型的方法，实际执行时本地/自动导入二选一。\n",
    "\n",
    "#### 设置模型路径\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d7e43cbe-4282-4c05-bf6c-881d14e572a6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# # 如果你已经配置过了，可以直接在 Notebook 中执行下面的命令下载。\n",
    "# !export HF_ENDPOINT=https://hf-mirror.com\n",
    "# !./hfd.sh solidrust/Mistral-7B-Instruct-v0.3-AWQ --tool aria2c -x 8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "275cbbd2-03a1-4246-8b89-ae4b5e7d5363",
   "metadata": {},
   "source": [
    "如果已经在本地下载了模型，可以通过指定模型路径来加载模型。以下示例假设模型位于当前目录的 `Mistral-7B-Instruct-v0.3-AWQ` 文件夹下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6362c6e6-4a72-45a8-a51f-539550798f62",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 指定本地模型的路径\n",
    "# model_path = \"./Mistral-7B-Instruct-v0.3-AWQ\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f136eca1-3f7f-4731-bf79-0ac659b5566d",
   "metadata": {},
   "source": [
    "如果没有本地模型，设置远程路径（`id` + `/` + `model_name`），导入的时候会自动从 Hugging Face 下载模型：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3d5c9108-fa94-4cf1-a26e-3b9e24ce29f0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 指定远程模型的路径\n",
    "model_path = \"solidrust/Mistral-7B-Instruct-v0.3-AWQ\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a752722-eb3f-4e15-aafa-b89ec8cb27d7",
   "metadata": {},
   "source": [
    "#### 加载模型\n",
    "\n",
    "一些权重不会被加载，对于当前任务来说这是预期的行为。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "177e052c-d8bb-41d2-bd0a-8113ec1597b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 05-17 23:56:00 [utils.py:326] non-default args: {'model': 'solidrust/Mistral-7B-Instruct-v0.3-AWQ', 'disable_log_stats': True, 'quantization': 'awq'}\n",
      "INFO 05-17 23:56:14 [__init__.py:711] Resolved architecture: MistralForCausalLM\n",
      "INFO 05-17 23:56:14 [__init__.py:1750] Using max model len 32768\n",
      "INFO 05-17 23:56:16 [awq_marlin.py:121] Detected that the model can run with awq_marlin, however you specified quantization=awq explicitly, so forcing awq. Use quantization=awq_marlin for faster inference\n",
      "WARNING 05-17 23:56:16 [__init__.py:1171] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.\n",
      "INFO 05-17 23:56:18 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192.\n",
      "WARNING 05-17 23:56:21 [__init__.py:2921] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages/transformers/utils/hub.py:111: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO 05-17 23:56:27 [__init__.py:241] Automatically detected platform cuda.\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:31 [core.py:636] Waiting for init message from front-end.\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:31 [core.py:74] Initializing a V1 LLM engine (v0.10.1) with config: model='solidrust/Mistral-7B-Instruct-v0.3-AWQ', speculative_config=None, tokenizer='solidrust/Mistral-7B-Instruct-v0.3-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=solidrust/Mistral-7B-Instruct-v0.3-AWQ, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={\"level\":3,\"debug_dump_path\":\"\",\"cache_dir\":\"\",\"backend\":\"\",\"custom_ops\":[],\"splitting_ops\":[\"vllm.unified_attention\",\"vllm.unified_attention_with_output\",\"vllm.mamba_mixer2\"],\"use_inductor\":true,\"compile_sizes\":[],\"inductor_compile_config\":{\"enable_auto_functionalized_v2\":false},\"inductor_passes\":{},\"cudagraph_mode\":1,\"use_cudagraph\":true,\"cudagraph_num_of_warmups\":1,\"cudagraph_capture_sizes\":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],\"cudagraph_copy_inputs\":false,\"full_cuda_graph\":false,\"pass_config\":{},\"max_capture_size\":512,\"local_cache_dir\":null}\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:34 [parallel_state.py:1134] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m WARNING 05-17 23:56:34 [topk_topp_sampler.py:61] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:34 [gpu_model_runner.py:1953] Starting to load model solidrust/Mistral-7B-Instruct-v0.3-AWQ...\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:34 [gpu_model_runner.py:1985] Loading model from scratch...\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:34 [cuda.py:328] Using Flash Attention backend on V1 engine.\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:35 [weight_utils.py:296] Using model weights format ['*.safetensors']\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:36 [weight_utils.py:312] Time spent downloading weights for solidrust/Mistral-7B-Instruct-v0.3-AWQ: 0.619642 seconds\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:36 [weight_utils.py:349] No model.safetensors.index.json found in remote.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]\n",
      "Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.04it/s]\n",
      "Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.04it/s]\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:37 [default_loader.py:262] Loading weights took 1.15 seconds\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:38 [gpu_model_runner.py:2007] Model loading took 3.8854 GiB and 3.482841 seconds\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:49 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/1c3a2dbb69/rank_0_0/backbone for vLLM's torch.compile\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:49 [backends.py:559] Dynamo bytecode transform time: 10.37 s\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:56 [backends.py:161] Directly load the compiled graph(s) for dynamic shape from the cache, took 6.955 s\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:56:58 [monitor.py:34] torch.compile takes 10.37 s in total\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:57:00 [gpu_worker.py:276] Available KV cache memory: 16.24 GiB\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:57:00 [kv_cache_utils.py:849] GPU KV cache size: 133,040 tokens\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:57:00 [kv_cache_utils.py:853] Maximum concurrency for 32,768 tokens per request: 4.06x\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:07<00:00,  9.56it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:57:07 [gpu_model_runner.py:2708] Graph capturing finished in 7 secs, took 0.76 GiB\n",
      "\u001b[1;36m(EngineCore_0 pid=10924)\u001b[0;0m INFO 05-17 23:57:07 [core.py:214] init engine (profile, create kv cache, warmup model) took 29.43 seconds\n",
      "INFO 05-17 23:57:09 [llm.py:298] Supported_tasks: ['generate']\n"
     ]
    }
   ],
   "source": [
    "# 下载并加载模型（vLLM会自动处理tokenizer）\n",
    "model = LLM(\n",
    "    model=model_path,\n",
    "    quantization=\"awq\",\n",
    "    tensor_parallel_size=1,\n",
    "    gpu_memory_utilization=0.9,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a909e888-42a1-4a3a-954a-c090bf1afdcf",
   "metadata": {},
   "source": [
    "#### 推理测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "72bf0b23-4328-439f-8265-c48f82ae2a2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbb98fcf69ad42b8827d5af6b3d4a915",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0d1a92121a84419b2ced7c423b9109b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "I’m a writer and I’m a nerd. I like stories that take me to other worlds, that challenge me, that inspire me. I love books and movies and TV shows and video games and board games and music.\n"
     ]
    }
   ],
   "source": [
    "# 设置生成参数\n",
    "sampling_params = SamplingParams(\n",
    "    max_tokens=50,\n",
    "    temperature=0.8,\n",
    "    top_p=0.9\n",
    ")\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"Hello, World!\"\n",
    "\n",
    "# 生成输出（vLLM自动处理编码和解码）\n",
    "outputs = model.generate([input_text], sampling_params)\n",
    "\n",
    "# 打印生成的文本\n",
    "output_text = outputs[0].outputs[0].text\n",
    "print(output_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "261f1f2d-6ed3-4caa-b3e4-04e8ac1cd452",
   "metadata": {},
   "source": [
    "### 统一方式加载\n",
    "\n",
    "“要是所有的模型都能统一就好了，这样就不用查阅其他库的文档了。”\n",
    "\n",
    "事实上，我们可以选择使用 transformers 进行二者的加载（如果你对参数设置没有更细致的要求）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3c81a4b-f979-4549-8dbf-86e9126f27e7",
   "metadata": {},
   "source": [
    "#### 下载 autoawq 库（可选）\n",
    "\n",
    "如果想用 transformers 加载，则需要下载 autoawq 库。\n",
    "\n",
    "从 `v0.2.7.post3` 开始 awq 会使用 `flash-attn` 库，但部分 GPU 比如在线平台 Kaggle 和 Colab 提供的 T4 不支持，运行后续代码会报错：`RuntimeError: FlashAttention only supports Ampere GPUs or newer.`\n",
    "\n",
    "故选择下载固定版本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d107fe54-520c-4530-a96f-3f8a8143d1cf",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: autoawq==0.2.7.post2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.2.7.post2)\n",
      "Requirement already satisfied: torch>=2.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (2.7.1)\n",
      "Requirement already satisfied: triton in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (3.3.1)\n",
      "Requirement already satisfied: transformers>=4.35.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (4.56.2)\n",
      "Requirement already satisfied: tokenizers>=0.12.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (0.22.2)\n",
      "Requirement already satisfied: typing-extensions>=4.8.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (4.15.0)\n",
      "Requirement already satisfied: accelerate in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (1.13.0)\n",
      "Requirement already satisfied: datasets>=2.20 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (4.0.0)\n",
      "Requirement already satisfied: zstandard in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq==0.2.7.post2) (0.25.0)\n",
      "Requirement already satisfied: autoawq-kernels in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from autoawq[kernels]==0.2.7.post2) (0.0.9)\n",
      "Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (3.29.0)\n",
      "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (2.2.6)\n",
      "Requirement already satisfied: pyarrow>=15.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (23.0.1)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (0.3.8)\n",
      "Requirement already satisfied: pandas in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (2.3.3)\n",
      "Requirement already satisfied: requests>=2.32.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (2.33.1)\n",
      "Requirement already satisfied: tqdm>=4.66.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (4.67.3)\n",
      "Requirement already satisfied: xxhash in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (3.6.0)\n",
      "Requirement already satisfied: multiprocess<0.70.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (0.70.16)\n",
      "Requirement already satisfied: fsspec<=2025.3.0,>=2023.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (2025.3.0)\n",
      "Requirement already satisfied: huggingface-hub>=0.24.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (0.36.2)\n",
      "Requirement already satisfied: packaging in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (26.1)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from datasets>=2.20->autoawq==0.2.7.post2) (6.0.3)\n",
      "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (3.13.5)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (2.6.1)\n",
      "Requirement already satisfied: aiosignal>=1.4.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (1.4.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (26.1.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (1.8.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (6.7.1)\n",
      "Requirement already satisfied: propcache>=0.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (0.4.1)\n",
      "Requirement already satisfied: yarl<2.0,>=1.17.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (1.23.0)\n",
      "Requirement already satisfied: idna>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from yarl<2.0,>=1.17.0->aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2025.3.0,>=2023.1.0->datasets>=2.20->autoawq==0.2.7.post2) (3.11)\n",
      "Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub>=0.24.0->datasets>=2.20->autoawq==0.2.7.post2) (1.4.3)\n",
      "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=2.20->autoawq==0.2.7.post2) (3.4.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=2.20->autoawq==0.2.7.post2) (2.6.3)\n",
      "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.32.2->datasets>=2.20->autoawq==0.2.7.post2) (2026.2.25)\n",
      "Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (79.0.1)\n",
      "Requirement already satisfied: sympy>=1.13.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (1.14.0)\n",
      "Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (3.6.1)\n",
      "Requirement already satisfied: jinja2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (3.1.6)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.6.77)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.6.77)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.6.80)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (9.5.1.17)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.6.4.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (11.3.0.4)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (10.3.7.77)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (11.7.1.2)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.5.4.2)\n",
      "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (0.6.3)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (2.26.2)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.6.77)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (12.6.85)\n",
      "Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=2.2.0->autoawq==0.2.7.post2) (1.11.1.6)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch>=2.2.0->autoawq==0.2.7.post2) (1.3.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.35.0->autoawq==0.2.7.post2) (2026.4.4)\n",
      "Requirement already satisfied: safetensors>=0.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers>=4.35.0->autoawq==0.2.7.post2) (0.7.0)\n",
      "Requirement already satisfied: psutil in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from accelerate->autoawq==0.2.7.post2) (7.2.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from jinja2->torch>=2.2.0->autoawq==0.2.7.post2) (3.0.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets>=2.20->autoawq==0.2.7.post2) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets>=2.20->autoawq==0.2.7.post2) (2026.1.post1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas->datasets>=2.20->autoawq==0.2.7.post2) (2026.1)\n",
      "Requirement already satisfied: six>=1.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets>=2.20->autoawq==0.2.7.post2) (1.17.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install autoawq==0.2.7.post2 'autoawq[kernels]==0.2.7.post2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6f3c9529-59ea-4888-bf64-66e887048f70",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[rank0]:[W517 23:57:14.191196062 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())\n"
     ]
    }
   ],
   "source": [
    "# 释放之前 AWQ 演示占用的显存\n",
    "import gc\n",
    "\n",
    "del model\n",
    "gc.collect()\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "23672cc5-e408-49c6-b0d9-12be2e06fd86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "加载 GPTQ 模型: neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO   Kernel: Auto-selection: adding candidate `TritonV2QuantLinear`          \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.\n",
      "INFO:accelerate.utils.modeling:We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).\n",
      "Some weights of the model checkpoint at neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit were not used when initializing MistralForCausalLM: ['model.layers.0.mlp.down_proj.bias', 'model.layers.0.mlp.gate_proj.bias', 'model.layers.0.mlp.up_proj.bias', 'model.layers.0.self_attn.k_proj.bias', 'model.layers.0.self_attn.o_proj.bias', 'model.layers.0.self_attn.q_proj.bias', 'model.layers.0.self_attn.v_proj.bias', 'model.layers.1.mlp.down_proj.bias', 'model.layers.1.mlp.gate_proj.bias', 'model.layers.1.mlp.up_proj.bias', 'model.layers.1.self_attn.k_proj.bias', 'model.layers.1.self_attn.o_proj.bias', 'model.layers.1.self_attn.q_proj.bias', 'model.layers.1.self_attn.v_proj.bias', 'model.layers.10.mlp.down_proj.bias', 'model.layers.10.mlp.gate_proj.bias', 'model.layers.10.mlp.up_proj.bias', 'model.layers.10.self_attn.k_proj.bias', 'model.layers.10.self_attn.o_proj.bias', 'model.layers.10.self_attn.q_proj.bias', 'model.layers.10.self_attn.v_proj.bias', 'model.layers.11.mlp.down_proj.bias', 'model.layers.11.mlp.gate_proj.bias', 'model.layers.11.mlp.up_proj.bias', 'model.layers.11.self_attn.k_proj.bias', 'model.layers.11.self_attn.o_proj.bias', 'model.layers.11.self_attn.q_proj.bias', 'model.layers.11.self_attn.v_proj.bias', 'model.layers.12.mlp.down_proj.bias', 'model.layers.12.mlp.gate_proj.bias', 'model.layers.12.mlp.up_proj.bias', 'model.layers.12.self_attn.k_proj.bias', 'model.layers.12.self_attn.o_proj.bias', 'model.layers.12.self_attn.q_proj.bias', 'model.layers.12.self_attn.v_proj.bias', 'model.layers.13.mlp.down_proj.bias', 'model.layers.13.mlp.gate_proj.bias', 'model.layers.13.mlp.up_proj.bias', 'model.layers.13.self_attn.k_proj.bias', 'model.layers.13.self_attn.o_proj.bias', 'model.layers.13.self_attn.q_proj.bias', 'model.layers.13.self_attn.v_proj.bias', 'model.layers.14.mlp.down_proj.bias', 'model.layers.14.mlp.gate_proj.bias', 'model.layers.14.mlp.up_proj.bias', 'model.layers.14.self_attn.k_proj.bias', 'model.layers.14.self_attn.o_proj.bias', 'model.layers.14.self_attn.q_proj.bias', 'model.layers.14.self_attn.v_proj.bias', 'model.layers.15.mlp.down_proj.bias', 'model.layers.15.mlp.gate_proj.bias', 'model.layers.15.mlp.up_proj.bias', 'model.layers.15.self_attn.k_proj.bias', 'model.layers.15.self_attn.o_proj.bias', 'model.layers.15.self_attn.q_proj.bias', 'model.layers.15.self_attn.v_proj.bias', 'model.layers.16.mlp.down_proj.bias', 'model.layers.16.mlp.gate_proj.bias', 'model.layers.16.mlp.up_proj.bias', 'model.layers.16.self_attn.k_proj.bias', 'model.layers.16.self_attn.o_proj.bias', 'model.layers.16.self_attn.q_proj.bias', 'model.layers.16.self_attn.v_proj.bias', 'model.layers.17.mlp.down_proj.bias', 'model.layers.17.mlp.gate_proj.bias', 'model.layers.17.mlp.up_proj.bias', 'model.layers.17.self_attn.k_proj.bias', 'model.layers.17.self_attn.o_proj.bias', 'model.layers.17.self_attn.q_proj.bias', 'model.layers.17.self_attn.v_proj.bias', 'model.layers.18.mlp.down_proj.bias', 'model.layers.18.mlp.gate_proj.bias', 'model.layers.18.mlp.up_proj.bias', 'model.layers.18.self_attn.k_proj.bias', 'model.layers.18.self_attn.o_proj.bias', 'model.layers.18.self_attn.q_proj.bias', 'model.layers.18.self_attn.v_proj.bias', 'model.layers.19.mlp.down_proj.bias', 'model.layers.19.mlp.gate_proj.bias', 'model.layers.19.mlp.up_proj.bias', 'model.layers.19.self_attn.k_proj.bias', 'model.layers.19.self_attn.o_proj.bias', 'model.layers.19.self_attn.q_proj.bias', 'model.layers.19.self_attn.v_proj.bias', 'model.layers.2.mlp.down_proj.bias', 'model.layers.2.mlp.gate_proj.bias', 'model.layers.2.mlp.up_proj.bias', 'model.layers.2.self_attn.k_proj.bias', 'model.layers.2.self_attn.o_proj.bias', 'model.layers.2.self_attn.q_proj.bias', 'model.layers.2.self_attn.v_proj.bias', 'model.layers.20.mlp.down_proj.bias', 'model.layers.20.mlp.gate_proj.bias', 'model.layers.20.mlp.up_proj.bias', 'model.layers.20.self_attn.k_proj.bias', 'model.layers.20.self_attn.o_proj.bias', 'model.layers.20.self_attn.q_proj.bias', 'model.layers.20.self_attn.v_proj.bias', 'model.layers.21.mlp.down_proj.bias', 'model.layers.21.mlp.gate_proj.bias', 'model.layers.21.mlp.up_proj.bias', 'model.layers.21.self_attn.k_proj.bias', 'model.layers.21.self_attn.o_proj.bias', 'model.layers.21.self_attn.q_proj.bias', 'model.layers.21.self_attn.v_proj.bias', 'model.layers.22.mlp.down_proj.bias', 'model.layers.22.mlp.gate_proj.bias', 'model.layers.22.mlp.up_proj.bias', 'model.layers.22.self_attn.k_proj.bias', 'model.layers.22.self_attn.o_proj.bias', 'model.layers.22.self_attn.q_proj.bias', 'model.layers.22.self_attn.v_proj.bias', 'model.layers.23.mlp.down_proj.bias', 'model.layers.23.mlp.gate_proj.bias', 'model.layers.23.mlp.up_proj.bias', 'model.layers.23.self_attn.k_proj.bias', 'model.layers.23.self_attn.o_proj.bias', 'model.layers.23.self_attn.q_proj.bias', 'model.layers.23.self_attn.v_proj.bias', 'model.layers.24.mlp.down_proj.bias', 'model.layers.24.mlp.gate_proj.bias', 'model.layers.24.mlp.up_proj.bias', 'model.layers.24.self_attn.k_proj.bias', 'model.layers.24.self_attn.o_proj.bias', 'model.layers.24.self_attn.q_proj.bias', 'model.layers.24.self_attn.v_proj.bias', 'model.layers.25.mlp.down_proj.bias', 'model.layers.25.mlp.gate_proj.bias', 'model.layers.25.mlp.up_proj.bias', 'model.layers.25.self_attn.k_proj.bias', 'model.layers.25.self_attn.o_proj.bias', 'model.layers.25.self_attn.q_proj.bias', 'model.layers.25.self_attn.v_proj.bias', 'model.layers.26.mlp.down_proj.bias', 'model.layers.26.mlp.gate_proj.bias', 'model.layers.26.mlp.up_proj.bias', 'model.layers.26.self_attn.k_proj.bias', 'model.layers.26.self_attn.o_proj.bias', 'model.layers.26.self_attn.q_proj.bias', 'model.layers.26.self_attn.v_proj.bias', 'model.layers.27.mlp.down_proj.bias', 'model.layers.27.mlp.gate_proj.bias', 'model.layers.27.mlp.up_proj.bias', 'model.layers.27.self_attn.k_proj.bias', 'model.layers.27.self_attn.o_proj.bias', 'model.layers.27.self_attn.q_proj.bias', 'model.layers.27.self_attn.v_proj.bias', 'model.layers.28.mlp.down_proj.bias', 'model.layers.28.mlp.gate_proj.bias', 'model.layers.28.mlp.up_proj.bias', 'model.layers.28.self_attn.k_proj.bias', 'model.layers.28.self_attn.o_proj.bias', 'model.layers.28.self_attn.q_proj.bias', 'model.layers.28.self_attn.v_proj.bias', 'model.layers.29.mlp.down_proj.bias', 'model.layers.29.mlp.gate_proj.bias', 'model.layers.29.mlp.up_proj.bias', 'model.layers.29.self_attn.k_proj.bias', 'model.layers.29.self_attn.o_proj.bias', 'model.layers.29.self_attn.q_proj.bias', 'model.layers.29.self_attn.v_proj.bias', 'model.layers.3.mlp.down_proj.bias', 'model.layers.3.mlp.gate_proj.bias', 'model.layers.3.mlp.up_proj.bias', 'model.layers.3.self_attn.k_proj.bias', 'model.layers.3.self_attn.o_proj.bias', 'model.layers.3.self_attn.q_proj.bias', 'model.layers.3.self_attn.v_proj.bias', 'model.layers.30.mlp.down_proj.bias', 'model.layers.30.mlp.gate_proj.bias', 'model.layers.30.mlp.up_proj.bias', 'model.layers.30.self_attn.k_proj.bias', 'model.layers.30.self_attn.o_proj.bias', 'model.layers.30.self_attn.q_proj.bias', 'model.layers.30.self_attn.v_proj.bias', 'model.layers.31.mlp.down_proj.bias', 'model.layers.31.mlp.gate_proj.bias', 'model.layers.31.mlp.up_proj.bias', 'model.layers.31.self_attn.k_proj.bias', 'model.layers.31.self_attn.o_proj.bias', 'model.layers.31.self_attn.q_proj.bias', 'model.layers.31.self_attn.v_proj.bias', 'model.layers.4.mlp.down_proj.bias', 'model.layers.4.mlp.gate_proj.bias', 'model.layers.4.mlp.up_proj.bias', 'model.layers.4.self_attn.k_proj.bias', 'model.layers.4.self_attn.o_proj.bias', 'model.layers.4.self_attn.q_proj.bias', 'model.layers.4.self_attn.v_proj.bias', 'model.layers.5.mlp.down_proj.bias', 'model.layers.5.mlp.gate_proj.bias', 'model.layers.5.mlp.up_proj.bias', 'model.layers.5.self_attn.k_proj.bias', 'model.layers.5.self_attn.o_proj.bias', 'model.layers.5.self_attn.q_proj.bias', 'model.layers.5.self_attn.v_proj.bias', 'model.layers.6.mlp.down_proj.bias', 'model.layers.6.mlp.gate_proj.bias', 'model.layers.6.mlp.up_proj.bias', 'model.layers.6.self_attn.k_proj.bias', 'model.layers.6.self_attn.o_proj.bias', 'model.layers.6.self_attn.q_proj.bias', 'model.layers.6.self_attn.v_proj.bias', 'model.layers.7.mlp.down_proj.bias', 'model.layers.7.mlp.gate_proj.bias', 'model.layers.7.mlp.up_proj.bias', 'model.layers.7.self_attn.k_proj.bias', 'model.layers.7.self_attn.o_proj.bias', 'model.layers.7.self_attn.q_proj.bias', 'model.layers.7.self_attn.v_proj.bias', 'model.layers.8.mlp.down_proj.bias', 'model.layers.8.mlp.gate_proj.bias', 'model.layers.8.mlp.up_proj.bias', 'model.layers.8.self_attn.k_proj.bias', 'model.layers.8.self_attn.o_proj.bias', 'model.layers.8.self_attn.q_proj.bias', 'model.layers.8.self_attn.v_proj.bias', 'model.layers.9.mlp.down_proj.bias', 'model.layers.9.mlp.gate_proj.bias', 'model.layers.9.mlp.up_proj.bias', 'model.layers.9.self_attn.k_proj.bias', 'model.layers.9.self_attn.o_proj.bias', 'model.layers.9.self_attn.q_proj.bias', 'model.layers.9.self_attn.v_proj.bias']\n",
      "- This IS expected if you are initializing MistralForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing MistralForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Format: Converting `checkpoint_format` from `FORMAT.GPTQ` to internal `FORMAT.GPTQ_V2`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  Format: Conversion complete: 0.011736154556274414s                       \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"margin:0; white-space:pre; font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;\"></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "模型设备: cuda:0\n",
      "模型数据类型: torch.float16\n",
      "输入: Hello, World!\n",
      "生成: \n",
      "\n",
      "# Binary Search Tree Implementation in Rust\n",
      "\n",
      "In this article, we'll explore the implementation of a binary search tree (BST) in Rust. We'll discuss the key properties of a BST, its advantages, and then we'll dive into the code.\n",
      "\n",
      "## What is a Binary Search Tree?\n",
      "\n",
      "A binary search tree (BST) is a tree data structure where each node has at most two children: a left child\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# 设置模型下载镜像\n",
    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
    "\n",
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "def load_quantized_model(model_path):\n",
    "    \"\"\"统一加载GPTQ或AWQ模型的函数\"\"\"\n",
    "    \n",
    "    # 加载分词器\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\n",
    "        model_path,\n",
    "        trust_remote_code=True\n",
    "    )\n",
    "    \n",
    "    # 修复pad_token\n",
    "    if tokenizer.pad_token is None:\n",
    "        tokenizer.pad_token = tokenizer.unk_token or tokenizer.eos_token\n",
    "    \n",
    "    # 加载模型 - transformers会自动检测量化格式\n",
    "    model = AutoModelForCausalLM.from_pretrained(\n",
    "        model_path,\n",
    "        torch_dtype=\"auto\",\n",
    "        device_map=\"auto\",\n",
    "        # attn_implementation=\"flash_attention_2\"  # 可选：使用flash attention\n",
    "    )\n",
    "    \n",
    "    return model, tokenizer\n",
    "\n",
    "# 测试不同的模型\n",
    "models = {\n",
    "    \"GPTQ\": \"neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit\",\n",
    "    \"AWQ\": \"solidrust/Mistral-7B-Instruct-v0.3-AWQ\"\n",
    "}\n",
    "\n",
    "# 选择要测试的模型\n",
    "model_type = \"GPTQ\"  # 或 \"AWQ\"\n",
    "model_path = models[model_type]\n",
    "\n",
    "print(f\"加载 {model_type} 模型: {model_path}\")\n",
    "\n",
    "# 加载模型\n",
    "model, tokenizer = load_quantized_model(model_path)\n",
    "\n",
    "print(f\"模型设备: {model.device}\")\n",
    "print(f\"模型数据类型: {model.dtype}\")\n",
    "\n",
    "# 输入文本\n",
    "input_text = \"Hello, World!\"\n",
    "\n",
    "# 将输入文本编码为模型可接受的格式\n",
    "inputs = tokenizer(\n",
    "    input_text, \n",
    "    return_tensors=\"pt\",\n",
    "    return_attention_mask=True\n",
    ").to(model.device)\n",
    "\n",
    "# 生成输出\n",
    "with torch.no_grad():\n",
    "    output_ids = model.generate(\n",
    "        **inputs,\n",
    "        max_new_tokens=100,  # max_new_tokens 直接等价于生成的 tokens 数量，参数 max_length = len(input) + max_new_tokens\n",
    "        do_sample=True,\n",
    "        temperature=0.7,\n",
    "        top_p=0.9,\n",
    "        pad_token_id=tokenizer.pad_token_id\n",
    "    )\n",
    "\n",
    "# 只解码新生成的部分\n",
    "input_length = inputs['input_ids'].shape[1]\n",
    "generated_text = tokenizer.decode(output_ids[0][input_length:], skip_special_tokens=True)\n",
    "\n",
    "print(f\"输入: {input_text}\")\n",
    "print(f\"生成: {generated_text}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532ea31a-a5d8-47c3-9e7c-5fe6ee5c290f",
   "metadata": {},
   "source": [
    "### 了解提示词模版（prompt_template）\n",
    "\n",
    "其实非常简单，就是曾经提到的占位符（下图对于 `{{question}}` 的应用）。\n",
    "\n",
    "![占位符](../Guide/assets/%E5%8D%A0%E4%BD%8D%E7%AC%A6-6055722.png)\n",
    "\n",
    "举个直观的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "212a83d1-e986-4dae-a7b7-962e0444a466",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "问：人工智能的未来发展方向是什么？\n",
      "答：\n"
     ]
    }
   ],
   "source": [
    "# 定义 Prompt Template\n",
    "prompt_template = \"问：{question}\\n答：\"\n",
    "\n",
    "# 定义问题\n",
    "question = \"人工智能的未来发展方向是什么？\"\n",
    "\n",
    "# 使用 Prompt Template 生成完整的提示\n",
    "prompt = prompt_template.format(question=question)\n",
    "print(prompt)\n",
    "# print(\"\\n\")\n",
    "# print(f\"问：{question}\\n答：\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05446cce-b0fa-4c49-a602-bcae318fd314",
   "metadata": {},
   "source": [
    "#### tokenizer.chat_template\n",
    "\n",
    "查看模型的 `chat_template`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cc212d22-5316-4bf9-bc2d-58212e533b5f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}\n"
     ]
    }
   ],
   "source": [
    "# 打印 chat_template 信息（如果存在的话）\n",
    "if hasattr(tokenizer, 'chat_template'):\n",
    "    print(tokenizer.chat_template)\n",
    "else:\n",
    "    print(\"Tokenizer 没有 'chat_template' 属性。\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe9a3a9-cac7-47ea-890d-430b1ba20fd4",
   "metadata": {},
   "source": [
    "### 流式输出\n",
    "\n",
    "在项目初期认识 API 的时候，文章[《01. 初识 LLM API：环境配置与多轮对话演示》](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/01.%20初识%20LLM%20API：环境配置与多轮对话演示.md#流式输出)有提到过流式输出，这是我们一直以来见到的大模型输出方式：逐字（token）打印而非等全部生成完打印。\n",
    "\n",
    "执行下面的代码试试（无论之前导入的是哪种模型，都可以继续）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "545b6478-3937-46e7-9235-7f91c02bfedd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "人工智能的未来发展方向有以下几个方向：\n",
      "\n",
      "1. 强化学习：强化学习是一种机器学习方法，它通过反馈来学习如何在环境中取得最大的奖励。强化学习已经在游戏中取得了成功，但在更复杂的环境中还有很多挑战需要解决。\n",
      "\n",
      "2. 深度学习：深度学习是一种机器学习方法，它使用多层神经网络来学习复杂的数据表示。深度学习已经在图像识别、语音识别和自然语言处理等领域取得了成\n"
     ]
    }
   ],
   "source": [
    "from transformers import TextStreamer\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# 创建 TextStreamer 实例\n",
    "streamer = TextStreamer(\n",
    "    tokenizer, \n",
    "    skip_prompt=True,         # 在输出时跳过输入的提示部分，仅显示生成的文本\n",
    "    skip_special_tokens=True  # 忽略生成过程中的特殊标记（比如 <pad> / <eos> ...）\n",
    ")\n",
    "\n",
    "# 将提示编码为模型输入\n",
    "input_ids = tokenizer.encode(prompt, return_tensors=\"pt\").to(device)\n",
    "\n",
    "# 设置生成参数\n",
    "generation_kwargs = {\n",
    "    \"input_ids\": input_ids,  # 模型的输入 ID，注意，这不是 Embedding\n",
    "    \"max_length\": 200,       # 生成的最大 token 数\n",
    "    \"streamer\": streamer,    # 使用 TextStreamer 实现生成过程中逐步输出文本\n",
    "    \"pad_token_id\": tokenizer.eos_token_id  # 默认行为，消除 open-end 警告\n",
    "}\n",
    "\n",
    "# 开始生成文本\n",
    "with torch.no_grad():\n",
    "    # ** 是 Python 中的解包操作符，它将字典中的键值对解包为函数的关键字参数。\n",
    "    # 在这里，**generation_kwargs 将字典中的参数逐一传递给 model.generate() 方法，\n",
    "    # 等效于直接写出所有参数：\n",
    "    # model.generate(input_ids=input_ids, max_length=200, do_sample=True, ...)\n",
    "    # 你需要注意到，这和之前采用了不同的传参方式，但本质是一样的。\n",
    "    # 在后续的教程中，会较少地使用这种方式进行传参。\n",
    "    # 因为这很好的分离了参数，所以也增加了乍一看之下的抽象度，为了初见的直观，将减少使用。\n",
    "    model.generate(**generation_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b235a01-1780-4573-b49e-3d889b0d4b76",
   "metadata": {},
   "source": [
    "### 单轮对话\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056aae13-8ba2-4654-8825-db9e97dd17cc",
   "metadata": {},
   "source": [
    "（如果重新启动内核的话，遵循 `导入库`-> `导入模型` -> `当前代码块` 的顺序执行。）\n",
    "\n",
    "让我们直接设计 `messages`，并应用 `chat_template` 进行对话：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "16da6253-3280-4761-aa07-1b6b39336a6e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "User:  人工智能的未来发展方向是什么？\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "人工智能（AI）的未来发展方向有以下几个方面：\n",
      "\n",
      "1. 强化学习：强化学习是一种机器学习方法，它允许机器人在环境中学习如何做出最佳的动作，以最大化长期回报。强化学习已被应用于游戏、自动驾驶车辆、健康保健等领域。\n",
      "2. 深度学习：深度学习是一种机器学习方法，它使用多层神经网络来学习复杂的数据表示。深度学习已被应用于图像识别、语音识别、自然语言处理等领域。\n",
      "3. 智能物联网：智能物联网是一种网络，它将物体和计算机系统联系在一起，以实现自动化、智能化和连接性。智能物联网将在未来几年发展成一个重要的AI应用领域。\n",
      "4. 自然语言处理（NLP）：自然语言处理是一种研究自然语言的机器学习方法。NLP已被应用于语音助手、翻译工具、情感分析等领域。\n",
      "5. 人工智能伦理：随着人工智能技术的发展，人工智能伦理问题也越来越重要。人工智能伦理涉及到人工智能技术的使用、开发和应用的道德、伦理和社会影响问题。\n",
      "6. 人工智能安全：随着人工智能技术的发展，人工智能安全问题也越来越重要。人工智能安全涉及到人工智能系统的安全性、可靠性和可靠性问题\n"
     ]
    }
   ],
   "source": [
    "from transformers import TextStreamer\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# 定义输入\n",
    "prompt = input(\"User: \")\n",
    "\n",
    "# 定义消息列表\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": prompt}\n",
    "]\n",
    "\n",
    "# 使用 tokenizer.apply_chat_template() 生成模型输入\n",
    "input_ids = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(device)\n",
    "\n",
    "# 创建 TextStreamer 实例\n",
    "streamer = TextStreamer(\n",
    "    tokenizer, \n",
    "    skip_prompt=True,         # 在输出时跳过输入的提示部分，仅显示生成的文本\n",
    "    skip_special_tokens=True  # 忽略生成过程中的特殊标记（比如 <pad> / <eos> ...）\n",
    ")\n",
    "\n",
    "# 设置生成参数\n",
    "generation_kwargs = {\n",
    "    \"input_ids\": input_ids,  # 模型的输入 ID，注意，这不是 Embedding\n",
    "    \"max_length\": 500,      # 生成的最大 token 数\n",
    "    \"streamer\": streamer,    # 使用 TextStreamer 实现生成过程中逐步输出文本\n",
    "    \"pad_token_id\": tokenizer.eos_token_id  # 默认行为，消除 open-end 警告\n",
    "}\n",
    "\n",
    "# 开始生成文本\n",
    "with torch.no_grad():\n",
    "    model.generate(**generation_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b0bbef9-d9c7-4840-b6f0-8b042a811a2c",
   "metadata": {},
   "source": [
    "### 多轮对话\n",
    "\n",
    "如果重新启动内核的话，遵循 `导入库`-> `导入模型` -> `当前代码块` 的顺序执行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "dea07ea5-e61d-43a9-abee-6aab5322cb15",
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "User:  如果你是大模型面试官，你会怎么出面试题\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "作为大模型面试官，我会出面试题来测试候选人的知识、思维能力、问题解决能力以及与我们团队合作的能力。以下是一些可能的面试题：\n",
      "\n",
      "1. 解释什么是深度学习，并用一个实际的例子来说明其优势。\n",
      "2. 描述一下你对深度学习框架（如 TensorFlow、PyTorch 等）的了解，并说明你在项目中使用过哪些框架，以及为什么选择了那个框架。\n",
      "3. 解释什么是梯度下降法，并说明其优缺点。\n",
      "4. 描述一下你对卷积神经网络（CNN）和递归神经网络（RNN）的了解，并说明它们在实际应用中的优势和劣势。\n",
      "5. 解释什么是过拟合和欠拟合，并说明如何在训练过程中避免它们。\n",
      "6. 描述一下你对数据增强技术的了解，并说明你在项目中使用过哪些数据增强技术，以及为什么选择了那个技术。\n",
      "7. 描述一下你对模型评估指标的了解，并说明你在项目中使用过哪些指标，以及为什么选择了那个指标。\n",
      "8. 描述一下你对模型优化技术的了解，并说明你在项目中使用过哪些优化技术，以及为什么选择了那个技术。\n",
      "9. 描述一下你对模型部署的了解，并说明你在项目中使用过哪些部署方式，以及为什么选择了那个方式。\n",
      "10. 描述一下你对深度学习的未来发展趋势的看法，并说明\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "User:  对于第十个问题能否给我答案\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "对于第十个问题，我会给出一些可能的答案，但是需要注意的是，深度学习的未来发展趋势可能会因为技术的发展、市场需求、政策等因素而发生变化。以下是一些可能的答案：\n",
      "\n",
      "1. 自然语言处理（NLP）：深度学习已经在语音识别、机器翻译等领域取得了成功，未来可能会在更广泛的领域，如文本生成、情感分析等领域取得更大的成功。\n",
      "2. 图像识别：深度学习已经在图像识别、物体检测等领域取得了成功，未来可能会在更复杂的场景中取得更好的性能，如自动驾驶、医疗诊断等领域。\n",
      "3. 强化学习：强化学习是一种机器学习方法，它可以让机器在环境中学习如何取得最大的奖励。未来可能会在更广泛的领域，如游戏AI、自动驾驶、机器人控制等领域取得更大的成功。\n",
      "4. 生物医学：深度学习已经在生物医学领域取得了成功，如肿瘤分类、病症诊断等。未来可能会在更广泛的生物医学领域取得更大的成功，如药物设计、疾病预测等。\n",
      "5. 物联网（IoT）：物联网已经在各种场景中广泛应用，未来可能会在更复杂的场景中取得更好的性能，如物体识别、预测性维护等。\n",
      "6. 自动化：深度学习已经在各种自动化任务中\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "User:  bye\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Goodbye!\n"
     ]
    }
   ],
   "source": [
    "from transformers import TextStreamer\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# 初始化对话历史\n",
    "messages = []\n",
    "\n",
    "# 开始多轮对话\n",
    "while True:\n",
    "    # 获取输入\n",
    "    prompt = input(\"User: \")\n",
    "    \n",
    "    # 退出对话条件（当然，你也可以直接终止代码块）\n",
    "    if prompt.lower() in [\"exit\", \"quit\", \"bye\"]:\n",
    "        print(\"Goodbye!\")\n",
    "        break\n",
    "    \n",
    "    # 将输入添加到对话历史\n",
    "    messages.append({\"role\": \"user\", \"content\": prompt})\n",
    "    \n",
    "    # 使用 tokenizer.apply_chat_template() 生成模型输入\n",
    "    input_ids = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(device)\n",
    "    \n",
    "    # 创建 TextStreamer 实例\n",
    "    streamer = TextStreamer(\n",
    "        tokenizer, \n",
    "        skip_prompt=True,         # 在输出时跳过输入的提示部分，仅显示生成的文本\n",
    "        skip_special_tokens=True  # 忽略生成过程中的特殊标记（比如 <pad> / <eos> ...）\n",
    "    )\n",
    "    \n",
    "    # 设置生成参数\n",
    "    generation_kwargs = {\n",
    "        \"input_ids\": input_ids,                  # 模型的输入 ID\n",
    "        \"max_length\": input_ids.shape[1] + 500,  # 生成的最大 token 数，input_ids.shape[1] 即输入对应的 tokens 数量\n",
    "        \"streamer\": streamer,                    # 使用 TextStreamer 实现生成过程中逐步输出文本\n",
    "        \"pad_token_id\": tokenizer.eos_token_id   # 默认行为，消除警告\n",
    "    }\n",
    "    \n",
    "    # 开始生成回复\n",
    "    with torch.no_grad():\n",
    "        output_ids = model.generate(**generation_kwargs)\n",
    "    \n",
    "    # 获取生成的回复文本\n",
    "    assistant_reply = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)\n",
    "    \n",
    "    # 将模型的回复添加到对话历史\n",
    "    messages.append({\"role\": \"assistant\", \"content\": assistant_reply})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a66a9f-776d-4d6e-81d5-77c6a60525c0",
   "metadata": {},
   "source": [
    "注意，这里有一个小坑，你不能简单使用 `output_ids[0]` 来保存回复，因为`output_ids` 中实际上包含了 `input_ids`，打印它们：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c31f7cc7-1c07-4715-bbed-386d06ad77a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "如果你是大模型面试官，你会怎么出面试题 作为大模型面试官，我会出面试题来测试候选人的知识、思维能力、问题解决能力以及与我们团队合作的能力。以下是一些可能的面试题：\n",
      "\n",
      "1. 解释什么是深度学习，并用一个实际的例子来说明其优势。\n",
      "2. 描述一下你对深度学习框架（如 TensorFlow、PyTorch 等）的了解，并说明你在项目中使用过哪些框架，以及为什么选择了那个框架。\n",
      "3. 解释什么是梯度下降法，并说明其优缺点。\n",
      "4. 描述一下你对卷积神经网络（CNN）和递归神经网络（RNN）的了解，并说明它们在实际应用中的优势和劣势。\n",
      "5. 解释什么是过拟合和欠拟合，并说明如何在训练过程中避免它们。\n",
      "6. 描述一下你对数据增强技术的了解，并说明你在项目中使用过哪些数据增强技术，以及为什么选择了那个技术。\n",
      "7. 描述一下你对模型评估指标的了解，并说明你在项目中使用过哪些指标，以及为什么选择了那个指标。\n",
      "8. 描述一下你对模型优化技术的了解，并说明你在项目中使用过哪些优化技术，以及为什么选择了那个技术。\n",
      "9. 描述一下你对模型部署的了解，并说明你在项目中使用过哪些部署方式，以及为什么选择了那个方式。\n",
      "10. 描述一下你对深度学习的未来发展趋势的看法，并说明 对于第十个问题能否给我答案 \n",
      "如果你是大模型面试官，你会怎么出面试题 作为大模型面试官，我会出面试题来测试候选人的知识、思维能力、问题解决能力以及与我们团队合作的能力。以下是一些可能的面试题：\n",
      "\n",
      "1. 解释什么是深度学习，并用一个实际的例子来说明其优势。\n",
      "2. 描述一下你对深度学习框架（如 TensorFlow、PyTorch 等）的了解，并说明你在项目中使用过哪些框架，以及为什么选择了那个框架。\n",
      "3. 解释什么是梯度下降法，并说明其优缺点。\n",
      "4. 描述一下你对卷积神经网络（CNN）和递归神经网络（RNN）的了解，并说明它们在实际应用中的优势和劣势。\n",
      "5. 解释什么是过拟合和欠拟合，并说明如何在训练过程中避免它们。\n",
      "6. 描述一下你对数据增强技术的了解，并说明你在项目中使用过哪些数据增强技术，以及为什么选择了那个技术。\n",
      "7. 描述一下你对模型评估指标的了解，并说明你在项目中使用过哪些指标，以及为什么选择了那个指标。\n",
      "8. 描述一下你对模型优化技术的了解，并说明你在项目中使用过哪些优化技术，以及为什么选择了那个技术。\n",
      "9. 描述一下你对模型部署的了解，并说明你在项目中使用过哪些部署方式，以及为什么选择了那个方式。\n",
      "10. 描述一下你对深度学习的未来发展趋势的看法，并说明 对于第十个问题能否给我答案  对于第十个问题，我会给出一些可能的答案，但是需要注意的是，深度学习的未来发展趋势可能会因为技术的发展、市场需求、政策等因素而发生变化。以下是一些可能的答案：\n",
      "\n",
      "1. 自然语言处理（NLP）：深度学习已经在语音识别、机器翻译等领域取得了成功，未来可能会在更广泛的领域，如文本生成、情感分析等领域取得更大的成功。\n",
      "2. 图像识别：深度学习已经在图像识别、物体检测等领域取得了成功，未来可能会在更复杂的场景中取得更好的性能，如自动驾驶、医疗诊断等领域。\n",
      "3. 强化学习：强化学习是一种机器学习方法，它可以让机器在环境中学习如何取得最大的奖励。未来可能会在更广泛的领域，如游戏AI、自动驾驶、机器人控制等领域取得更大的成功。\n",
      "4. 生物医学：深度学习已经在生物医学领域取得了成功，如肿瘤分类、病症诊断等。未来可能会在更广泛的生物医学领域取得更大的成功，如药物设计、疾病预测等。\n",
      "5. 物联网（IoT）：物联网已经在各种场景中广泛应用，未来可能会在更复杂的场景中取得更好的性能，如物体识别、预测性维护等。\n",
      "6. 自动化：深度学习已经在各种自动化任务中\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.decode(input_ids[0], skip_special_tokens=True))\n",
    "print(tokenizer.decode(output_ids[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d32c70b9-bf14-46ad-acc8-8bd493c31a89",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ai",
   "language": "python",
   "name": "ai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}