{
"cells": [
{
"cell_type": "markdown",
"id": "a13831c9-2a09-40e7-98ae-3ee6be8a52cd",
"metadata": {},
"source": [
"# a. 尝试使用 LoRA 微调 Stable Diffusion 模型（文生图）\n",
"\n",
"> [HW10: Stable Diffusion Fine-tuning](https://colab.research.google.com/drive/1dI_-HVggxyIwDVoreymviwg6ZOvEHiLS?usp=sharing#scrollTo=CnJtiRaRuTFX) 中文镜像版\n",
">\n",
"> 这次不是1:1的镜像，我重构了主要的代码逻辑，以便于阅读。对于可以访问原版作业 Colab 的同学来说，一样建议使用下面这份代码进行学习。\n",
">\n",
"> 修改：\n",
"> 1. Colab 中的代码每次训练时先导入了 checkpoint-last 文件夹下的 unet 和 text_encoder，这就意味着每次重新运行是强制 resume 的。\n",
"> 我对这个行为进行了修改，你现在可以通过指定 prepare_lora_model 中的 resume 参数来选择是否接着之前的进行训练。\n",
"> 建议设置为 False，每次重新运行才能对不同参数有直观的感受。\n",
">\n",
"> 2. 原代码中导入了 PEFT 并设置了 LoRA 参数却没有用到，出于学习而非提交作业的目的，我会重新写这一模块。\n",
"> 重写之后的可训练参数层与数量已经经过验证，与 [yahcreeper/GenAI-HW10-Model](https://huggingface.co/yahcreeper/GenAI-HW10-Model/tree/main/checkpoint-last) 完全一致，可放心使用当前版本（关于可训练数量，可以参考[《14. PEFT：在大模型中快速应用 LoRA.md》](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/14.%20PEFT：在大模型中快速应用%20LoRA.md)使用 `print_trainable_parameters()` 进行打印）。\n",
">\n",
"> 3. 会增加注释以便理解。\n",
">\n",
"> 指导文章：[16. 用 LoRA 微调 Stable Diffusion：拆开炼丹炉，动手实现你的第一次 AI 绘画](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/16.%20用%20LoRA%20微调%20Stable%20Diffusion：拆开炼丹炉，动手实现你的第一次%20AI%20绘画.md)\n",
"> \n",
"\n",
"你将微调你自己的 Stable Diffusion 模型，从给定的文本描述生成自定义图像，在初始化项目之后，你将可以在 `SD/Datasets/Brad` 下看到图像和对应的文本描述。\n",
"\n",
"当前代码使用到的数据集 `dataset.zip` 已经放在 Demos/data/14 下面。\n",
"\n",
"你需要注意的是，这份代码倾向于如何微调而非加载数据集或者评估，对于初学者，可以仅关注训练参数和LoRA部分，其余部分的代码细节不用关注。\n",
"\n",
"这次的学习可以在两种版本中二选一，学习效果一致。\n",
"\n",
"[版本 b - 精简版](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Demos/14b.%20尝试使用%20LoRA%20微调%20Stable%20Diffusion%20模型%20-%20精简版.ipynb)\n",
"\n",
"这里还有一个简单的 [🎡 SD LoRA 脚本](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/CodePlayground/sd_lora.py)供你尝试，详见：[CodePlayground](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/CodePlayground/README.md#当前的玩具)，点击 `►` 或对应的文本展开。\n",
"\n",
"在线链接：[Colab](https://colab.research.google.com/drive/1vF7bXHfctyLvaLPjRd0b8kyv-cehUbkj?usp=sharing) | [Kaggle - 精简版](https://www.kaggle.com/code/aidemos/14b-lora-stable-diffusion) | [Colab - 精简版](https://colab.research.google.com/drive/1E7kF00jcjrJMax5iP2DBD86PGysRT8Ux?usp=sharing)"
]
},
{
"cell_type": "markdown",
"id": "d8a8deb6-6844-400c-bb10-388d89d3a020",
"metadata": {},
"source": [
"## 安装必要的库\n",
"\n",
"本单元中将安装一些库用于微调。\n",
"\n",
"安装大约需要 5 分钟。"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "3629cf04-6988-483d-bfc9-509f8b59a9f5",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
"Requirement already satisfied: transformers==4.56.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.56.2)\n",
"Requirement already satisfied: diffusers==0.37.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.37.1)\n",
"Requirement already satisfied: peft==0.17.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.17.1)\n",
"Requirement already satisfied: accelerate==1.13.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (1.13.0)\n",
"Requirement already satisfied: safetensors==0.7.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.7.0)\n",
"Requirement already satisfied: opencv-python in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (4.13.0.92)\n",
"Requirement already satisfied: requests in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (2.33.1)\n",
"Requirement already satisfied: deepface==0.0.100 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.0.100)\n",
"Requirement already satisfied: tf-keras in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (2.21.0)\n",
"Requirement already satisfied: filelock in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (3.29.0)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.34.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (0.36.2)\n",
"Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (2.4.4)\n",
"Requirement already satisfied: packaging>=20.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (26.1)\n",
"Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (6.0.3)\n",
"Requirement already satisfied: regex!=2019.12.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (2026.4.4)\n",
"Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (0.22.2)\n",
"Requirement already satisfied: tqdm>=4.27 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from transformers==4.56.2) (4.67.3)\n",
"Requirement already satisfied: importlib-metadata in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from diffusers==0.37.1) (9.0.0)\n",
"Requirement already satisfied: httpx<1.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from diffusers==0.37.1) (0.28.1)\n",
"Requirement already satisfied: Pillow in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from diffusers==0.37.1) (10.2.0)\n",
"Requirement already satisfied: psutil in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft==0.17.1) (7.2.2)\n",
"Requirement already satisfied: torch>=1.13.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from peft==0.17.1) (2.7.1)\n",
"Requirement already satisfied: pandas>=0.23.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (2.3.3)\n",
"Requirement already satisfied: gdown>=3.10.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (6.0.0)\n",
"Requirement already satisfied: tensorflow>=1.9.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (2.21.0)\n",
"Requirement already satisfied: keras>=2.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (3.14.1)\n",
"Requirement already satisfied: Flask>=1.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (3.1.3)\n",
"Requirement already satisfied: flask-cors>=4.0.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (6.0.2)\n",
"Requirement already satisfied: mtcnn>=0.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (1.0.0)\n",
"Requirement already satisfied: retina-face>=0.0.14 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (0.0.17)\n",
"Requirement already satisfied: fire>=0.4.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (0.7.1)\n",
"Requirement already satisfied: gunicorn>=20.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (26.0.0)\n",
"Requirement already satisfied: lightphe>=0.0.15 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (0.0.25)\n",
"Requirement already satisfied: lightdsa>=0.0.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (0.0.3)\n",
"Requirement already satisfied: python-dotenv>=1.0.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from deepface==0.0.100) (1.2.2)\n",
"Requirement already satisfied: anyio in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0->diffusers==0.37.1) (4.13.0)\n",
"Requirement already satisfied: certifi in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0->diffusers==0.37.1) (2026.2.25)\n",
"Requirement already satisfied: httpcore==1.* in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0->diffusers==0.37.1) (1.0.9)\n",
"Requirement already satisfied: idna in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpx<1.0.0->diffusers==0.37.1) (3.11)\n",
"Requirement already satisfied: h11>=0.16 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from httpcore==1.*->httpx<1.0.0->diffusers==0.37.1) (0.16.0)\n",
"Requirement already satisfied: fsspec>=2023.5.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers==4.56.2) (2025.3.0)\n",
"Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers==4.56.2) (1.4.3)\n",
"Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.34.0->transformers==4.56.2) (4.15.0)\n",
"Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests) (3.4.7)\n",
"Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests) (2.6.3)\n",
"Requirement already satisfied: absl-py>=1.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (2.4.0)\n",
"Requirement already satisfied: astunparse>=1.6.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (1.6.3)\n",
"Requirement already satisfied: flatbuffers>=25.9.23 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (25.12.19)\n",
"Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (0.7.0)\n",
"Requirement already satisfied: google_pasta>=0.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (0.2.0)\n",
"Requirement already satisfied: libclang>=13.0.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (18.1.1)\n",
"Requirement already satisfied: opt_einsum>=2.3.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (3.4.0)\n",
"Requirement already satisfied: protobuf<8.0.0,>=6.31.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (7.34.1)\n",
"Requirement already satisfied: setuptools in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (82.0.1)\n",
"Requirement already satisfied: six>=1.12.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (1.17.0)\n",
"Requirement already satisfied: termcolor>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (3.3.0)\n",
"Requirement already satisfied: wrapt>=1.11.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (2.1.2)\n",
"Requirement already satisfied: grpcio<2.0,>=1.24.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (1.80.0)\n",
"Requirement already satisfied: h5py<3.15.0,>=3.11.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (3.14.0)\n",
"Requirement already satisfied: ml_dtypes<1.0.0,>=0.5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from tensorflow>=1.9.0->deepface==0.0.100) (0.5.4)\n",
"Requirement already satisfied: wheel<1.0,>=0.23.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from astunparse>=1.6.0->tensorflow>=1.9.0->deepface==0.0.100) (0.47.0)\n",
"Requirement already satisfied: blinker>=1.9.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from Flask>=1.1.2->deepface==0.0.100) (1.9.0)\n",
"Requirement already satisfied: click>=8.1.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from Flask>=1.1.2->deepface==0.0.100) (8.3.2)\n",
"Requirement already satisfied: itsdangerous>=2.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from Flask>=1.1.2->deepface==0.0.100) (2.2.0)\n",
"Requirement already satisfied: jinja2>=3.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from Flask>=1.1.2->deepface==0.0.100) (3.1.6)\n",
"Requirement already satisfied: markupsafe>=2.1.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from Flask>=1.1.2->deepface==0.0.100) (3.0.3)\n",
"Requirement already satisfied: werkzeug>=3.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from Flask>=1.1.2->deepface==0.0.100) (3.1.8)\n",
"Requirement already satisfied: beautifulsoup4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from gdown>=3.10.1->deepface==0.0.100) (4.14.3)\n",
"Requirement already satisfied: rich in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from keras>=2.2.0->deepface==0.0.100) (15.0.0)\n",
"Requirement already satisfied: namex in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from keras>=2.2.0->deepface==0.0.100) (0.1.0)\n",
"Requirement already satisfied: optree in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from keras>=2.2.0->deepface==0.0.100) (0.19.1)\n",
"Requirement already satisfied: lightecc in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from lightdsa>=0.0.3->deepface==0.0.100) (0.0.6)\n",
"Requirement already satisfied: sympy in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from lightdsa>=0.0.3->deepface==0.0.100) (1.14.0)\n",
"Requirement already satisfied: joblib>=1.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from mtcnn>=0.1.0->deepface==0.0.100) (1.5.3)\n",
"Requirement already satisfied: lz4>=4.3.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from mtcnn>=0.1.0->deepface==0.0.100) (4.4.5)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas>=0.23.4->deepface==0.0.100) (2.9.0.post0)\n",
"Requirement already satisfied: pytz>=2020.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas>=0.23.4->deepface==0.0.100) (2026.1.post1)\n",
"Requirement already satisfied: tzdata>=2022.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from pandas>=0.23.4->deepface==0.0.100) (2026.1)\n",
"Requirement already satisfied: mpmath<1.4,>=1.1.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from sympy->lightdsa>=0.0.3->deepface==0.0.100) (1.3.0)\n",
"Requirement already satisfied: networkx in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (3.6.1)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.6.77)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.6.77)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.6.80)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==9.5.1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (9.5.1.17)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.6.4.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (11.3.0.4)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (10.3.7.77)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (11.7.1.2)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.5.4.2)\n",
"Requirement already satisfied: nvidia-cusparselt-cu12==0.6.3 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (0.6.3)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.26.2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (2.26.2)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.6.77)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (12.6.85)\n",
"Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (1.11.1.6)\n",
"Requirement already satisfied: triton==3.3.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from torch>=1.13.0->peft==0.17.1) (3.3.1)\n",
"Requirement already satisfied: soupsieve>=1.6.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from beautifulsoup4->gdown>=3.10.1->deepface==0.0.100) (2.8.3)\n",
"Requirement already satisfied: zipp>=3.20 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from importlib-metadata->diffusers==0.37.1) (3.23.1)\n",
"Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests[socks]->gdown>=3.10.1->deepface==0.0.100) (1.7.1)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from rich->keras>=2.2.0->deepface==0.0.100) (4.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from rich->keras>=2.2.0->deepface==0.0.100) (2.20.0)\n",
"Requirement already satisfied: mdurl~=0.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from markdown-it-py>=2.2.0->rich->keras>=2.2.0->deepface==0.0.100) (0.1.2)\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install \\\n",
" \"transformers==4.56.2\" \\\n",
" \"diffusers==0.37.1\" \\\n",
" \"peft==0.17.1\" \\\n",
" \"accelerate==1.13.0\" \\\n",
" \"safetensors==0.7.0\" \\\n",
" \"opencv-python\" \\\n",
" \"requests\" \\\n",
" \"deepface==0.0.100\" \\\n",
" \"tf-keras\""
]
},
{
"cell_type": "markdown",
"id": "f5a92e17-0911-4f29-9bc1-6f81e1dc287d",
"metadata": {},
"source": [
"## 导入"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8bd2c745-a37a-42d8-940c-2cdcbc784b0f",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
"I0000 00:00:1780047995.019260 17486 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
"I0000 00:00:1780047995.077906 17486 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
"To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
"I0000 00:00:1780047996.473858 17486 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n"
]
}
],
"source": [
"# ========== 标准库模块 ==========\n",
"import os\n",
"# 设置模型下载镜像\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"os.environ['TF_USE_LEGACY_KERAS'] = '1' # 让 tf 2.16+ 走 Keras 2 兼容层，deepface 需要\n",
"\n",
"try:\n",
" import tensorflow as _tf\n",
" _tf.config.set_visible_devices([], \"GPU\")\n",
"except Exception:\n",
" pass\n",
"\n",
"\n",
"import math\n",
"import random\n",
"import glob\n",
"import shutil\n",
"import argparse\n",
"from pathlib import Path\n",
"import subprocess\n",
"\n",
"# ========== 第三方库 ==========\n",
"import numpy as np\n",
"import torch\n",
"import torch.nn.functional as F\n",
"from PIL import Image\n",
"from tqdm.auto import tqdm\n",
"\n",
"# ========== 深度学习相关库 ==========\n",
"from torchvision import transforms\n",
"from torchvision.utils import save_image\n",
"\n",
"# Transformers (Hugging Face)\n",
"from transformers import AutoProcessor, AutoModel, CLIPTextModel, CLIPTokenizer\n",
"\n",
"# Diffusers (Hugging Face)\n",
"from diffusers import (\n",
" AutoencoderKL,\n",
" DDPMScheduler,\n",
" DiffusionPipeline,\n",
" StableDiffusionPipeline,\n",
" UNet2DConditionModel\n",
")\n",
"from diffusers.optimization import get_scheduler\n",
"from diffusers.utils import convert_state_dict_to_diffusers\n",
"from diffusers.training_utils import compute_snr\n",
"from diffusers.utils.torch_utils import is_compiled_module\n",
"\n",
"# ========== LoRA 模型库 ==========\n",
"from peft import LoraConfig, get_peft_model\n",
"\n",
"# ========== 面部检测库 ==========\n",
"from deepface import DeepFace\n",
"\n",
"# ========== IPython 和 Widgets ==========\n",
"import ipywidgets as widgets\n",
"from IPython.display import display, Markdown, clear_output\n",
"\n",
"import cv2"
]
},
{
"cell_type": "markdown",
"id": "62438d83-b6c5-43a3-afe9-30ceebc50d46",
"metadata": {},
"source": [
"## 准备项目\n",
"\n",
"直接运行代码，这个模块不用修改参数，不用关心这里的代码细节，除非你对交互感兴趣。\n",
"\n",
"注意：记得点击**初始化项目**\n",
"\n",
"当你看见✅时，代表项目已经准备好。"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "7fbde642-16f6-4728-8b7b-05d220644de0",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "784b96502b354ee0be909ad846f58fb2",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Text(value='Brad', description='项目名称:', style=TextStyle(description_width='initial'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "36196e5a2f9a4f2791389df89765e3ed",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Text(value='Brad', description='数据集:', disabled=True, style=TextStyle(description_width='initial'))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "481b7139fb8042848db4bff79ae79d8e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Button(description='初始化项目', style=ButtonStyle())"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b008b14f26164bbb9d438b340aaaa1de",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# 创建项目名称输入框\n",
"project_name_widget = widgets.Text(\n",
" value=\"Brad\",\n",
" description=\"项目名称:\",\n",
" style={'description_width': 'initial'}\n",
")\n",
"\n",
"# 创建数据集名称输入框\n",
"dataset_name_widget = widgets.Text(\n",
" value=\"Brad\", # \"Brad-512\", \"Anne-512\"\n",
" description=\"数据集:\",\n",
" style={'description_width': 'initial'},\n",
" disabled=True\n",
")\n",
"\n",
"# 创建提交按钮\n",
"submit_button = widgets.Button(description=\"初始化项目\")\n",
"\n",
"# 输出区域\n",
"output_area = widgets.Output()\n",
"\n",
"# 定义项目初始化逻辑\n",
"def initialize_project(b):\n",
" global project_name, dataset_name, root_dir, main_dir, project_dir, model_path, images_folder, prompts_folder, captions_folder\n",
" with output_area:\n",
" clear_output()\n",
" project_name = project_name_widget.value.strip()\n",
" dataset_name = dataset_name_widget.value.strip()\n",
" \n",
" if not project_name or any(c in project_name for c in \" .()\\\"'\\\\\") or project_name.count(\"/\") > 1:\n",
" print(\"请输入有效的项目名称。\")\n",
" else:\n",
" # 创建项目目录\n",
" project_base = project_name if \"/\" not in project_name else project_name[:project_name.rfind(\"/\")]\n",
" project_subfolder = project_name if \"/\" not in project_name else project_name[project_name.rfind(\"/\")+1:]\n",
" root_dir = \"./\" # 当前目录\n",
" main_dir = os.path.join(root_dir, \"SD\") # 主目录\n",
" project_dir = os.path.join(main_dir, project_name) # 项目目录\n",
" \n",
" # 确保目录存在\n",
" os.makedirs(main_dir, exist_ok=True)\n",
" os.makedirs(project_dir, exist_ok=True)\n",
" \n",
" # 定义数据集和模型路径\n",
" zip_file = os.path.join(\"./\", \"data/14/Datasets.zip\")\n",
" model_path = os.path.join(project_dir, \"logs\", \"checkpoint-last\")\n",
" images_folder = os.path.join(main_dir, \"Datasets\", dataset_name)\n",
" prompts_folder = os.path.join(main_dir, \"Datasets\", \"prompts\")\n",
" captions_folder = images_folder\n",
"\n",
" # 检查并解压数据集\n",
" print(\"📂 正在检查并解压样例数据集...\")\n",
" \n",
" if not os.path.exists(zip_file):\n",
" os.makedirs(os.path.dirname(zip_file), exist_ok=True)\n",
" print(\"📥 数据集 Datasets.zip 未找到，开始下载…\")\n",
" subprocess.run([\n",
" \"wget\", \"-nc\", \"-O\", zip_file,\n",
" \"https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/raw/master/Demos/data/14/Datasets.zip\"\n",
" ], check=True)\n",
"\n",
" subprocess.run(f\"unzip -q -o {zip_file} -d {main_dir}\", shell=True)\n",
"\n",
" # # 克隆模型文件\n",
" print(\"📂 默认不克隆LoRA模型文件而是自己训练，因为下载需要占用2G且等价于你先进行1000个step训练。\\n\")\n",
"\n",
" # 创建必要的文件夹\n",
" os.makedirs(images_folder, exist_ok=True)\n",
"\n",
" print(f\"✅ 项目 {project_name} 已准备好！\")\n",
"\n",
"# 将提交按钮与项目初始化逻辑绑定\n",
"submit_button.on_click(initialize_project)\n",
"\n",
"# 显示控件\n",
"display(project_name_widget, dataset_name_widget, submit_button, output_area)"
]
},
{
"cell_type": "markdown",
"id": "b44210b9-245d-4640-ae1c-d4a55420c984",
"metadata": {},
"source": [
"## 设置参数"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "f253cc2c-42eb-476d-aa9b-a740c432caf0",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# 请勿随意更改以下参数，除非你自己知道改的参数对应的什么，否则可能因 GPU 内存不足导致进程崩溃。\n",
"output_folder = os.path.join(project_dir, \"logs\") # 存放 model checkpoints 和 validation 的文件夹\n",
"seed = 1126 # 随机数种子\n",
"train_batch_size = 2 # 训练批次大小，即每次训练中处理的样本数量\n",
"resolution = 512 # 训练图像的分辨率\n",
"weight_dtype = torch.bfloat16 # 权重数据类型，使用 bfloat16 以节省内存并加快计算速度\n",
"snr_gamma = 5 # SNR 参数，用于信噪比加权损失的调节系数"
]
},
{
"cell_type": "markdown",
"id": "255ee76f-e838-4296-bf6d-fe175d9e08b8",
"metadata": {},
"source": [
"### Stable Diffusion LoRA 的微调参数"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a07a3378-e074-4793-ac5e-f8ba7bab2056",
"metadata": {},
"outputs": [],
"source": [
"# 预训练的 Stable Diffusion 模型路径，用于加载模型进行微调\n",
"pretrained_model_name_or_path = \"stablediffusionapi/cyberrealistic-v42\"\n",
"\n",
"# LoRA 配置\n",
"lora_config = LoraConfig(\n",
" r=32, # LoRA 的秩，即低秩矩阵的维度，决定了参数调整的自由度\n",
" lora_alpha=16, # 缩放系数，控制 LoRA 权重对模型的影响\n",
" target_modules=[\n",
" \"q_proj\", \"v_proj\", \"k_proj\", \"out_proj\", # 指定 Text encoder 的 LoRA 应用对象（用于调整注意力机制中的投影矩阵）\n",
" \"to_k\", \"to_q\", \"to_v\", \"to_out.0\" # 指定 UNet 的 LoRA 应用对象（用于调整 UNet 中的注意力机制）\n",
" ],\n",
" lora_dropout=0 # LoRA dropout 概率，0 表示不使用 dropout\n",
")"
]
},
{
"cell_type": "markdown",
"id": "806d27ec-5425-454a-bb6c-506db30fe69c",
"metadata": {},
"source": [
"### 其他设置"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e1b00988-3201-4c16-b363-17528f5cadb4",
"metadata": {},
"outputs": [],
"source": [
"# 学习率调度器部分\n",
"lr_scheduler_name = \"cosine_with_restarts\" # 设置学习率的调度器\n",
"lr_warmup_steps = 100 # 设置学习率预热步数\n",
"\n",
"# prompt 处理\n",
"validation_prompt_name = \"validation_prompt.txt\"\n",
"validation_prompt_path = os.path.join(prompts_folder, validation_prompt_name)\n",
"with open(validation_prompt_path, \"r\") as f:\n",
" validation_prompt = [line.strip() for line in f.readlines()]"
]
},
{
"cell_type": "markdown",
"id": "a74e1b82-00a7-4e4e-92cb-8aaf50aa2a60",
"metadata": {},
"source": [
"直接运行代码，交互式修改参数，不用关心这里的代码细节，除非你对交互感兴趣。"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a70fdd8b-3e5b-4657-bacd-206b087375b1",
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"▶️ **Learning Rate**
学习率是影响结果最重要的参数，当前默认三者一致。
如果你想慢速训练且有大量图像，或者如果 LoRA rank（有些文生图UI可能用的是dim） 和 alpha 值较高，建议将 unet 的学习率调至 5e-5 或更低。
需要注意的是，实际上learning_rate在这里不起作用，我们只使用对应模块的学习率。
这里进行展示是为了让你之后在炼丹炉上能够更好的理解设置。"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "58f981205861434fa8c1ed1b17293402",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatText(value=0.0001, description='learning_rate:', disabled=True, layout=Layout(width='300px'), style=Descr…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c10ee48a7542415abe1cf5f2d114697f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatText(value=0.0001, description='unet_learning_rate:', layout=Layout(width='300px'), style=DescriptionStyl…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "82a182f466d74dbeb7e113ca7b1ef7e0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatText(value=0.0001, description='text_encoder_learning_rate:', layout=Layout(width='300px'), style=Descrip…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"▶️ **Steps**
选择训练步骤和每次验证生成的图像数量"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a07e91a34db44389b9d794bdcbf8196c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"IntSlider(value=200, continuous_update=False, description='max_train_steps:', layout=Layout(width='400px'), ma…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0de5f1096d914b2aa551f6168373e035",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"IntSlider(value=3, description='validation_prompt_num:', layout=Layout(width='400px'), max=5, min=1, style=Sli…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ca1503c909b44234ade3fdb906e484ea",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"FloatSlider(value=1.0, description='validation_step_ratio:', layout=Layout(width='400px'), max=1.0, style=Slid…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "65954a79ae4e41e3ac82797c252204a3",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Text(value='./SD/Datasets/prompts/validation_prompt.txt', description='validation_prompt_path:', disabled=True…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1ed411486ad44eb3b5e6ac3d7f6a1c2e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Button(description='提交配置', style=ButtonStyle())"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6cbff75a3bbb4c289a93e9744787d973",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# 学习率部分\n",
"learning_rate_markdown = Markdown(\"▶️ **Learning Rate**
学习率是影响结果最重要的参数，当前默认三者一致。
如果你想慢速训练且有大量图像，或者如果 LoRA rank（有些文生图UI可能用的是dim） 和 alpha 值较高，建议将 unet 的学习率调至 5e-5 或更低。
需要注意的是，实际上learning_rate在这里不起作用，我们只使用对应模块的学习率。
这里进行展示是为了让你之后在炼丹炉上能够更好的理解设置。\")\n",
"learning_rate_widget = widgets.FloatText(\n",
" value=1e-4,\n",
" description='learning_rate:',\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='300px'),\n",
" disabled=True # 禁用手动修改\n",
")\n",
"\n",
"unet_learning_rate_widget = widgets.FloatText(\n",
" value=1e-4,\n",
" description='unet_learning_rate:',\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='300px')\n",
")\n",
"\n",
"text_encoder_learning_rate_widget = widgets.FloatText(\n",
" value=1e-4,\n",
" description='text_encoder_learning_rate:',\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='300px')\n",
")\n",
"\n",
"# 训练步骤部分\n",
"max_train_steps_markdown = Markdown(\"▶️ **Steps**
选择训练步骤和每次验证生成的图像数量\")\n",
"max_train_steps_widget = widgets.IntSlider(\n",
" value=200,\n",
" min=200,\n",
" max=2000,\n",
" step=100,\n",
" description='max_train_steps:',\n",
" continuous_update=False,\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='400px')\n",
")\n",
"\n",
"\n",
"validation_prompt_num_widget = widgets.IntSlider(\n",
" value=3,\n",
" min=1,\n",
" max=5,\n",
" step=1,\n",
" description='validation_prompt_num:',\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='400px')\n",
")\n",
"\n",
"\n",
"validation_step_ratio_widget = widgets.FloatSlider(\n",
" value=1,\n",
" min=0,\n",
" max=1,\n",
" step=0.1,\n",
" description='validation_step_ratio:',\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='400px')\n",
")\n",
"\n",
"\n",
"validation_prompt_widget = widgets.Text(\n",
" value = validation_prompt_path,\n",
" description='validation_prompt_path:',\n",
" style={'description_width': '200px'},\n",
" layout=widgets.Layout(width='550px'),\n",
" disabled=True # 禁用手动修改\n",
")\n",
"\n",
"# 创建输出区域来显示打印内容\n",
"output_area = widgets.Output()\n",
"\n",
"# 获取用户输入值并打印配置\n",
"def on_button_click(b):\n",
" global unet_learning_rate, text_encoder_learning_rate, max_train_steps, validation_prompt, validation_prompt_num, validation_step_ratio\n",
" with output_area: # 使用 Output 小部件捕获输出\n",
" output_area.clear_output() # 清除之前的输出\n",
" unet_learning_rate = unet_learning_rate_widget.value\n",
" text_encoder_learning_rate = text_encoder_learning_rate_widget.value\n",
" max_train_steps = max_train_steps_widget.value\n",
" validation_prompt_num = validation_prompt_num_widget.value\n",
" validation_step_ratio = validation_step_ratio_widget.value\n",
"\n",
" # 打印配置\n",
" print(f\"UNet 学习率: {unet_learning_rate}\")\n",
" print(f\"文本编码器学习率: {text_encoder_learning_rate}\")\n",
" print(f\"训练步骤: {max_train_steps}\")\n",
" print(f\"验证prompt数量: {validation_prompt_num}\")\n",
" print(f\"验证步骤比例: {validation_step_ratio}\")\n",
" print(f\"用于验证的prompt文件位置: {validation_prompt_path}\")\n",
"\n",
"# 创建提交按钮\n",
"submit_button = widgets.Button(description=\"提交配置\")\n",
"submit_button.on_click(on_button_click)\n",
"\n",
"# 显示带有说明的所有小部件\n",
"display(learning_rate_markdown, learning_rate_widget, unet_learning_rate_widget, text_encoder_learning_rate_widget)\n",
"display(max_train_steps_markdown, max_train_steps_widget, validation_prompt_num_widget, validation_step_ratio_widget, validation_prompt_widget)\n",
"\n",
"# 显示提交按钮\n",
"display(submit_button)\n",
"\n",
"# 显示输出区域\n",
"display(output_area)\n"
]
},
{
"cell_type": "markdown",
"id": "d98bedfc-4764-4a33-b883-cc1e8f3a89dc",
"metadata": {},
"source": [
"## 定义一些有用的函数和类\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "953edb1b-a08f-4cd0-876d-a0b352ba7fb4",
"metadata": {},
"outputs": [],
"source": [
"DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"IMAGE_EXTENSIONS = [\".png\", \".jpg\", \".jpeg\", \".webp\", \".bmp\", \".PNG\", \".JPG\", \".JPEG\", \".WEBP\", \".BMP\"]\n",
"\n",
"# 数据增强\n",
"train_transform = transforms.Compose(\n",
" [\n",
" transforms.Resize(resolution, interpolation=transforms.InterpolationMode.BILINEAR), # 调整图像大小\n",
" transforms.CenterCrop(resolution), # 中心裁剪图像\n",
" transforms.RandomHorizontalFlip(), # 随机水平翻转\n",
" transforms.ToTensor(), # 将图像转换为张量\n",
" ]\n",
")\n",
"class Text2ImageDataset(torch.utils.data.Dataset):\n",
" \"\"\"\n",
" 用于构建文本到图像模型的微调数据集。\n",
" \"\"\"\n",
" def __init__(self, images_folder, captions_folder, transform, tokenizer):\n",
" \"\"\"\n",
" 初始化数据集。\n",
"\n",
" 参数:\n",
" images_folder (str): 图像文件夹路径\n",
" captions_folder (str): 标注文件夹路径\n",
" transform (Callable): 将原始图像转换为 torch.Tensor 的变换函数\n",
" tokenizer (CLIPTokenizer): 将文本标注转为 token ids\n",
" \"\"\"\n",
" # 初始化图像路径列表，并根据指定的扩展名找到所有图像文件\n",
" self.image_paths = []\n",
" for ext in IMAGE_EXTENSIONS:\n",
" self.image_paths.extend(glob.glob(f\"{images_folder}/*{ext}\"))\n",
" self.image_paths = sorted(self.image_paths)\n",
"\n",
" # 通过 DeepFace 库提取每张图像的面部嵌入（特征向量）\n",
" self.train_emb = torch.tensor([DeepFace.represent(img_path, detector_backend=\"ssd\", model_name=\"GhostFaceNet\", enforce_detection=False)[0]['embedding'] for img_path in self.image_paths])\n",
"\n",
" # 加载对应的文本标注，依次读取每个文本文件中的内容\n",
" caption_paths = sorted(glob.glob(f\"{captions_folder}/*txt\"))\n",
" captions = []\n",
" for p in caption_paths:\n",
" with open(p, \"r\") as f:\n",
" captions.append(f.readline())\n",
"\n",
" # 使用 tokenizer 将文本标注转换为 word ids\n",
" inputs = tokenizer(\n",
" captions, max_length=tokenizer.model_max_length, padding=\"max_length\", truncation=True, return_tensors=\"pt\"\n",
" )\n",
" self.input_ids = inputs.input_ids\n",
" self.transform = transform\n",
"\n",
" def __getitem__(self, idx):\n",
" img_path = self.image_paths[idx]\n",
" input_id = self.input_ids[idx]\n",
" try:\n",
" # 加载图像并将其转换为 RGB 模式，然后应用数据增强\n",
" image = Image.open(img_path).convert(\"RGB\")\n",
" tensor = self.transform(image)\n",
" except Exception as e:\n",
" print(f\"无法加载图像路径: {img_path}, 错误: {e}\")\n",
" return None\n",
"\n",
" return tensor, input_id # 返回处理后的图像和相应的文本标注\n",
"\n",
" def __len__(self):\n",
" return len(self.image_paths)\n",
" \n",
"def prepare_lora_model(lora_config, pretrained_model_name_or_path=\"runwayml/stable-diffusion-v1-5\", model_path=None, resume=False):\n",
" \"\"\"\n",
" 加载完整的 Stable Diffusion 模型，包括 LoRA 层，并冻结非 LoRA 参数。\n",
"\n",
" 包括 Tokenizer、噪声调度器、UNet、VAE 和文本编码器。\n",
"\n",
" 参数:\n",
" lora_config (LoraConfig): LoRA 的配置对象\n",
" pretrained_model_name_or_path (str): Hugging Face 上的模型名称或路径\n",
" model_path (str): 预训练模型的路径\n",
" resume (bool): 是否从上一次训练中恢复\n",
"\n",
" 返回:\n",
" tokenizer (CLIPTokenizer)\n",
" noise_scheduler (DDPMScheduler)\n",
" unet (UNet2DConditionModel)\n",
" vae (AutoencoderKL)\n",
" text_encoder (CLIPTextModel)\n",
" \"\"\"\n",
" # 加载噪声调度器，用于控制扩散模型的噪声添加和移除过程\n",
" noise_scheduler = DDPMScheduler.from_pretrained(pretrained_model_name_or_path, subfolder=\"scheduler\")\n",
"\n",
" # 加载 Tokenizer，用于将文本标注转换为 tokens\n",
" tokenizer = CLIPTokenizer.from_pretrained(\n",
" pretrained_model_name_or_path,\n",
" subfolder=\"tokenizer\"\n",
" )\n",
"\n",
" # 加载 CLIP 文本编码器，用于将文本标注转换为特征向量\n",
" text_encoder = CLIPTextModel.from_pretrained(\n",
" pretrained_model_name_or_path,\n",
" torch_dtype=weight_dtype,\n",
" subfolder=\"text_encoder\",\n",
" variant=\"fp16\"\n",
" )\n",
"\n",
" # 加载 VAE 模型，用于在扩散模型中处理图像的潜在表示\n",
" vae = AutoencoderKL.from_pretrained(\n",
" pretrained_model_name_or_path,\n",
" subfolder=\"vae\",\n",
" variant=\"fp16\"\n",
" )\n",
"\n",
" # 加载 UNet 模型，负责处理扩散模型中的图像生成和推理过程\n",
" unet = UNet2DConditionModel.from_pretrained(\n",
" pretrained_model_name_or_path,\n",
" torch_dtype=weight_dtype,\n",
" subfolder=\"unet\",\n",
" variant=\"fp16\"\n",
" )\n",
"\n",
" # 将 LoRA 配置应用到 text_encoder 和 unet\n",
" text_encoder = get_peft_model(text_encoder, lora_config)\n",
" unet = get_peft_model(unet, lora_config)\n",
" \n",
" # 打印可训练参数数量\n",
" text_encoder.print_trainable_parameters()\n",
" unet.print_trainable_parameters()\n",
" \n",
" # 如果设置为继续训练，则加载上一次的模型权重，当然，你可以修改 model_path 来指定其他的路径\n",
" if resume:\n",
" if model_path is None or not os.path.exists(model_path):\n",
" raise ValueError(\"当resume设置为True的时候你必须提供有效的model_path\")\n",
" \n",
" # 加载上次训练的模型权重，注意这里只加载权重，而不是覆盖整个模型，覆盖：model = torch.load(...)\n",
" text_encoder = torch.load(os.path.join(model_path, \"text_encoder.pt\"))\n",
" unet = torch.load(os.path.join(model_path, \"unet.pt\"))\n",
" \n",
" # 冻结 VAE 参数\n",
" vae.requires_grad_(False)\n",
"\n",
" # 使用 get_peft_model() 后会自动冻结其中的非 LoRA 层，所以可以注释掉\n",
" # # 冻结 UNet 和文本编码器中的非 Lora 参数\n",
" # for name, param in unet.named_parameters():\n",
" # if \"lora\" in name:\n",
" # param.requires_grad_(True)\n",
" # else:\n",
" # param.requires_grad_(False)\n",
" # for name, param in text_encoder.named_parameters():\n",
" # if \"lora\" in name:\n",
" # param.requires_grad_(True)\n",
" # else:\n",
" # param.requires_grad_(False)\n",
"\n",
" # 将模型移动到 GPU 上并设置权重的数据类型\n",
" unet.to(DEVICE, dtype=weight_dtype)\n",
" vae.to(DEVICE, dtype=weight_dtype)\n",
" text_encoder.to(DEVICE, dtype=weight_dtype)\n",
" \n",
" return tokenizer, noise_scheduler, unet, vae, text_encoder\n",
"\n",
"\n",
"def prepare_optimizer(unet, text_encoder, unet_learning_rate=5e-4, text_encoder_learning_rate=1e-4):\n",
" \"\"\"\n",
" 为 UNet 和文本编码器的可训练参数分别设置优化器，并指定不同的学习率。\n",
"\n",
" 参数:\n",
" unet (UNet2DConditionModel): Hugging Face 的 UNet 模型\n",
" text_encoder (CLIPTextModel): Hugging Face 的文本编码器\n",
" unet_learning_rate (float): UNet 的学习率\n",
" text_encoder_learning_rate (float): 文本编码器的学习率\n",
"\n",
" 返回:\n",
" torch.optim.Optimizer: 优化器实例\n",
" \"\"\"\n",
" # 筛选出 UNet 中需要训练的 Lora 层参数\n",
" unet_lora_layers = [p for p in unet.parameters() if p.requires_grad]\n",
" \n",
" # 筛选出文本编码器中需要训练的 Lora 层参数\n",
" text_encoder_lora_layers = [p for p in text_encoder.parameters() if p.requires_grad]\n",
" \n",
" # 将需要训练的参数分组并设置不同的学习率\n",
" trainable_params = [\n",
" {\"params\": unet_lora_layers, \"lr\": unet_learning_rate},\n",
" {\"params\": text_encoder_lora_layers, \"lr\": text_encoder_learning_rate}\n",
" ]\n",
" \n",
" # 使用 AdamW 优化器\n",
" optimizer = torch.optim.AdamW(trainable_params)\n",
" \n",
" return optimizer\n",
"\n",
"\n",
"def evaluate(pretrained_model_name_or_path, weight_dtype, seed, unet_path, text_encoder_path, validation_prompt, output_folder, train_emb):\n",
" \"\"\"\n",
" 加载给定路径中的 UNet 和文本编码器模型，生成图像并计算面部相似性、CLIP 分数和无面部图像的数量。\n",
"\n",
" 参数:\n",
" pretrained_model_name_or_path (str): Hugging Face 的模型名称\n",
" weight_dtype (torch.dtype): 模型的权重数据类型\n",
" seed (int): 随机数种子\n",
" unet_path (str): UNet 模型检查点路径\n",
" text_encoder_path (str): 文本编码器模型检查点路径\n",
" validation_prompt (list): 存储验证文本的字符串列表\n",
" output_folder (str): 保存生成图像的文件夹\n",
" train_emb (Tensor): 训练图像的面部特征\n",
"\n",
" 返回:\n",
" face_score (float): 面部相似性评分\n",
" clip_score (float): CLIP 分数\n",
" mis (int): 无法检测到面部的图像数量\n",
" \"\"\"\n",
" # 加载 DiffusionPipeline 并设置 UNet 和文本编码器\n",
" pipeline = DiffusionPipeline.from_pretrained(\n",
" pretrained_model_name_or_path,\n",
" torch_dtype=weight_dtype,\n",
" safety_checker=None,\n",
" variant=\"fp16\",\n",
" )\n",
" pipeline.unet = torch.load(unet_path, weights_only=False) # 加载 UNet 模型\n",
" pipeline.text_encoder = torch.load(text_encoder_path, weights_only=False) # 加载文本编码器模型\n",
" pipeline = pipeline.to(DEVICE)\n",
"\n",
" # 加载 CLIP 模型用于计算图像与文本的相似性分数\n",
" clip_model_name = \"openai/clip-vit-base-patch32\"\n",
" clip_model = AutoModel.from_pretrained(clip_model_name).to(DEVICE)\n",
" clip_processor = AutoProcessor.from_pretrained(clip_model_name)\n",
" \n",
" # 推理生成图像\n",
" with torch.no_grad():\n",
" generator = torch.Generator(device=DEVICE)\n",
" generator.manual_seed(seed)\n",
" face_score = 0\n",
" clip_score = 0\n",
" mis = 0 # 记录无法检测到面部的图像数量\n",
" print(\"正在生成验证图像......\")\n",
" images = []\n",
" for i in range(0, len(validation_prompt), 4):\n",
" images.extend(pipeline(validation_prompt[i:min(i + 4, len(validation_prompt))], num_inference_steps=30, generator=generator).images)\n",
" \n",
" # 计算面部相似性和 CLIP 分数\n",
" print(\"正在计算验证分数......\")\n",
" valid_emb = []\n",
" for i, image in enumerate(tqdm(images)):\n",
" save_file = f\"{output_folder}/valid_image_{i}.png\"\n",
" image.save(save_file) # 将生成的图像保存到指定目录\n",
" opencvImage = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)\n",
"\n",
" # 使用 DeepFace 检测面部特征\n",
" emb = DeepFace.represent(\n",
" opencvImage,\n",
" detector_backend=\"ssd\",\n",
" model_name=\"GhostFaceNet\",\n",
" enforce_detection=False,\n",
" )\n",
" # 如果无法检测到面部，计入 mis\n",
" if emb == [] or emb[0]['face_confidence'] == 0:\n",
" mis += 1\n",
" continue\n",
"\n",
" # 计算 CLIP 分数，衡量图像与文本的相似度\n",
" emb = emb[0]\n",
" inputs = clip_processor(text=validation_prompt[i], images=image, return_tensors=\"pt\").to(DEVICE)\n",
" with torch.no_grad():\n",
" outputs = clip_model(**inputs)\n",
" sim = outputs.logits_per_image # 获取图像与文本的相似性得分\n",
" clip_score += sim.item()\n",
" valid_emb.append(emb['embedding'])\n",
"\n",
" # 如果生成的图片都没有人脸，直接 0 分\n",
" if len(valid_emb) == 0:\n",
" return 0, 0, mis\n",
" \n",
" # 计算面部相似性分数\n",
" valid_emb = torch.tensor(valid_emb)\n",
" valid_emb = (valid_emb / torch.norm(valid_emb, p=2, dim=-1)[:, None]).cuda()\n",
" train_emb = (train_emb / torch.norm(train_emb, p=2, dim=-1)[:, None]).cuda()\n",
" face_score = torch.cdist(valid_emb, train_emb, p=2).mean().item()\n",
" clip_score /= len(validation_prompt) - mis\n",
" \n",
" return face_score, clip_score, mis # 返回面部相似性、CLIP 分数和无面部图像的数量"
]
},
{
"cell_type": "markdown",
"id": "71b834a7-3f78-4d6f-9432-17444e7c29bd",
"metadata": {},
"source": [
"## 准备微调所需的模型和优化器\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f73b8290-6b15-4171-bc1b-403de3abe81b",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"`torch_dtype` is deprecated! Use `dtype` instead!\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARN Python GIL is enabled: Multi-gpu quant acceleration for MoE models is sub-optimal and multi-core accelerated cpu packing is also disabled. We recommend Python >= 3.13.3t with Pytorch > 2.8 for mult-gpu quantization and multi-cpu packing with env `PYTHON_GIL=0`.\n"
]
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARN Feature `utils/Perplexity` requires python GIL or Python >= 3.13.3T (T for Threading-Free edition of Python) plus Torch 2.8. Feature is currently skipped/disabled.\n"
]
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO ENV: Auto setting PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' for memory saving.\n"
]
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO ENV: Auto setting CUDA_DEVICE_ORDER=PCI_BUS_ID for correctness. \n"
]
},
{
"data": {
"text/html": [
""
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"trainable params: 2,359,296 || all params: 125,419,776 || trainable%: 1.8811\n",
"trainable params: 6,377,472 || all params: 865,898,436 || trainable%: 0.7365\n",
"准备完成！\n"
]
}
],
"source": [
"# 决定是否继续之前的训练\n",
"resume = False\n",
"\n",
"# 准备微调所需的 tokenizer, 噪声调度器, UNet, VAE 和文本编码器\n",
"tokenizer, noise_scheduler, unet, vae, text_encoder = prepare_lora_model(lora_config, pretrained_model_name_or_path, model_path, resume)\n",
"\n",
"# 准备优化器，用于更新 UNet 和 文本编码器中经过 LoRA 层调整的可训练参数\n",
"optimizer = prepare_optimizer(unet, text_encoder, unet_learning_rate, text_encoder_learning_rate)\n",
"\n",
"# 设置学习率调度器，控制学习率的变化\n",
"# get_scheduler 根据指定的调度器类型（例如线性、余弦等），为优化器提供一个动态调整学习率的策略\n",
"# num_warmup_steps: 在初始阶段进行学习率预热的步数\n",
"# num_training_steps: 总的训练步数\n",
"# num_cycles: 用于余弦调度器，指定余弦衰减周期的数量\n",
"lr_scheduler = get_scheduler(\n",
" lr_scheduler_name,\n",
" optimizer=optimizer,\n",
" num_warmup_steps=lr_warmup_steps,\n",
" num_training_steps=max_train_steps,\n",
" num_cycles=3\n",
")\n",
"\n",
"# 准备数据集\n",
"# Text2ImageDataset 会根据指定的图像文件夹和文本标注文件夹，加载并处理数据\n",
"# 具体：使用指定的 transform 来预处理图像数据，并使用 tokenizer 来将文本转换为 tokens\n",
"dataset = Text2ImageDataset(\n",
" images_folder=images_folder,\n",
" captions_folder=captions_folder,\n",
" transform=train_transform,\n",
" tokenizer=tokenizer,\n",
")\n",
"\n",
"# 设置数据加载器，负责将数据集分批次加载到模型中进行训练\n",
"# collate_fn: 自定义的批处理函数，将数据集中每个样本的图像和文本标注合并成批次\n",
"def collate_fn(examples):\n",
" pixel_values = []\n",
" input_ids = []\n",
" \n",
" # 将每个样本中的图像 tensor 和文本标注 input_ids 分别收集到列表中\n",
" for tensor, input_id in examples:\n",
" pixel_values.append(tensor) # 将图像张量收集到 pixel_values 列表中\n",
" input_ids.append(input_id) # 将文本标注收集到 input_ids 列表中\n",
" \n",
" # 将列表中的所有图像张量堆叠成一个大的 batch tensor\n",
" pixel_values = torch.stack(pixel_values, dim=0).float() # 形状为 (batch_size, channels, height, width)\n",
" input_ids = torch.stack(input_ids, dim=0) # 形状为 (batch_size, max_seq_length)\n",
" \n",
" # 返回批处理后的图像和文本标注\n",
" return {\"pixel_values\": pixel_values, \"input_ids\": input_ids}\n",
"\n",
"# 使用 PyTorch DataLoader 加载数据集，设置每个批次的大小和线程数\n",
"train_dataloader = torch.utils.data.DataLoader(\n",
" dataset,\n",
" shuffle=True, # 打乱数据\n",
" collate_fn=collate_fn, # 自定义批处理函数\n",
" batch_size=train_batch_size, # 每个批次的样本数量\n",
" num_workers=8, # 使用 8 个线程并行加载数据，加快数据加载速度\n",
")\n",
"\n",
"print(\"准备完成！\")"
]
},
{
"cell_type": "markdown",
"id": "61b27875-b3b2-4b03-a554-4675667e4415",
"metadata": {},
"source": [
"## 开始微调\n",
"\n",
"最后是开始进行微调的部分：\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "8836a890-8e06-4aed-bd11-71df8fd5bf14",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ea143a41524a44bdb8a2decf31872193",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"步骤: 0%| | 0/2000 [00:00 by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .\n",
"Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"正在生成验证图像......\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4d822a90feb9446698d3f395915af9cc",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/30 [00:00= max_train_steps:\n",
" break # 当达到最大训练步数时，终止训练\n",
" \n",
" # 编码图像为潜在表示（latent），通过 VAE 对图像进行编码\n",
" latents = vae.encode(batch[\"pixel_values\"].to(DEVICE, dtype=weight_dtype)).latent_dist.sample()\n",
" latents = latents * vae.config.scaling_factor # 根据 VAE 的缩放因子调整潜在空间\n",
"\n",
" # 为潜在表示添加噪声，生成带噪声的图像\n",
" noise = torch.randn_like(latents) # 生成与潜在表示相同形状的随机噪声\n",
" bsz = latents.shape[0] # 批次大小\n",
" timesteps = torch.randint(0, noise_scheduler.config.num_train_timesteps, (bsz,), device=latents.device).long() # 随机选择时间步\n",
" noisy_latents = noise_scheduler.add_noise(latents, noise, timesteps) # 添加噪声到潜在表示\n",
"\n",
" # 获取文本的嵌入表示\n",
" encoder_hidden_states = text_encoder(batch[\"input_ids\"].to(latents.device), return_dict=False)[0]\n",
" \n",
" # 计算目标值，基于预测类型（epsilon 或 v_prediction）\n",
" if noise_scheduler.config.prediction_type == \"epsilon\":\n",
" target = noise # 预测噪声\n",
" elif noise_scheduler.config.prediction_type == \"v_prediction\":\n",
" target = noise_scheduler.get_velocity(latents, noise, timesteps) # 预测速度向量\n",
"\n",
" # UNet 模型预测，输入噪声潜在空间、时间步和文本嵌入\n",
" model_pred = unet(noisy_latents, timesteps, encoder_hidden_states, return_dict=False)[0]\n",
" \n",
" # 计算损失，基于是否设置了 snr_gamma，这个参数在最开始的部分进行了设置\n",
" if not snr_gamma:\n",
" loss = F.mse_loss(model_pred.float(), target.float(), reduction=\"mean\") # 使用均方误差 (MSE) 作为损失\n",
" else:\n",
" # 计算信噪比 (SNR) 并根据 SNR 加权 MSE 损失\n",
" snr = compute_snr(noise_scheduler, timesteps)\n",
" mse_loss_weights = torch.stack([snr, snr_gamma * torch.ones_like(timesteps)], dim=1).min(dim=1)[0]\n",
" if noise_scheduler.config.prediction_type == \"epsilon\":\n",
" mse_loss_weights = mse_loss_weights / snr\n",
" elif noise_scheduler.config.prediction_type == \"v_prediction\":\n",
" mse_loss_weights = mse_loss_weights / (snr + 1)\n",
" \n",
" # 计算加权的 MSE 损失\n",
" loss = F.mse_loss(model_pred.float(), target.float(), reduction=\"none\")\n",
" loss = loss.mean(dim=list(range(1, len(loss.shape)))) * mse_loss_weights\n",
" loss = loss.mean()\n",
"\n",
" # 反向传播\n",
" loss.backward() # 计算梯度\n",
" optimizer.step() # 更新模型参数\n",
" lr_scheduler.step() # 更新学习率\n",
" optimizer.zero_grad() # 清零梯度\n",
" progress_bar.update(1) # 更新进度条\n",
" global_step += 1 # 更新全局步数\n",
"\n",
"\n",
" # 验证模型性能\n",
" if global_step % validation_step == 0 or global_step == max_train_steps:\n",
" # 保存当前检查点（包含 UNet 和 文本编码器的模型参数）\n",
" save_path = os.path.join(output_folder, f\"checkpoint-last\")\n",
" unet_path = os.path.join(save_path, \"unet.pt\")\n",
" text_encoder_path = os.path.join(save_path, \"text_encoder.pt\")\n",
" print(f\"正在保存检查点到 {save_path} ......\")\n",
" os.makedirs(save_path, exist_ok=True)\n",
" torch.save(unet, unet_path)\n",
" torch.save(text_encoder, text_encoder_path)\n",
" \n",
" # 进行验证\n",
" save_path = os.path.join(output_folder, f\"checkpoint-{global_step}\")\n",
" os.makedirs(save_path, exist_ok=True)\n",
" face_score, clip_score, mis = evaluate(\n",
" pretrained_model_name_or_path=pretrained_model_name_or_path,\n",
" weight_dtype=weight_dtype,\n",
" seed=seed,\n",
" unet_path=unet_path,\n",
" text_encoder_path=text_encoder_path,\n",
" validation_prompt=validation_prompt[:validation_prompt_num], # 验证提示文本\n",
" output_folder=save_path, # 保存验证结果的文件夹\n",
" train_emb=dataset.train_emb # 训练数据中的面部特征嵌入\n",
" )\n",
" print(\"步骤:\", global_step, \"面部相似度评分:\", face_score, \"CLIP评分:\", clip_score, \"无面部图像数量:\", mis)\n",
" \n",
" # 如果当前面部相似度评分优于之前的最佳记录，保存最佳模型\n",
" if face_score < best_face_score:\n",
" best_face_score = face_score\n",
" save_path = os.path.join(output_folder, f\"checkpoint-best\")\n",
" unet_path = os.path.join(save_path, \"unet.pt\")\n",
" text_encoder_path = os.path.join(save_path, \"text_encoder.pt\")\n",
" os.makedirs(save_path, exist_ok=True)\n",
" torch.save(unet, unet_path) # 保存最佳的 UNet 模型\n",
" torch.save(text_encoder, text_encoder_path) # 保存最佳的文本编码器模型\n",
"\n",
"print(\"微调完成！\")"
]
},
{
"cell_type": "markdown",
"id": "cdf90ba1-3511-425c-b278-a3c6da887ed5",
"metadata": {},
"source": [
"## 生成图像和测试\n",
"\n",
"微调过程已经完成。接下来将测试模型。\n",
"\n",
"首先加载之前保存的微调模型的检查点，并计算面部相似度、CLIP 评分以及没有面部的图像数量。"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "cb883783-f35b-46b7-9279-2cf96fdd5d4d",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "faf662203dfc4b09b9f655a3b14a9fc6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading pipeline components...: 0%| | 0/6 [00:00 by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"正在生成验证图像......\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f6bf5c08379e4dfb874d081868fa6ab7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/30 [00:00