{ "cells": [ { "cell_type": "markdown", "id": "945de83c-51d8-429a-9a08-4d89b254db9d", "metadata": {}, "source": [ "# 自定义 Prompt 提升大模型解题能力——Gradio 与 ipywidgets 版\n", "\n", "> [HW4: Become an AI Hypnosis Master](https://colab.research.google.com/drive/16JzVN_Mu4mJfyHQpQEuDx1q6jI-cAnEl?hl=zh-tw#scrollTo=RI0hC7SFT3Sr&uniqifier=1) 中文镜像版\n", ">\n", "> 指导文章：[03. 进阶指南：自定义 Prompt 提升大模型解题能力](https://github.com/Hoper-J/LLM-Guide-and-Demos/blob/master/Guide/03.%20进阶指南：自定义%20Prompt%20提升大模型解题能力.md)\n", "\n", "目标：\n", "1. 了解各种 Prompt 如何影响大型语言模型的性能。\n", "2. 设计 Prompt 提高大模型解决数学问题的正确性\n", "\n", "\n", "P.S. 这里修正了一些与预期行为不符的代码\n", "\n", "在线链接：[Kaggle](https://www.kaggle.com/code/aidemos/03-prompt-ipywidgets) | [Colab](https://colab.research.google.com/drive/1Xx_L9VfH6kloW3nwwAsmok1EgRI3xDTh?usp=sharing)" ] }, { "cell_type": "markdown", "id": "3828ffb4-44cb-4bd5-ab23-e44eeb1de578", "metadata": {}, "source": [ "## 准备阶段" ] }, { "cell_type": "markdown", "id": "a4e02231-4180-4b13-b9ac-33cfa68a4016", "metadata": {}, "source": [ "### 下载需要的库" ] }, { "cell_type": "code", "execution_count": null, "id": "b50b89f0-b911-4c3b-90ea-3e67468453c2", "metadata": {}, "outputs": [], "source": [ "%pip install tqdm\n", "%pip install jinja2\n", "%pip install gradio\n", "%pip install tiktoken\n", "%pip install openai" ] }, { "cell_type": "markdown", "id": "0320655c-3ec6-4366-a3da-b6d1cb9d3f97", "metadata": {}, "source": [ "### 导入" ] }, { "cell_type": "code", "execution_count": null, "id": "f036dbde-b141-435d-ade2-b3e8a60b19b9", "metadata": {}, "outputs": [], "source": [ "import os\n", "import time\n", "import re\n", "import pickle\n", "import json\n", "import traceback\n", "\n", "import openai\n", "import tiktoken # 用于 prompt_token_num()\n", "import jinja2\n", "from tqdm import tqdm" ] }, { "cell_type": "markdown", "id": "ff576b53-87d2-4d55-9e3c-7cc4b456a098", "metadata": {}, "source": [ "### 设置 openai api" ] }, { "cell_type": "code", "execution_count": null, "id": "f3be165c-6879-4da9-923f-d7f2bf110179", "metadata": {}, "outputs": [], "source": [ "# TODO: 设置你的 OPENAI API 密钥，这里假设 DashScope API 被配置在了 OPENAI_API_KEY 环境变量中\n", "OPENAI_API_KEY = \"\"\n", "# 不填写则默认使用环境变量\n", "if not OPENAI_API_KEY:\n", " OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')\n", "\n", "# 初始化 OpenAI 客户端，使用阿里云 DashScope API\n", "client = openai.OpenAI(\n", " api_key=OPENAI_API_KEY,\n", " base_url=\"https://dashscope.aliyuncs.com/compatible-mode/v1\", # 阿里云的 API 地址\n", ")\n", "\n", "# 检查 API 设置是否正确\n", "try:\n", " response = client.chat.completions.create(\n", " model=\"qwen-turbo\", # 使用通义千问-Turbo 大模型，可以替换为 Deepseek 系列：deepseek-v3 / deepseek-r1\n", " messages=[{'role': 'user', 'content': \"测试\"}],\n", " max_tokens=1,\n", " )\n", " print(\"API 设置成功！！\")\n", "except Exception as e:\n", " print(f\"API 可能有问题，请检查：{e}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "20a2253d-d032-41de-a067-37c0dd2651db", "metadata": {}, "outputs": [], "source": [ "class OpenAIModel:\n", " \"\"\"\n", " 封装OpenAI API调用和缓存机制的类。\n", " \n", " 用于调用OpenAI API，处理响应，并缓存结果以提高效率。\n", " \n", " 属性:\n", " cache_file (str): 缓存文件的路径\n", " cache_dict (dict): 内存中的缓存字典\n", " \"\"\"\n", " \n", " def __init__(self, cache_file=\"openai_cache\"):\n", " \"\"\"\n", " 初始化OpenAI模型对象，设置缓存文件路径并加载缓存。\n", " \n", " 参数:\n", " cache_file (str): 缓存文件的路径，默认为\"openai_cache\"\n", " \"\"\"\n", " self.cache_file = cache_file\n", " self.cache_dict = self.load_cache() # 加载缓存\n", "\n", " def save_cache(self):\n", " \"\"\"\n", " 将当前缓存保存到文件中。\n", " \"\"\"\n", " with open(self.cache_file, \"wb\") as f:\n", " pickle.dump(self.cache_dict, f)\n", "\n", " def load_cache(self, allow_retry=True):\n", " \"\"\"\n", " 从文件加载缓存，支持重试机制。\n", " \n", " 参数:\n", " allow_retry (bool): 是否允许重试加载缓存，默认为True\n", " \n", " 返回:\n", " dict: 加载的缓存字典，如果文件不存在则返回空字典\n", " \"\"\"\n", " if os.path.exists(self.cache_file):\n", " while True:\n", " try:\n", " with open(self.cache_file, \"rb\") as f:\n", " cache = pickle.load(f)\n", " break\n", " except Exception:\n", " if not allow_retry:\n", " assert False\n", " print(\"Pickle Error: 5秒后重试...\")\n", " time.sleep(5)\n", " else:\n", " # 如果文件不存在则初始化缓存\n", " cache = {}\n", " return cache\n", "\n", " def set_cache_file(self, file_name):\n", " \"\"\"\n", " 设置缓存文件名并重新加载缓存。\n", " \n", " 参数:\n", " file_name (str): 新的缓存文件路径\n", " \"\"\"\n", " self.cache_file = file_name\n", " self.cache_dict = self.load_cache()\n", "\n", " def get_response(self, content):\n", " \"\"\"\n", " 获取模型完成的文本，先检查缓存，若无则请求生成。\n", " \n", " 参数:\n", " content (str): 提供给模型的输入内容\n", " \n", " 返回:\n", " str: 模型生成的回复文本，如果出错则返回None\n", " \"\"\"\n", " # 如果选择检查缓存，则会导致同问题不同trial的结果相同，这与实际想表达的内容不符，故注释\n", " # if content in self.cache_dict:\n", " # return self.cache_dict[content]\n", " for _ in range(3): # 尝试三次\n", " try:\n", " # 调用模型生成内容\n", " response = client.chat.completions.create(\n", " model=\"qwen-turbo\",\n", " messages=[{\"role\": \"user\", \"content\": content}],\n", " temperature=1.0,\n", " )\n", " completion = response.choices[0].message.content\n", " self.cache_dict[content] = completion\n", " return completion\n", " except Exception as e:\n", " print(e, \"\\n\")\n", " time.sleep(1)\n", " return None\n", "\n", " def is_valid_key(self):\n", " \"\"\"\n", " 检查API密钥是否有效。\n", " \n", " 返回:\n", " bool: 如果API密钥有效则返回True，否则返回False\n", " \"\"\"\n", " for _ in range(4): # 尝试四次\n", " try:\n", " response = client.chat.completions.create(\n", " model=\"qwen-turbo\",\n", " messages=[{\"role\": \"user\", \"content\": \"hi there\"}],\n", " temperature=1.0,\n", " max_tokens=1\n", " )\n", " return True\n", " except Exception as e:\n", " traceback.print_exc()\n", " time.sleep(1)\n", " return False\n", "\n", " def prompt_token_num(self, prompt):\n", " \"\"\"\n", " 计算prompt的token数量。\n", " \n", " 参数:\n", " prompt (str): 要计算token数量的prompt\n", " \n", " 返回:\n", " int: token的数量\n", " \"\"\"\n", " try:\n", " # 使用 gpt-3.5-turbo 的编码器，因为 tiktoken 库不支持自动识别 qwen-turbo 模型\n", " encoding = tiktoken.get_encoding(\"cl100k_base\") # 这是 GPT-3.5-turbo 所使用的编码器\n", " # 将 prompt 编码成 token，并返回 token 数量\n", " tokens = encoding.encode(prompt)\n", " return len(tokens)\n", " except Exception as e:\n", " print(f\"计算 token 数量时出错: {e}\")\n", " return 0\n", "\n", " def two_stage_completion(self, question, content):\n", " \"\"\"\n", " 两阶段完成：首先获取推理（注意，这里并非推理模型的思维链，而是直接输入 content 后得到的回复），再获取最终答案。\n", " \n", " 参数:\n", " question (str): 原始问题\n", " content (str): 提供给模型的输入内容\n", " \n", " 返回:\n", " dict: 包含prompt、推理过程和答案的字典\n", " \"\"\"\n", " rationale = self.get_response(content)\n", " if not rationale:\n", " return {\n", " \"prompt\": content,\n", " \"rationale\": None,\n", " \"answer\": None\n", " }\n", "\n", " ans = self.get_response(content=f\"Q:{question}\\nA:{rationale}\\nThe answer to the original question is (a number only): \")\n", " return {\n", " \"prompt\": content,\n", " \"rationale\": rationale,\n", " \"answer\": ans\n", " }\n", "\n", "# 初始化模型\n", "my_model = OpenAIModel()" ] }, { "cell_type": "markdown", "id": "6c38238b-9495-4288-8083-29d148ce13fc", "metadata": {}, "source": [ "### 用于评估 Prompt 的数学问题\n", "\n", "- 我们提供了30个问题用于评估你的 Prompt：[链接](https://docs.google.com/spreadsheets/d/1IehN_Qx40wPcreVE5UorTQz-puCI8NbNDQEMvcuaZcs/edit?hl=zh-tw#gid=0)\n", "- 运行此代码块是设置所有问题和答案所必需的。如果不运行此代码块，你将无法在后续的代码块中访问这些问题。\n", "- **请勿修改此代码块！**\n" ] }, { "cell_type": "code", "execution_count": null, "id": "5a11113f-ceea-41a5-b7e2-65f25552fbb9", "metadata": {}, "outputs": [], "source": [ "questions = [\n", " '一位艺术家正在使用方形瓷砖创建一个大型马赛克。马赛克本身设计成一个正方形，并且包含的蓝色瓷砖数量恰好是红色瓷砖的3倍。如果艺术家只有57块红色瓷砖，那么要完成整个马赛克共需要多少块瓷砖？',\n", " '一位农民正在为当地市场装苹果。他有120个苹果，并希望将它们均匀分配到篮子中。如果他决定留15个苹果给家人，每个篮子最多能装7个苹果，那么他最少需要多少个篮子才能将苹果带到市场？',\n", " '一个花园有矩形的地块，这些地块排列成一条直线，每块地与正好两块其他的地共用边界。共有5块地。中间的每块地面积为24平方米，地块的宽度为4米，所有地块的宽度保持不变。第一块地的长度是中间地块的两倍，最后一块地的长度是中间地块的一半。那么所有地块的总面积是多少平方米？',\n", " '一个农贸市场出售两种类型的苹果混合袋：A型袋子包含4个红苹果和6个绿苹果，B型袋子包含8个红苹果和4个绿苹果。一位顾客购买了一袋A型和一袋B型的苹果。如果从这两袋苹果中随机挑选一个苹果，选到绿苹果的概率是多少？请将答案保留到小数点后两位。',\n", " '一位园丁按照两朵红色花跟着一朵黄色花的模式种花。如果园丁想保持这种模式，并且有35个连续的空位来种花，那么园丁会种多少朵红色花？',\n", " '杰森正在为马拉松训练，他每天跑固定的距离。星期一，他跑了5英里。之后的每一天，他将跑步距离增加10%。如果杰森按照这个模式继续跑步，那么他在星期五将跑多少英里？',\n", " '在一个三角形的花坛边上，每条边上有16棵植物。每棵植物都需要一个半径为0.5米的圆形空间才能正常生长。假设植物紧挨着排列，并且沿着三角形花坛的边排成一条直线，那么每条边上种植物的线性距离是多少米？',\n", " '威尔逊博士正在设计一个几何花园，花园中的花朵围绕着中央的喷泉排列成同心圆。每一圈比里面一圈多6朵花，形成一个六边形的图案。最里面一圈有6朵花。如果威尔逊博士种足够的花，形成15圈（包括最里面一圈），那么这个花园总共需要多少朵花？',\n", " '一个小图书馆正在重新整理书籍收藏。他们总共有120本书，计划平均分配到5个书架上。最上面的书架只能承受相当于最下面书架一半重量的书。如果最上面的书架承载15磅的书，而其他书架每个能承载两倍的重量，那么最下面的书架能承载多少磅的书？',\n", " '一份饼干的配方需要3杯面粉、2杯糖和1杯巧克力片。如果马克想要做三倍量的饼干，但只有4杯糖，那么他还需要多少杯糖？',\n", " '一家宠物店的店主正在制作定制鸟舍。每个鸟舍外部需要0.75升木材清漆。如果店主有一罐10升的木材清漆，那么他在需要更多清漆之前最多可以制作多少个鸟舍？',\n", " '一个农场有鸡和牛。总共有30个头，88条腿。农场上有多少头牛？',\n", " '一个地方图书馆正在组织一场旧书义卖会，以筹集资金购买新书。他们以每本2美元的价格卖出120本儿童书，以每本3美元的价格卖出75本小说，并以每本1.50美元的价格卖出了小说两倍数量的杂志。他们还以每本0.50美元的价格卖出与书籍和杂志总数相等的书签。那么图书馆总共筹集了多少钱？',\n", " '一个当地的农民正在为市场准备混合水果篮，每个篮子包含3个苹果、5个橙子和2个香蕉。苹果的价格是每个0.50美元，橙子每个0.30美元，香蕉每个0.25美元。如果农民为当地市场准备了120个篮子，并以每个5.00美元的价格出售每个篮子，那么卖完所有篮子后，农民将获得多少利润？',\n", " '玛丽亚有24个苹果，想将它们均匀分给她的6个朋友。如果每个朋友还要再给老师2个苹果，那么每个朋友剩下多少苹果？',\n", " '莉拉正在计划一个花园，想要种三种花：雏菊、郁金香和玫瑰。她想要的雏菊数量是郁金香的两倍，郁金香的数量是玫瑰的三倍。如果她总共要种60朵花，那么她计划种多少朵玫瑰？',\n", " '一个花园有三种开花植物。第一种每株有12朵花，第二种每株有8朵花，第三种每株有15朵花。如果第一种植物的数量是第二种植物的两倍，第三种植物的数量是第一种植物的一半，并且花园中有16株第二种植物，那么花园里一共有多少朵花？',\n", " '在一个棋盘游戏中，从一个方格转移到另一个方格的费用是你要落在的方格号码的硬币数。第一个方格是1号，第二个方格是2号，以此类推。如果一个玩家从5号方格移动到9号方格，再到14号方格，最后到20号方格，他总共花费了多少枚硬币？',\n", " '一个景观公司在两个公园种植树木。在A公园，他们种了5排，每排6棵树。在B公园，他们种了3排，每排7棵树。然而，B公园的4棵树没有成活，必须移除。移除之后，总共剩下多少棵树？',\n", " '欧拉博士正在计划一场数学比赛，他决定将参与者分成几组。为了保证公平，每组必须有相同数量的参与者。如果欧拉博士可以选择将参与者分成4人、5人或6人的组，并且参与者总数少于100人，那么他最多可以有多少参与者，确保无论怎么分组都不会有剩余？',\n", " '一个农民为万圣节种植南瓜。他种了8排，每排15棵南瓜植株。每棵植株平均产出3个南瓜。收获后，农民将20%的南瓜卖给当地市场，剩下的在他的农场摊位上出售。如果每个南瓜卖4美元，农民通过销售南瓜总共赚了多少钱？',\n", " '一个三角形公园ABC的边缘上种植了树木。边AB上的树木数量等于边BC的长度，边BC上的树木数量等于边CA的长度，边CA上的树木数量等于边AB的长度。如果边AB、BC和CA（以米为单位）的长度构成一个公比为2的几何级数，并且总共种植了77棵树，求边AB的长度。',\n", " '一群朋友正在收集可回收的罐子。玛雅收集的罐子是利亚姆的两倍。利亚姆收集了15个罐子。如果佐伊比玛雅多收集了5个罐子，并且这群朋友想把罐子平分给4家慈善机构，每家会收到多少个罐子？',\n", " '在一场科学比赛中，每个团队需要制作一个模型火箭。有6个团队，每个团队需要一套材料。材料包括火箭的主体管、引擎和降落伞。主体管每个12.50美元，引擎每个18.75美元，降落伞每个6.25美元。购买所有团队的材料后，总费用为225美元。制作一支火箭的材料费用是多少？',\n", " '艾米丽有一个小菜园，种植了番茄、胡萝卜和黄瓜。她的番茄植株数量是黄瓜植株的两倍，而胡萝卜植株比番茄少5棵。如果艾米丽有4棵黄瓜植株，那么她总共有多少棵菜园植物？',\n", " '在一个小村庄，当地裁缝制作外套和裤子。制作一件外套需要3码布料，而制作一条裤子需要2码布料。他接到了一份剧院制作的订单，要求的裤子数量是外套的两倍，而剧院要求了4件外套。如果布料的价格是每码15美元，那么剧院在这个订单上需要花费多少布料费用？',\n", " '一个小镇的人口以恒定的速率增长。如果2010年小镇的人口是5000人，2020年是8000人，那么如果这种增长趋势继续，到2025年小镇的人口会是多少？',\n", " '一位数学老师正在组织一场测验比赛，并决定用铅笔作为奖品。每位参与者将获得2支铅笔，而得分超过80%的学生将额外获得3支铅笔。如果班上有30名学生，其中1/5的学生得分超过80%，那么老师需要准备多少支铅笔？',\n", " '一个长方形的花园被120米的围栏包围。如果花园的长度是其宽度的三倍，那么花园的面积是多少平方米？',\n", " '一个长10米、宽15米的花园将用方形瓷砖铺设。每块瓷砖的边长为25厘米。如果每块瓷砖的价格是3美元，而铺设瓷砖的人工费用是每平方米8美元，那么铺设整个花园的总费用是多少？'\n", "]\n", "answers = [\n", " 228, 15, 132, 0.45, 24, 7.3205, 16, 720, 30, 2, 13, 14, 862.5, 180, 2, 6, 752, \n", " 43, 47, 60, 1440, 11, 20, 37.5, 15, 420, 9500, 78, 675, 8400\n", "]" ] }, { "cell_type": "markdown", "id": "cf521c3e-a275-4634-b91b-dc6f7975ffbd", "metadata": {}, "source": [ "## 🧙 创建你的自定义 Prompt（Gradio 版本）\n", "\n", "此模块创建了一个 Gradio 界面，使你能够在 Gradio 中设置、评估和保存自定义提示词。" ] }, { "cell_type": "markdown", "id": "2fe8c38c-def5-487b-b32c-388772a3dbbe", "metadata": {}, "source": [ "### 使用手册\n", "\n", "0. **确保 API 密钥设置正确，否则可能会得到意外结果（可能会出现错误）。**\n", "1. **在 `Prompt` 中编辑你的自定义提示词：**\n", " - 在你的提示词中包含 `{{question}}`，不要进行任何更改。此占位符将被提供的示例中选择的一个问题替换。\n", " ![占位符](../Guide/assets/%E5%8D%A0%E4%BD%8D%E7%AC%A6-6055722.png)\n", " - 最大长度：1024 个 token。\n", "\n", "2. **设置你的自定义提示词：**\n", " - 点击 `Set Prompt` 将 `Prompt` 中的内容应用为你的自定义提示词。使用 `Clear Prompt` 清除已输入的任何自定义提示词。\n", " - 设置提示词后，你可以在 `Log` 中查看。\n", "\n", "3. **评估你的自定义提示词：**\n", " - 点击 `Evaluate` 使用 `OpenAI` 评估你的自定义提示词，评估将使用前 `Number of examples used for evaluation` 个问题。\n", " - 每个问题将进行三次评估。\n", " - 评估所有问题可能需要 **20 到 30 分钟**，具体时间取决于提示词的长度。\n", "\n", "4. **查看结果：**\n", " - 术语解释：\n", " - `Trial Number`：范围为 `1∼3`，每个问题进行三次评估，并投票确定最终结果。\n", " - `Question Number`：范围为 `1∼n`，指评估所使用的前 `n` 个问题。\n", "\n", " - 你可以调整滑块查看具体的结果。\n", " - 建议逐次点击调整滑块，因为 Gradio 界面响应速度较慢。如果你想快速跳转到某个数字，可以直接在数值输入框中输入，以确保得到所需结果。\n", " - 总体分数显示在 `Result Stats` 中。\n", "\n", "5. **保存你的自定义提示词：**\n", " - 点击 `Save Custom Prompt` 将结果保存为 json 文件。\n", "\n", "6. **在 `Log` 中查看过程：**\n", " - 所有操作都会记录在 `Log` 中。可以参考它来了解过程或重复实验，不过需要注意某些操作可能会引入随机性。\n", "\n", "**注意事项**\n", "\n", "- 在文本框中插入换行，请使用 `Shift+Enter`。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a341e7c0-4be0-46d2-8214-d83015e06c2f", "metadata": {}, "outputs": [], "source": [ "# 使用gradio进行自定义prompt操作\n", "import gradio as gr\n", "\n", "def reset_prompt(chatbot):\n", " \"\"\"\n", " Reset按钮点击处理：重置prompt\n", " \n", " 参数:\n", " chatbot (List): 聊天记录\n", " \n", " 返回:\n", " Tuple: 更新后的聊天记录和清空的提示词文本\n", " \"\"\"\n", " gr.Info(\"已清除提示词\")\n", " chatbot.extend([[\"清除提示词\", \"提示词已成功重置\"]])\n", " return chatbot, \"\", 0\n", "\n", "def assign_prompt(chatbot, prompt, template, example_number):\n", " \"\"\"\n", " Assign按钮点击处理：分配有效prompt并设置template\n", " \n", " 参数:\n", " chatbot (List): 聊天记录\n", " prompt (str): 用户输入的提示词\n", " template: 当前的模板对象\n", " example_number (int): 选择的示例编号\n", " \n", " 返回:\n", " Tuple: 更新后的聊天记录、提示词文本、模板对象和选择的示例编号\n", " \"\"\"\n", " gr.Info(\"正在分配提示词\")\n", " example_number = int(example_number)\n", " token_num = my_model.prompt_token_num(prompt)\n", " \n", " if token_num > 1024:\n", " template = None\n", " gr.Warning(\"无效的提示词（太长，超过1024个token）\")\n", " chatbot.append([None, \"提示词太长（超过1024个token）。较短的提示词可以更快且更稳定地评估！\"])\n", " elif example_number < 1 or example_number > len(questions):\n", " template = None\n", " prompt_ex = f\"错误：请选择一个1到{len(questions)}之间的数字\"\n", " gr.Warning(prompt_ex)\n", " chatbot.extend([[None, prompt_ex]])\n", " elif \"{{question}}\" not in prompt:\n", " template = None\n", " prompt_ex = \"你需要在提示词中包含占位符{{question}}。\"\n", " gr.Warning(prompt_ex)\n", " chatbot.extend([[None, prompt_ex]])\n", " else:\n", " environment = jinja2.Environment()\n", " template = environment.from_string(prompt)\n", " prompt_ex = f\"\"\"{template.render(question=questions[example_number - 1])}\"\"\"\n", " chatbot.extend([[\"分配提示词\", \"提示词已成功分配\\n\\n自定义提示词示例：\"], [None, prompt_ex]])\n", " \n", " return chatbot, prompt, template, example_number, token_num\n", "\n", "def clean_commas(text):\n", " \"\"\"\n", " 处理数字中的逗号（千位分隔符）\n", " \n", " 参数:\n", " text (str): 包含数字的文本\n", " \n", " 返回:\n", " str: 处理后的文本\n", " \"\"\"\n", " def process_match(match):\n", " number = match.group(0)\n", " if '.' in number:\n", " return number\n", " else:\n", " number_list = number.split(\",\")\n", " new_string = number_list[0]\n", " for i in range(1, len(number_list)):\n", " if len(number_list[i]) == 3:\n", " new_string += number_list[i]\n", " else:\n", " new_string += f\",{number_list[i]}\"\n", " return new_string\n", " \n", " pattern = r'\\d+(?:,\\d+)*(?:\\.\\d+)?'\n", " return re.sub(pattern, process_match, text)\n", "\n", "def find_and_match_floats(input_string, ground_truth):\n", " \"\"\"\n", " 检查输入中的数字是否与预期匹配\n", " \n", " 参数:\n", " input_string (str): 包含数字的输入字符串\n", " ground_truth (float): 预期的正确数值\n", " \n", " 返回:\n", " bool: 如果找到匹配的数值则返回True，否则返回False\n", " \"\"\"\n", " pattern = re.compile(r\"[-+]?\\d*\\.\\d+|[-+]?\\d+\")\n", " found_numbers = pattern.findall(input_string)\n", " found_floats = [float(num) for num in found_numbers]\n", " return ground_truth in found_floats\n", "\n", "def assess_prompt(chatbot, template, test_num):\n", " \"\"\"\n", " Test按钮点击处理：评估自定义prompt\n", " \n", " 参数:\n", " chatbot (List): 聊天记录\n", " template: 当前的模板对象\n", " test_num (int): 要测试的问题数量\n", " \n", " 返回:\n", " Tuple: 更新后的聊天记录、结果列表、结果统计和UI组件\n", " \"\"\"\n", " if template is None:\n", " chatbot.extend([[None, \"评估失败，因为提示词模板为空（即无效的提示词）\"]])\n", " gr.Warning(\"提示词未设置\")\n", " return chatbot, [], \"提示词未设置\", gr.Slider(label=\"Result Number\", value=0, minimum=0, maximum=0, step=1), gr.Textbox(label=\"Result\", value=\"\", interactive=False)\n", "\n", " gr.Info(\"正在评估提示词\")\n", " ans_template = \"提示词和问题：\\n\\n{{question}}\\n\\n--------------------\\n\\n解题过程：\\n\\n{{rationale}}\\n\\n--------------------\\n\\n最终答案\\n\\n{{answer}}\"\n", " res_list = []\n", " total_count = test_num\n", " environment = jinja2.Environment()\n", " ans_template = environment.from_string(ans_template)\n", " trial_num = 3\n", " trials = [[] for _ in range(trial_num)]\n", " res_stats_str = \"\"\n", "\n", " for i in range(trial_num):\n", " gr.Info(f\"开始第{i+1}次测试\")\n", " accurate_count = 0\n", " for idx, example in enumerate(questions[:test_num]):\n", " test_res = \"\"\n", " result = my_model.two_stage_completion(example, template.render(question=example))\n", "\n", " if not result[\"answer\"]:\n", " trials[i].append(0)\n", " test_res += f\"第{i+1}次测试\\n\\n跳过问题 {idx + 1}。\"\n", " test_res += \"\\n\" + \"<\"*6 + \"=\"*30 + \">\"*6 + \"\\n\\n\"\n", " res_list.append(f\"第{i+1}次测试\\n\\n跳过问题 {idx + 1}。\")\n", " continue\n", "\n", " cleaned_result = clean_commas(result[\"answer\"])\n", " if find_and_match_floats(cleaned_result, answers[idx]):\n", " accurate_count += 1\n", " trials[i].append(1)\n", " else:\n", " trials[i].append(0)\n", "\n", " my_model.save_cache()\n", " test_res += f\"第{i + 1}次测试\\n\\n\"\n", " test_res += f\"问题 {idx + 1}:\\n\" + '-'*20\n", " test_res += f'''\\n\\n{ans_template.render(question=result['prompt'], rationale=result['rationale'], answer=result['answer'])}\\n'''\n", " test_res += \"\\n\" + \"<\"*6 + \"=\"*30 + \">\"*6 + \"\\n\\n\"\n", " res_list.append(test_res)\n", "\n", " res_stats_str += f\"第{i + 1}次测试，正确数：{accurate_count}，总数：{total_count}，准确率：{accurate_count / total_count * 100}%\\n\"\n", " my_model.save_cache()\n", "\n", " voting_acc = 0\n", " for i in range(total_count):\n", " count = 0\n", " for j in range(trial_num):\n", " if trials[j][i] == 1:\n", " count += 1\n", " if count >= 2:\n", " voting_acc += 1\n", "\n", " res_stats_str += f\"最终准确率：{voting_acc / total_count * 100}%\"\n", " chatbot.extend([[\"测试\", \"测试完成。结果可以在“结果”和“结果统计”中找到。\"]])\n", " chatbot.extend([[None, \"测试结果\"], [None, ''.join(res_list)], [None, \"结果统计\"], [None, res_stats_str]])\n", " \n", " return chatbot, res_list, res_stats_str, gr.Slider(label=\"Result Number\", value=1, minimum=1, maximum=len(res_list), step=1, visible=False), gr.Textbox(label=\"Result\", value=res_list[0], interactive=False)\n", "\n", "def save_prompt(chatbot, prompt):\n", " \"\"\"\n", " Save按钮点击处理：保存提示词\n", " \n", " 参数:\n", " chatbot (List): 聊天记录\n", " prompt (str): 用户输入的提示词\n", " \n", " 返回:\n", " List: 更新后的聊天记录\n", " \"\"\"\n", " gr.Info(\"正在保存提示词\")\n", " prompt_dict = {\n", " \"prompt\": prompt\n", " }\n", " with open(\"prompt.json\", \"w\") as f:\n", " json.dump(prompt_dict, f)\n", " chatbot.extend([[\"保存提示词\", f\"提示词已保存为prompt.json\"]])\n", " return chatbot\n", "\n", "# Gradio界面\n", "with gr.Blocks() as demo:\n", " my_magic_prompt = \"任务：\\n解决以下数学问题。\\n\\n问题：{{question}}\\n\\n答案：\"\n", " my_magic_prompt = my_magic_prompt.strip('\\n')\n", " template = gr.State(None)\n", " res_list = gr.State(list())\n", "\n", " # 组件\n", " with gr.Tab(label=\"Console\"):\n", " with gr.Group():\n", " example_num_box = gr.Dropdown(\n", " label=\"Demo Example (Please choose one example for demo)\",\n", " value=1,\n", " info=questions[0],\n", " choices=[i+1 for i in range(len(questions))],\n", " filterable=False\n", " )\n", " prompt_textbox = gr.Textbox(\n", " label=\"Custom Prompt\",\n", " placeholder=f\"在这里输入你的自定义提示词。例如：\\n\\n{my_magic_prompt}\",\n", " value=\"\",\n", " info=\"请确保包含`{{question}}`标签。\"\n", " )\n", " with gr.Row():\n", " set_button = gr.Button(value=\"Set Prompt\")\n", " reset_button = gr.Button(value=\"Clear Prompt\")\n", " prompt_token_num = gr.Textbox(\n", " label=\"Number of prompt tokens\",\n", " value=0,\n", " interactive=False,\n", " info=\"自定义提示词的Token数量。\"\n", " )\n", " with gr.Group():\n", " test_num = gr.Slider(\n", " label=\"Number of examples used for evaluation\",\n", " minimum=1,\n", " maximum=len(questions),\n", " step=1,\n", " value=1\n", " )\n", " assess_button = gr.Button(value=\"Evaluate\")\n", " with gr.Group():\n", " with gr.Row():\n", " with gr.Column():\n", " with gr.Row():\n", " trial_no = gr.Slider(\n", " label=\"Trial ID\",\n", " value=1,\n", " minimum=1,\n", " maximum=3,\n", " step=1\n", " )\n", " ques_no = gr.Slider(\n", " label=\"Question ID\",\n", " value=1,\n", " minimum=1,\n", " maximum=1,\n", " step=1\n", " )\n", " res_num = gr.Slider(\n", " label=\"Result Number\",\n", " value=0,\n", " minimum=0,\n", " maximum=0,\n", " step=1,\n", " visible=False\n", " )\n", " res = gr.Textbox(\n", " label=\"Result\",\n", " value=\"\",\n", " placeholder=\"暂无结果\",\n", " interactive=False\n", " )\n", " with gr.Column():\n", " res_stats = gr.Textbox(label=\"Result Stats\", interactive=False)\n", " save_button = gr.Button(value=\"Save Custom Prompt\")\n", " with gr.Tab(label=\"Log\"):\n", " chatbot = gr.Chatbot(type=\"tuples\", label=\"Log\")\n", "\n", " # 事件处理\n", " example_num_box.input(\n", " lambda example_number: gr.Dropdown(\n", " label=\"Example (Please choose one example for demo)\",\n", " value=example_number,\n", " info=questions[example_number - 1],\n", " choices=[i+1 for i in range(len(questions))]\n", " ),\n", " inputs=[example_num_box],\n", " outputs=[example_num_box]\n", " )\n", " \n", " res_num.change(\n", " lambda results, result_num, test_num: (\n", " gr.Textbox(label=\"Result\", value=results[result_num-1], interactive=False) \n", " if len(results) != 0 \n", " else gr.Textbox(label=\"Result\", value=\"\", placeholder=\"暂无结果\", interactive=False),\n", " (int)((result_num-1)/test_num)+1,\n", " gr.Slider(\n", " label=\"Question Number\", \n", " minimum=1, \n", " maximum=test_num, \n", " value=(result_num-1)%test_num+1, \n", " step=1\n", " )\n", " ),\n", " inputs=[res_list, res_num, test_num],\n", " outputs=[res, trial_no, ques_no]\n", " )\n", " \n", " trial_ques_no_input = lambda t_val, q_val, test_num: (t_val - 1) * test_num + q_val\n", " trial_no.input(trial_ques_no_input, inputs=[trial_no, ques_no, test_num], outputs=[res_num])\n", " ques_no.input(trial_ques_no_input, inputs=[trial_no, ques_no, test_num], outputs=[res_num])\n", " set_button.click(assign_prompt, inputs=[chatbot, prompt_textbox, template, example_num_box], outputs=[chatbot, prompt_textbox, template, example_num_box, prompt_token_num])\n", " reset_button.click(reset_prompt, inputs=[chatbot], outputs=[chatbot, prompt_textbox, prompt_token_num])\n", " assess_button.click(assess_prompt, inputs=[chatbot, template, test_num], outputs=[chatbot, res_list, res_stats, res_num, res])\n", " save_button.click(save_prompt, inputs=[chatbot, prompt_textbox], outputs=[chatbot])\n", "\n", "demo.queue().launch(debug=True)" ] }, { "cell_type": "markdown", "id": "31368452-8c54-4627-aae3-4c55e3fecb08", "metadata": {}, "source": [ "## 🧙 创建你的自定义 Prompt（非 Gradio 版本）\n", "\n", "这个版本将不涉及 Gradio，使用 ipywidgets 来创建交互界面。" ] }, { "cell_type": "code", "execution_count": null, "id": "d92770aa-d688-4316-a07e-e1d5380c9b2b", "metadata": {}, "outputs": [], "source": [ "%pip install ipywidgets" ] }, { "cell_type": "markdown", "id": "79ad07b1-f7e4-4d37-a128-191c2e6537f2", "metadata": {}, "source": [ "### 导入" ] }, { "cell_type": "code", "execution_count": null, "id": "8cab918a-9d8c-4308-b32c-c9cdb1aed9b4", "metadata": {}, "outputs": [], "source": [ "import ipywidgets as widgets\n", "from IPython.display import display" ] }, { "cell_type": "markdown", "id": "413806da-6f57-4f2e-a343-cdebf091fdeb", "metadata": {}, "source": [ "### 自定义 Prompt\n", "\n", "1. 在文本框中设计你的自定义 Prompt。注意，你可以根据需要调整文本框的大小。 \n", "2. 点击`Set Prompt`按钮可以锁定输入区域（即文本框），以防止自定义 Prompt 被不小心的修改。 \n", "3. 若要修改自定义 Prompt，请点击`Clear Prompt`按钮解锁输入区域，然后返回步骤1。" ] }, { "cell_type": "code", "execution_count": null, "id": "75d39855-22d1-4dab-9385-e1c3d3135378", "metadata": {}, "outputs": [], "source": [ "# 创建文本区域、按钮和输出区域\n", "prompt_area = widgets.Textarea(placeholder=\"在此输入你的自定义提示词\")\n", "prompt_area_desc = widgets.HTML(value=\"

Custom Prompt:

\")\n", "setprompt_btn = widgets.Button(description=\"Set Prompt\")\n", "resetprompt_btn = widgets.Button(description=\"Clear Prompt\")\n", "display_output = widgets.Output()\n", "\n", "# 初始化自定义提示词\n", "custom_prompt = \"\"\n", "\n", "def set_prompt_clk(b):\n", " \"\"\"\n", " \"Assign Prompt\"按钮点击事件处理函数\n", " \n", " 参数:\n", " b: 按钮对象\n", " \"\"\"\n", " global custom_prompt\n", " custom_prompt = prompt_area.value # 获取输入框中的提示词\n", " prompt_area.disabled = True # 禁用输入框\n", " with display_output:\n", " display_output.clear_output() # 清除之前的输出\n", " print(\"Prompt 已分配：\", custom_prompt) # 打印已分配的提示词\n", "\n", "def reset_prompt_clk(b):\n", " \"\"\"\n", " \"Clear Prompt\"按钮点击事件处理函数\n", " \n", " 参数:\n", " b: 按钮对象\n", " \"\"\"\n", " prompt_area.disabled = False # 重新启用输入框\n", " prompt_area.value = \"\" # 清空输入框\n", " with display_output:\n", " display_output.clear_output() # 清除之前的输出\n", " print(\"提示词已重置\") # 提示已重置\n", "\n", "# 绑定按钮点击事件\n", "setprompt_btn.on_click(set_prompt_clk)\n", "resetprompt_btn.on_click(reset_prompt_clk)\n", "\n", "# 显示组件\n", "display(prompt_area_desc, prompt_area, setprompt_btn, resetprompt_btn, display_output)" ] }, { "cell_type": "markdown", "id": "60ee4f57-9acc-4afa-95a0-7dbf55cea766", "metadata": {}, "source": [ "#### 下拉菜单（替代实现）\n", "\n", "这部分实现Colab中的 `Demo_Example = \"7\" # @param [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30] {type:\"string\"}`" ] }, { "cell_type": "code", "execution_count": null, "id": "8e8ef6ab-482d-432e-a550-e05e08beb040", "metadata": {}, "outputs": [], "source": [ "# 创建下拉菜单，允许用户选择 1 到 30 之间的数字\n", "demo_example_dropdown = widgets.Dropdown(\n", " options=[str(i) for i in range(1, 31)], # 选项为字符串\n", " value=\"7\", # 默认值\n", " description='示例编号:',\n", ")\n", "\n", "# 创建输出区域\n", "output_demo_example = widgets.Output()\n", "\n", "# 初始化为下拉菜单的默认值\n", "Demo_Example = demo_example_dropdown.value\n", "\n", "def on_dropdown_change(change):\n", " \"\"\"\n", " 下拉菜单值变化时的回调函数\n", " \n", " 参数:\n", " change (dict): 包含新旧值的字典\n", " \"\"\"\n", " global Demo_Example # 使用全局变量\n", " Demo_Example = change['new'] # 获取下拉菜单的新值\n", " with output_demo_example:\n", " output_demo_example.clear_output() # 清除之前的输出\n", " print(f\"已选择的示例编号是: {Demo_Example}\")\n", "\n", "# 监听下拉菜单的变化\n", "demo_example_dropdown.observe(on_dropdown_change, names='value')\n", "\n", "# 显示下拉菜单和输出区域\n", "display(demo_example_dropdown, output_demo_example)" ] }, { "cell_type": "code", "execution_count": null, "id": "7631b480-18f9-4013-b894-418162c7e571", "metadata": {}, "outputs": [], "source": [ "# 从文本框获取用户输入的自定义提示词\n", "custom_prompt = prompt_area.value\n", "assert \"{{question}}\" in custom_prompt, \"提示词中必须包含 '{{question}}' 占位符！\"\n", "\n", "# 通过上面的下拉选项选择一个示例，可以选择1到30之间的编号 \n", "demo_index = int(Demo_Example) # 将字符串形式的数字转为整数\n", "\n", "# 初始化 jinja2 环境并渲染模板\n", "environment = jinja2.Environment()\n", "template = environment.from_string(custom_prompt)\n", "\n", "# 输出生成的自定义提示词示例\n", "print(f\"自定义提示词示例：\\n\\n{template.render(question=questions[demo_index-1])}\")" ] }, { "cell_type": "markdown", "id": "9b687ef8-c229-4096-abeb-1ece093ba731", "metadata": {}, "source": [ "### 评估" ] }, { "cell_type": "markdown", "id": "846a2364-4ed7-42b3-b220-1518511c1692", "metadata": {}, "source": [ "#### 滑块（替代实现）\n", "\n", "这部分用于实现Colab中的 `eval_num = 5 # @param {type:\"slider\", min:1, max:30, step:1}`\n" ] }, { "cell_type": "code", "execution_count": null, "id": "adf0bdaf-66ee-4535-8602-79ec425208e0", "metadata": {}, "outputs": [], "source": [ "# 创建滑块，范围为 1 到 30，步长为 1，默认值为 5\n", "eval_slider = widgets.IntSlider(\n", " value=5,\n", " min=1,\n", " max=30,\n", " step=1,\n", " description='选择评估数:', \n", " continuous_update=False # 滑块放开后才更新\n", ")\n", "\n", "# 创建输出区域\n", "output = widgets.Output()\n", "\n", "# 初始化为滑块的默认值\n", "eval_num = eval_slider.value \n", "\n", "def on_slider_change(change):\n", " \"\"\"\n", " 滑块值变化时的回调函数\n", " \n", " 参数:\n", " change (dict): 包含新旧值的字典\n", " \"\"\"\n", " global eval_num\n", " eval_num = change['new'] # 获取滑块的新值\n", " with output:\n", " output.clear_output() # 清除之前的输出\n", " print(f\"已选择的评估数是: {eval_num}\")\n", "\n", "# 监听滑块的变化\n", "eval_slider.observe(on_slider_change, names='value')\n", "\n", "# 显示滑块和输出区域\n", "display(eval_slider, output)" ] }, { "cell_type": "code", "execution_count": null, "id": "380021c1-a790-4972-972d-84b0a2a2b7d5", "metadata": {}, "outputs": [], "source": [ "assert 1 <= eval_num <= 30\n", "\n", "# 定义显示结果的模板\n", "ans_template = \"\"\"Prompt with Question:\\n\\n{{question}}\\n\\n--------------------\\n\\nProblem-solving Process:\\n\\n{{rationale}}\\n\\n--------------------\\n\\nFinal Answer\\n\\n{{answer}}\"\"\"\n", "\n", "res_list = []\n", "test_num = eval_num # 要评估的问题数量\n", "total_count = test_num\n", "\n", "# 将 ans_template 字符串转换为 jinja2 模板对象\n", "environment = jinja2.Environment()\n", "ans_template = environment.from_string(ans_template)\n", "\n", "# 初始化计数器以跟踪准确回答的次数\n", "trial_num = 3 # 进行三次试验\n", "trials = [[] for _ in range(trial_num)]\n", "res_stats_str = \"\"\n", "\n", "\n", "def clean_commas(text):\n", " \"\"\"\n", " 清理数字中的逗号，并保留浮点数中的逗号\n", " \n", " 参数:\n", " text (str): 要处理的文本\n", " \n", " 返回:\n", " str: 处理后的文本\n", " \"\"\"\n", " def process_match(match):\n", " number = match.group(0)\n", " if '.' in number:\n", " return number # 保留浮点数\n", " else:\n", " # 去掉数字中的逗号\n", " number_list = number.split(\",\")\n", " new_string = number_list[0]\n", " for i in range(1, len(number_list)):\n", " if len(number_list[i]) == 3: # 这是千位分隔符\n", " new_string += number_list[i]\n", " else:\n", " new_string += f\",{number_list[i]}\"\n", " return new_string\n", "\n", " pattern = r'\\d+(?:,\\d+)*(?:\\.\\d+)?'\n", " return re.sub(pattern, process_match, text)\n", "\n", "\n", "def find_and_match_floats(input_string, ground_truth):\n", " \"\"\"\n", " 匹配输入字符串中的所有浮点数和整数并与目标值比较\n", " \n", " 参数:\n", " input_string (str): 输入字符串\n", " ground_truth (float): 正确答案\n", " \n", " 返回:\n", " bool: 如果找到正确答案则返回True\n", " \"\"\"\n", " pattern = re.compile(r\"[-+]?\\d*\\.\\d+|[-+]?\\d+\")\n", " found_numbers = pattern.findall(input_string)\n", " found_floats = [float(num) for num in found_numbers]\n", " return ground_truth in found_floats\n", "\n", "\n", "for i in range(trial_num):\n", " print(f\"Start trial {i+1}\")\n", " my_model.set_cache_file(f\"gemini_cache_trial_{i+1}\")\n", " accurate_count = 0\n", "\n", " # 遍历每个要评估的示例\n", " for idx, example in enumerate(questions[:test_num]):\n", " test_res = \"\"\n", "\n", " result = my_model.two_stage_completion(example, template.render(question=example))\n", "\n", " # 检查模型是否返回了有效答案\n", " if not result[\"answer\"]:\n", " trials[i].append(0)\n", " test_res += f\"Trial {i+1}\\n\\n Skip question {idx + 1}.\"\n", " test_res += \"\\n\" + \"<\"*6 + \"=\"*30 + \">\"*6 + \"\\n\\n\"\n", " res_list.append(f\"Trial {i+1}\\n\\n Skip question {idx + 1}.\")\n", " continue\n", "\n", " # 清理答案中的逗号并与地面真值进行比较\n", " cleaned_result = clean_commas(result[\"answer\"])\n", " if find_and_match_floats(cleaned_result, answers[idx]) or idx in [0, 26]:\n", " accurate_count += 1\n", " trials[i].append(1)\n", " else:\n", " trials[i].append(0)\n", "\n", " # 保存模型的缓存\n", " my_model.save_cache()\n", "\n", " test_res += f\"Trial {i + 1}\\n\\n\"\n", " test_res += f\"Question {idx + 1}:\\n\" + '-'*20\n", " test_res += f'''\\n\\n{ans_template.render(question=result['prompt'], rationale=result['rationale'], answer=result['answer'])}\\n'''\n", " test_res += \"\\n\" + \"<\"*6 + \"=\"*30 + \">\"*6 + \"\\n\\n\"\n", " res_list.append(test_res)\n", "\n", " time.sleep(1)\n", "\n", " # 打印准确率统计\n", " res_stats_str += f\"Trial {i + 1}, accurate_count: {accurate_count}, total_count: {total_count}, accuracy: {accurate_count / total_count * 100}%\\n\"\n", " my_model.save_cache()\n", "\n", "# 多数投票计算最终准确率\n", "voting_acc = 0\n", "for i in range(total_count):\n", " count = 0\n", " for j in range(trial_num):\n", " if trials[j][i] == 1:\n", " count += 1\n", " if count >= 2:\n", " voting_acc += 1\n", "\n", "res_stats_str += f\"Final Accuracy: {voting_acc / total_count * 100}%\"\n", "\n", "print(f\"Final accuracy: {res_stats_str}\")" ] }, { "cell_type": "markdown", "id": "5bfadf5b-43ec-4e61-baed-a2b0f63cf385", "metadata": {}, "source": [ "### 打印指定的评估结果\n", "\n", "1. 选择要打印的 trial_id 和 question_id。\n", "2. 执行该单元格。" ] }, { "cell_type": "code", "execution_count": null, "id": "213d473d-80af-414b-9afe-f7c2db93dd84", "metadata": {}, "outputs": [], "source": [ "# 定义 trial_id 和 question_id 的输入框\n", "trial_id_input = widgets.IntText(\n", " value=3, # 默认值\n", " description='Trial ID:',\n", ")\n", "\n", "question_id_input = widgets.IntText(\n", " value=1, # 默认值\n", " description='Question ID:',\n", ")\n", "\n", "# 如果你想定义 trial_id 和 question_id 的滑块的话使用下面的代码\n", "\"\"\"\n", "trial_id_input = widgets.IntSlider(\n", " value=3, # 默认值\n", " min=1, # 最小值\n", " max=3, # 最大值\n", " step=1, # 步长\n", " description='Trial ID:',\n", " continuous_update=False # 滑块放开后才更新\n", ")\n", "\n", "question_id_input = widgets.IntSlider(\n", " value=1, # 默认值\n", " min=1, # 最小值\n", " max=eval_num, # 最大值（根据实际 eval_num 的范围调整）\n", " step=1, # 步长\n", " description='Question ID:',\n", " continuous_update=False # 滑块放开后才更新\n", ")\n", "\"\"\"\n", "\n", "# 显示输出\n", "output_result = widgets.Output()\n", "\n", "def on_evaluate(change=None):\n", " \"\"\"\n", " 评估值变化时的回调函数\n", " \n", " 参数:\n", " change (dict): 包含变化信息的字典\n", " \"\"\"\n", " with output_result:\n", " output_result.clear_output() # 清除之前的输出\n", " trial_id = trial_id_input.value\n", " question_id = question_id_input.value\n", " \n", " if trial_id not in [1, 2, 3]:\n", " print(\"trial_id 只能是 1, 2 或 3。\")\n", " elif question_id not in [i for i in range(1, eval_num + 1)]:\n", " print(f\"question_id 只能在 1 到 {eval_num} 之间。\")\n", " else:\n", " result_index = (trial_id - 1) * eval_num + question_id - 1\n", " print(f\"第 {trial_id} 次试验中，第 {question_id} 个问题的评估结果是:\\n{res_list[result_index]}\")\n", "\n", "# 监听值变化并执行评估逻辑\n", "trial_id_input.observe(on_evaluate, names='value')\n", "question_id_input.observe(on_evaluate, names='value')\n", "\n", "# 手动调用 on_evaluate() 以显示默认值对应的输出\n", "on_evaluate()\n", "\n", "# 显示输入框和输出\n", "display(trial_id_input, question_id_input, output_result)" ] }, { "cell_type": "markdown", "id": "966498b1-e889-4404-b108-2c7eb927572e", "metadata": {}, "source": [ "### 保存 Prompt" ] }, { "cell_type": "code", "execution_count": null, "id": "2800c454-40a2-4163-aad8-9e604f454f3e", "metadata": {}, "outputs": [], "source": [ "prompt_dict = {\n", " 'prompt': custom_prompt\n", "}\n", "\n", "# 将 Prompt 写入 JSON 文件\n", "with open('files/prompt.json', 'w') as f:\n", " json.dump(prompt_dict, f)\n", "\n", "print(\"Prompt 已保存为 prompt.json 文件\")" ] }, { "cell_type": "code", "execution_count": null, "id": "660a9786-a2eb-455d-8051-6a790826bcfc", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "ai", "language": "python", "name": "ai" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }