{ "cells": [ { "cell_type": "markdown", "id": "186f8d20-3165-4ea1-96d3-6dfa9b8859b4", "metadata": {}, "source": [ "# b. 使用 Llama-cpp-python 加载量化后的 LLM 大模型（GGUF）\n", "\n", "> 引导文章：[19b. 从加载到对话：使用 Llama-cpp-python 本地运行量化 LLM 大模型（GGUF）](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/19b.%20从加载到对话：使用%20Llama-cpp-python%20本地运行量化%20LLM%20大模型（GGUF）.md)\n", "\n", "代码文件没有显卡要求，在个人计算机上均可进行对话。\n", "\n", "**模型文件约为 4 GB**。\n", "\n", "这里还有一个简单的 [🎡 AI Chat 脚本](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/CodePlayground/chat.py)供你尝试，详见：[CodePlayground](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/CodePlayground/README.md#当前的玩具)，点击 `►` 或对应的文本展开。\n", "\n", "Transformers 关于 GPTQ & AWQ 加载的相关链接：[文章 19a](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Guide/19a.%20从加载到对话：使用%20Transformers%20本地运行量化%20LLM%20大模型（GPTQ%20%26%20AWQ）.md) | [代码文件 16a](https://github.com/Hoper-J/AI-Guide-and-Demos-zh_CN/blob/master/Demos/16a.%20使用%20Transformers%20加载量化后的%20LLM%20大模型（GPTQ%20%26%20AWQ）.ipynb)\n", "\n", "在线链接：[Kaggle - b](https://www.kaggle.com/code/aidemos/16b-llama-cpp-python-llm-gguf) | [Colab - b](https://colab.research.google.com/drive/1AhgC0qDaqWBXAI9eSbwTStGgvgFfLOpf?usp=sharing)" ] }, { "cell_type": "markdown", "id": "7a3d1707-d7e2-4d0b-8fa6-807892c52bc2", "metadata": {}, "source": [ "## Llama-cpp-python " ] }, { "cell_type": "markdown", "id": "b34d8b2a", "metadata": {}, "source": [ "### 环境配置\n", "\n", "为了确保之后的 \"offload\" 正常工作，需要进行一些额外的工作。\n", "\n", "首先，找到 CUDA 的安装路径：" ] }, { "cell_type": "code", "execution_count": null, "id": "d83c9b96", "metadata": {}, "outputs": [], "source": [ "!find /usr/local -name \"cuda\" -exec readlink -f {} \\;" ] }, { "cell_type": "markdown", "id": "48546e57", "metadata": {}, "source": [ "复制对应（最短）的路径，修改 `cuda_home` 环境变量：" ] }, { "cell_type": "code", "execution_count": null, "id": "55d687df", "metadata": { "scrolled": true }, "outputs": [], "source": [ "import subprocess\n", "import os\n", "\n", "# 设置CUDA路径\n", "cuda_home = \"/usr/local/cuda-12.4\" # 请替换为你的路径\n", "\n", "# 检测GPU架构并转换为整数格式\n", "try:\n", " result = subprocess.run([\"nvidia-smi\", \"--query-gpu=name,compute_cap\", \"--format=csv,noheader,nounits\"], \n", " capture_output=True, text=True, check=True)\n", " gpu_info = result.stdout.strip().split('\\n')[0].strip()\n", " gpu_name, compute_cap = gpu_info.split(', ', 1)\n", " # 转换为整数格式：8.9 -> 89, 8.6 -> 86, 7.5 -> 75\n", " arch = compute_cap.replace('.', '')\n", " print(f\"检测到显卡: {gpu_name}\")\n", " print(f\"GPU架构: {compute_cap} -> {arch}\")\n", "except:\n", " arch = \"86\" # 最常用的默认架构\n", " print(\"GPU检测失败，使用默认架构: 86\")\n", "\n", "# 构建安装命令\n", "command = f\"\"\"\n", "CMAKE_ARGS=\"-DGGML_CUDA=on \\\n", " -DCUDA_PATH={cuda_home} \\\n", " -DCUDAToolkit_ROOT={cuda_home} \\\n", " -DCUDAToolkit_INCLUDE_DIR={cuda_home}/include \\\n", " -DCUDAToolkit_LIBRARY_DIR={cuda_home}/lib64 \\\n", " -DCMAKE_CUDA_COMPILER={cuda_home}/bin/nvcc \\\n", " -DCMAKE_CUDA_ARCHITECTURES={arch} \\\n", " -DGGML_CUDA_FORCE_DMMV=on\" \\\n", "CUDACXX={cuda_home}/bin/nvcc \\\n", "LD_LIBRARY_PATH=\"{cuda_home}/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/lib:$LD_LIBRARY_PATH\" \\\n", "FORCE_CMAKE=1 \\\n", "pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir --verbose\n", "\"\"\"\n", "\n", "subprocess.run(command, shell=True, check=True)" ] }, { "cell_type": "markdown", "id": "d352a092", "metadata": {}, "source": [ "### GGUF" ] }, { "cell_type": "markdown", "id": "f5047589", "metadata": {}, "source": [ "#### 安装库" ] }, { "cell_type": "code", "execution_count": 1, "id": "dd3af47c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n", "Requirement already satisfied: gguf in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (0.19.0)\n", "Requirement already satisfied: numpy>=1.17 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from gguf) (2.2.6)\n", "Requirement already satisfied: pyyaml>=5.1 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from gguf) (6.0.3)\n", "Requirement already satisfied: requests>=2.25 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from gguf) (2.33.1)\n", "Requirement already satisfied: tqdm>=4.27 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from gguf) (4.67.3)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.25->gguf) (3.4.7)\n", "Requirement already satisfied: idna<4,>=2.5 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.25->gguf) (3.11)\n", "Requirement already satisfied: urllib3<3,>=1.26 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.25->gguf) (2.6.3)\n", "Requirement already satisfied: certifi>=2023.5.7 in /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/.venv/lib/python3.12/site-packages (from requests>=2.25->gguf) (2026.2.25)\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install gguf" ] }, { "cell_type": "markdown", "id": "368d751d", "metadata": {}, "source": [ "#### 导入库" ] }, { "cell_type": "code", "execution_count": 2, "id": "460326a9", "metadata": {}, "outputs": [], "source": [ "import os\n", "# 设置模型下载镜像（注意，需要在导入 transformers 等模块前进行设置才能起效）\n", "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n", "\n", "from llama_cpp import Llama" ] }, { "cell_type": "markdown", "id": "a5bd1c6e", "metadata": {}, "source": [ "下面介绍两种导入模型的方法，实际执行时本地/自动导入二选一。" ] }, { "cell_type": "markdown", "id": "52283fcc", "metadata": {}, "source": [ "#### 本地导入模型\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "6fe8627e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0;32mUsing cached metadata: Mistral-7B-Instruct-v0.3-GGUF/.hfd/repo_metadata.json\u001b[0m\n", "\u001b[0;32mResume from file list: Mistral-7B-Instruct-v0.3-GGUF/.hfd/aria2c_urls.txt\u001b[0m\n", "\u001b[1;33mStarting download with aria2c to Mistral-7B-Instruct-v0.3-GGUF...\n", "\u001b[0mNo files to download.\n", "\u001b[0;32mDownload completed successfully. Repo directory: /root/autodl-tmp/AI-Guide-and-Demos-zh_CN/Demos/Mistral-7B-Instruct-v0.3-GGUF\n", "\u001b[0m" ] } ], "source": [ "# 如果你已经配置过了，可以直接在 Notebook 中执行下面的命令下载，也可以跳过\n", "!export HF_ENDPOINT=https://hf-mirror.com\n", "!./hfd.sh bartowski/Mistral-7B-Instruct-v0.3-GGUF --include Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --tool aria2c -x 8" ] }, { "cell_type": "markdown", "id": "05857047", "metadata": {}, "source": [ "根据模型路径导入模型，注意，文件位于 `` 文件夹下，以当前下载的文件为例：" ] }, { "cell_type": "code", "execution_count": 4, "id": "a0b244a9", "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24080 MiB):\n", " Device 0: NVIDIA GeForce RTX 4090 D, compute capability 8.9, VMM: yes, VRAM: 24080 MiB\n", "llama_model_loader: loaded meta data with 29 key-value pairs and 291 tensors from ./Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf (version GGUF V3 (latest))\n", "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n", "llama_model_loader: - kv 0: general.architecture str = llama\n", "llama_model_loader: - kv 1: general.name str = Mistral-7B-Instruct-v0.3\n", "llama_model_loader: - kv 2: llama.block_count u32 = 32\n", "llama_model_loader: - kv 3: llama.context_length u32 = 32768\n", "llama_model_loader: - kv 4: llama.embedding_length u32 = 4096\n", "llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336\n", "llama_model_loader: - kv 6: llama.attention.head_count u32 = 32\n", "llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8\n", "llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000\n", "llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010\n", "llama_model_loader: - kv 10: general.file_type u32 = 15\n", "llama_model_loader: - kv 11: llama.vocab_size u32 = 32768\n", "llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128\n", "llama_model_loader: - kv 13: tokenizer.ggml.model str = llama\n", "llama_model_loader: - kv 14: tokenizer.ggml.pre str = default\n", "llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32768] = [\"\", \"\", \"\", \"[INST]\", \"[...\n", "llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32768] = [0.000000, 0.000000, 0.000000, 0.0000...\n", "llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32768] = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n", "llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1\n", "llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 2\n", "llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0\n", "llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true\n", "llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false\n", "llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...\n", "llama_model_loader: - kv 24: general.quantization_version u32 = 2\n", "llama_model_loader: - kv 25: quantize.imatrix.file str = /models/Mistral-7B-Instruct-v0.3-GGUF...\n", "llama_model_loader: - kv 26: quantize.imatrix.dataset str = /training_data/calibration_data.txt\n", "llama_model_loader: - kv 27: quantize.imatrix.entries_count i32 = 224\n", "llama_model_loader: - kv 28: quantize.imatrix.chunks_count i32 = 228\n", "llama_model_loader: - type f32: 65 tensors\n", "llama_model_loader: - type q4_K: 193 tensors\n", "llama_model_loader: - type q6_K: 33 tensors\n", "print_info: file format = GGUF V3 (latest)\n", "print_info: file type = Q4_K - Medium\n", "print_info: file size = 4.07 GiB (4.83 BPW) \n", "llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4090 D) (0000:d6:00.0) - 23724 MiB free\n", "init_tokenizer: initializing tokenizer for type 1\n", "load: 0 unused tokens\n", "load: control token: 468 '[control_466]' is not marked as EOG\n", "load: control token: 464 '[control_462]' is not marked as EOG\n", "load: control token: 727 '[control_725]' is not marked as EOG\n", "load: control token: 343 '[control_341]' is not marked as EOG\n", "load: control token: 603 '[control_601]' is not marked as EOG\n", "load: control token: 332 '[control_330]' is not marked as EOG\n", "load: control token: 34 '[control_32]' is not marked as EOG\n", "load: control token: 412 '[control_410]' is not marked as EOG\n", "load: control token: 675 '[control_673]' is not marked as EOG\n", "load: control token: 177 '[control_175]' is not marked as EOG\n", "load: control token: 434 '[control_432]' is not marked as EOG\n", "load: control token: 166 '[control_164]' is not marked as EOG\n", "load: control token: 199 '[control_197]' is not marked as EOG\n", "load: control token: 252 '[control_250]' is not marked as EOG\n", "load: control token: 742 '[control_740]' is not marked as EOG\n", "load: control token: 611 '[control_609]' is not marked as EOG\n", "load: control token: 573 '[control_571]' is not marked as EOG\n", "load: control token: 455 '[control_453]' is not marked as EOG\n", "load: control token: 701 '[control_699]' is not marked as EOG\n", "load: control token: 583 '[control_581]' is not marked as EOG\n", "load: control token: 433 '[control_431]' is not marked as EOG\n", "load: control token: 301 '[control_299]' is not marked as EOG\n", "load: control token: 617 '[control_615]' is not marked as EOG\n", "load: control token: 536 '[control_534]' is not marked as EOG\n", "load: control token: 285 '[control_283]' is not marked as EOG\n", "load: control token: 376 '[control_374]' is not marked as EOG\n", "load: control token: 79 '[control_77]' is not marked as EOG\n", "load: control token: 671 '[control_669]' is not marked as EOG\n", "load: control token: 391 '[control_389]' is not marked as EOG\n", "load: control token: 174 '[control_172]' is not marked as EOG\n", "load: control token: 708 '[control_706]' is not marked as EOG\n", "load: control token: 729 '[control_727]' is not marked as EOG\n", "load: control token: 311 '[control_309]' is not marked as EOG\n", "load: control token: 384 '[control_382]' is not marked as EOG\n", "load: control token: 766 '[control_764]' is not marked as EOG\n", "load: control token: 94 '[control_92]' is not marked as EOG\n", "load: control token: 304 '[control_302]' is not marked as EOG\n", "load: control token: 197 '[control_195]' is not marked as EOG\n", "load: control token: 17 '[control_15]' is not marked as EOG\n", "load: control token: 604 '[control_602]' is not marked as EOG\n", "load: control token: 576 '[control_574]' is not marked as EOG\n", "load: control token: 359 '[control_357]' is not marked as EOG\n", "load: control token: 302 '[control_300]' is not marked as EOG\n", "load: control token: 453 '[control_451]' is not marked as EOG\n", "load: control token: 374 '[control_372]' is not marked as EOG\n", "load: control token: 207 '[control_205]' is not marked as EOG\n", "load: control token: 739 '[control_737]' is not marked as EOG\n", "load: control token: 689 '[control_687]' is not marked as EOG\n", "load: control token: 335 '[control_333]' is not marked as EOG\n", "load: control token: 485 '[control_483]' is not marked as EOG\n", "load: control token: 283 '[control_281]' is not marked as EOG\n", "load: control token: 249 '[control_247]' is not marked as EOG\n", "load: control token: 752 '[control_750]' is not marked as EOG\n", "load: control token: 330 '[control_328]' is not marked as EOG\n", "load: control token: 635 '[control_633]' is not marked as EOG\n", "load: control token: 161 '[control_159]' is not marked as EOG\n", "load: control token: 731 '[control_729]' is not marked as EOG\n", "load: control token: 631 '[control_629]' is not marked as EOG\n", "load: control token: 323 '[control_321]' is not marked as EOG\n", "load: control token: 48 '[control_46]' is not marked as EOG\n", "load: control token: 544 '[control_542]' is not marked as EOG\n", "load: control token: 687 '[control_685]' is not marked as EOG\n", "load: control token: 463 '[control_461]' is not marked as EOG\n", "load: control token: 47 '[control_45]' is not marked as EOG\n", "load: control token: 86 '[control_84]' is not marked as EOG\n", "load: control token: 331 '[control_329]' is not marked as EOG\n", "load: control token: 13 '[control_11]' is not marked as EOG\n", "load: control token: 760 '[control_758]' is not marked as EOG\n", "load: control token: 612 '[control_610]' is not marked as EOG\n", "load: control token: 473 '[control_471]' is not marked as EOG\n", "load: control token: 601 '[control_599]' is not marked as EOG\n", "load: control token: 146 '[control_144]' is not marked as EOG\n", "load: control token: 734 '[control_732]' is not marked as EOG\n", "load: control token: 141 '[control_139]' is not marked as EOG\n", "load: control token: 138 '[control_136]' is not marked as EOG\n", "load: control token: 61 '[control_59]' is not marked as EOG\n", "load: control token: 46 '[control_44]' is not marked as EOG\n", "load: control token: 173 '[control_171]' is not marked as EOG\n", "load: control token: 367 '[control_365]' is not marked as EOG\n", "load: control token: 417 '[control_415]' is not marked as EOG\n", "load: control token: 740 '[control_738]' is not marked as EOG\n", "load: control token: 216 '[control_214]' is not marked as EOG\n", "load: control token: 498 '[control_496]' is not marked as EOG\n", "load: control token: 512 '[control_510]' is not marked as EOG\n", "load: control token: 122 '[control_120]' is not marked as EOG\n", "load: control token: 246 '[control_244]' is not marked as EOG\n", "load: control token: 745 '[control_743]' is not marked as EOG\n", "load: control token: 451 '[control_449]' is not marked as EOG\n", "load: control token: 280 '[control_278]' is not marked as EOG\n", "load: control token: 654 '[control_652]' is not marked as EOG\n", "load: control token: 679 '[control_677]' is not marked as EOG\n", "load: control token: 77 '[control_75]' is not marked as EOG\n", "load: control token: 638 '[control_636]' is not marked as EOG\n", "load: control token: 472 '[control_470]' is not marked as EOG\n", "load: control token: 59 '[control_57]' is not marked as EOG\n", "load: control token: 145 '[control_143]' is not marked as EOG\n", "load: control token: 318 '[control_316]' is not marked as EOG\n", "load: control token: 640 '[control_638]' is not marked as EOG\n", "load: control token: 690 '[control_688]' is not marked as EOG\n", "load: control token: 256 '[control_254]' is not marked as EOG\n", "load: control token: 476 '[control_474]' is not marked as EOG\n", "load: control token: 21 '[control_19]' is not marked as EOG\n", "load: control token: 288 '[control_286]' is not marked as EOG\n", "load: control token: 255 '[control_253]' is not marked as EOG\n", "load: control token: 113 '[control_111]' is not marked as EOG\n", "load: control token: 190 '[control_188]' is not marked as EOG\n", "load: control token: 108 '[control_106]' is not marked as EOG\n", "load: control token: 211 '[control_209]' is not marked as EOG\n", "load: control token: 551 '[control_549]' is not marked as EOG\n", "load: control token: 14 '[control_12]' is not marked as EOG\n", "load: control token: 737 '[control_735]' is not marked as EOG\n", "load: control token: 555 '[control_553]' is not marked as EOG\n", "load: control token: 227 '[control_225]' is not marked as EOG\n", "load: control token: 210 '[control_208]' is not marked as EOG\n", "load: control token: 230 '[control_228]' is not marked as EOG\n", "load: control token: 7 '[/AVAILABLE_TOOLS]' is not marked as EOG\n", "load: control token: 55 '[control_53]' is not marked as EOG\n", "load: control token: 525 '[control_523]' is not marked as EOG\n", "load: control token: 533 '[control_531]' is not marked as EOG\n", "load: control token: 195 '[control_193]' is not marked as EOG\n", "load: control token: 584 '[control_582]' is not marked as EOG\n", "load: control token: 203 '[control_201]' is not marked as EOG\n", "load: control token: 322 '[control_320]' is not marked as EOG\n", "load: control token: 757 '[control_755]' is not marked as EOG\n", "load: control token: 642 '[control_640]' is not marked as EOG\n", "load: control token: 168 '[control_166]' is not marked as EOG\n", "load: control token: 389 '[control_387]' is not marked as EOG\n", "load: control token: 368 '[control_366]' is not marked as EOG\n", "load: control token: 767 '[control_765]' is not marked as EOG\n", "load: control token: 222 '[control_220]' is not marked as EOG\n", "load: control token: 420 '[control_418]' is not marked as EOG\n", "load: control token: 530 '[control_528]' is not marked as EOG\n", "load: control token: 535 '[control_533]' is not marked as EOG\n", "load: control token: 765 '[control_763]' is not marked as EOG\n", "load: control token: 661 '[control_659]' is not marked as EOG\n", "load: control token: 88 '[control_86]' is not marked as EOG\n", "load: control token: 581 '[control_579]' is not marked as EOG\n", "load: control token: 327 '[control_325]' is not marked as EOG\n", "load: control token: 201 '[control_199]' is not marked as EOG\n", "load: control token: 115 '[control_113]' is not marked as EOG\n", "load: control token: 709 '[control_707]' is not marked as EOG\n", "load: control token: 291 '[control_289]' is not marked as EOG\n", "load: control token: 265 '[control_263]' is not marked as EOG\n", "load: control token: 148 '[control_146]' is not marked as EOG\n", "load: control token: 185 '[control_183]' is not marked as EOG\n", "load: control token: 574 '[control_572]' is not marked as EOG\n", "load: control token: 360 '[control_358]' is not marked as EOG\n", "load: control token: 127 '[control_125]' is not marked as EOG\n", "load: control token: 325 '[control_323]' is not marked as EOG\n", "load: control token: 183 '[control_181]' is not marked as EOG\n", "load: control token: 116 '[control_114]' is not marked as EOG\n", "load: control token: 540 '[control_538]' is not marked as EOG\n", "load: control token: 293 '[control_291]' is not marked as EOG\n", "load: control token: 559 '[control_557]' is not marked as EOG\n", "load: control token: 527 '[control_525]' is not marked as EOG\n", "load: control token: 156 '[control_154]' is not marked as EOG\n", "load: control token: 338 '[control_336]' is not marked as EOG\n", "load: control token: 519 '[control_517]' is not marked as EOG\n", "load: control token: 516 '[control_514]' is not marked as EOG\n", "load: control token: 10 '[control_8]' is not marked as EOG\n", "load: control token: 8 '[TOOL_RESULTS]' is not marked as EOG\n", "load: control token: 505 '[control_503]' is not marked as EOG\n", "load: control token: 503 '[control_501]' is not marked as EOG\n", "load: control token: 500 '[control_498]' is not marked as EOG\n", "load: control token: 496 '[control_494]' is not marked as EOG\n", "load: control token: 492 '[control_490]' is not marked as EOG\n", "load: control token: 489 '[control_487]' is not marked as EOG\n", "load: control token: 538 '[control_536]' is not marked as EOG\n", "load: control token: 596 '[control_594]' is not marked as EOG\n", "load: control token: 481 '[control_479]' is not marked as EOG\n", "load: control token: 475 '[control_473]' is not marked as EOG\n", "load: control token: 336 '[control_334]' is not marked as EOG\n", "load: control token: 670 '[control_668]' is not marked as EOG\n", "load: control token: 50 '[control_48]' is not marked as EOG\n", "load: control token: 456 '[control_454]' is not marked as EOG\n", "load: control token: 105 '[control_103]' is not marked as EOG\n", "load: control token: 421 '[control_419]' is not marked as EOG\n", "load: control token: 430 '[control_428]' is not marked as EOG\n", "load: control token: 429 '[control_427]' is not marked as EOG\n", "load: control token: 575 '[control_573]' is not marked as EOG\n", "load: control token: 425 '[control_423]' is not marked as EOG\n", "load: control token: 424 '[control_422]' is not marked as EOG\n", "load: control token: 680 '[control_678]' is not marked as EOG\n", "load: control token: 57 '[control_55]' is not marked as EOG\n", "load: control token: 356 '[control_354]' is not marked as EOG\n", "load: control token: 458 '[control_456]' is not marked as EOG\n", "load: control token: 313 '[control_311]' is not marked as EOG\n", "load: control token: 418 '[control_416]' is not marked as EOG\n", "load: control token: 70 '[control_68]' is not marked as EOG\n", "load: control token: 759 '[control_757]' is not marked as EOG\n", "load: control token: 416 '[control_414]' is not marked as EOG\n", "load: control token: 238 '[control_236]' is not marked as EOG\n", "load: control token: 568 '[control_566]' is not marked as EOG\n", "load: control token: 409 '[control_407]' is not marked as EOG\n", "load: control token: 550 '[control_548]' is not marked as EOG\n", "load: control token: 571 '[control_569]' is not marked as EOG\n", "load: control token: 618 '[control_616]' is not marked as EOG\n", "load: control token: 623 '[control_621]' is not marked as EOG\n", "load: control token: 247 '[control_245]' is not marked as EOG\n", "load: control token: 400 '[control_398]' is not marked as EOG\n", "load: control token: 396 '[control_394]' is not marked as EOG\n", "load: control token: 392 '[control_390]' is not marked as EOG\n", "load: control token: 552 '[control_550]' is not marked as EOG\n", "load: control token: 651 '[control_649]' is not marked as EOG\n", "load: control token: 390 '[control_388]' is not marked as EOG\n", "load: control token: 186 '[control_184]' is not marked as EOG\n", "load: control token: 565 '[control_563]' is not marked as EOG\n", "load: control token: 92 '[control_90]' is not marked as EOG\n", "load: control token: 508 '[control_506]' is not marked as EOG\n", "load: control token: 373 '[control_371]' is not marked as EOG\n", "load: control token: 299 '[control_297]' is not marked as EOG\n", "load: control token: 562 '[control_560]' is not marked as EOG\n", "load: control token: 537 '[control_535]' is not marked as EOG\n", "load: control token: 372 '[control_370]' is not marked as EOG\n", "load: control token: 366 '[control_364]' is not marked as EOG\n", "load: control token: 196 '[control_194]' is not marked as EOG\n", "load: control token: 478 '[control_476]' is not marked as EOG\n", "load: control token: 621 '[control_619]' is not marked as EOG\n", "load: control token: 53 '[control_51]' is not marked as EOG\n", "load: control token: 40 '[control_38]' is not marked as EOG\n", "load: control token: 347 '[control_345]' is not marked as EOG\n", "load: control token: 345 '[control_343]' is not marked as EOG\n", "load: control token: 339 '[control_337]' is not marked as EOG\n", "load: control token: 234 '[control_232]' is not marked as EOG\n", "load: control token: 770 '[control_768]' is not marked as EOG\n", "load: control token: 444 '[control_442]' is not marked as EOG\n", "load: control token: 317 '[control_315]' is not marked as EOG\n", "load: control token: 693 '[control_691]' is not marked as EOG\n", "load: control token: 321 '[control_319]' is not marked as EOG\n", "load: control token: 320 '[control_318]' is not marked as EOG\n", "load: control token: 179 '[control_177]' is not marked as EOG\n", "load: control token: 214 '[control_212]' is not marked as EOG\n", "load: control token: 666 '[control_664]' is not marked as EOG\n", "load: control token: 712 '[control_710]' is not marked as EOG\n", "load: control token: 303 '[control_301]' is not marked as EOG\n", "load: control token: 440 '[control_438]' is not marked as EOG\n", "load: control token: 314 '[control_312]' is not marked as EOG\n", "load: control token: 397 '[control_395]' is not marked as EOG\n", "load: control token: 312 '[control_310]' is not marked as EOG\n", "load: control token: 129 '[control_127]' is not marked as EOG\n", "load: control token: 545 '[control_543]' is not marked as EOG\n", "load: control token: 58 '[control_56]' is not marked as EOG\n", "load: control token: 509 '[control_507]' is not marked as EOG\n", "load: control token: 541 '[control_539]' is not marked as EOG\n", "load: control token: 443 '[control_441]' is not marked as EOG\n", "load: control token: 45 '[control_43]' is not marked as EOG\n", "load: control token: 469 '[control_467]' is not marked as EOG\n", "load: control token: 532 '[control_530]' is not marked as EOG\n", "load: control token: 64 '[control_62]' is not marked as EOG\n", "load: control token: 365 '[control_363]' is not marked as EOG\n", "load: control token: 563 '[control_561]' is not marked as EOG\n", "load: control token: 346 '[control_344]' is not marked as EOG\n", "load: control token: 282 '[control_280]' is not marked as EOG\n", "load: control token: 450 '[control_448]' is not marked as EOG\n", "load: control token: 526 '[control_524]' is not marked as EOG\n", "load: control token: 672 '[control_670]' is not marked as EOG\n", "load: control token: 232 '[control_230]' is not marked as EOG\n", "load: control token: 273 '[control_271]' is not marked as EOG\n", "load: control token: 272 '[control_270]' is not marked as EOG\n", "load: control token: 636 '[control_634]' is not marked as EOG\n", "load: control token: 260 '[control_258]' is not marked as EOG\n", "load: control token: 518 '[control_516]' is not marked as EOG\n", "load: control token: 353 '[control_351]' is not marked as EOG\n", "load: control token: 257 '[control_255]' is not marked as EOG\n", "load: control token: 126 '[control_124]' is not marked as EOG\n", "load: control token: 124 '[control_122]' is not marked as EOG\n", "load: control token: 751 '[control_749]' is not marked as EOG\n", "load: control token: 383 '[control_381]' is not marked as EOG\n", "load: control token: 175 '[control_173]' is not marked as EOG\n", "load: control token: 75 '[control_73]' is not marked as EOG\n", "load: control token: 72 '[control_70]' is not marked as EOG\n", "load: control token: 614 '[control_612]' is not marked as EOG\n", "load: control token: 167 '[control_165]' is not marked as EOG\n", "load: control token: 465 '[control_463]' is not marked as EOG\n", "load: control token: 341 '[control_339]' is not marked as EOG\n", "load: control token: 703 '[control_701]' is not marked as EOG\n", "load: control token: 501 '[control_499]' is not marked as EOG\n", "load: control token: 90 '[control_88]' is not marked as EOG\n", "load: control token: 89 '[control_87]' is not marked as EOG\n", "load: control token: 593 '[control_591]' is not marked as EOG\n", "load: control token: 107 '[control_105]' is not marked as EOG\n", "load: control token: 241 '[control_239]' is not marked as EOG\n", "load: control token: 713 '[control_711]' is not marked as EOG\n", "load: control token: 212 '[control_210]' is not marked as EOG\n", "load: control token: 80 '[control_78]' is not marked as EOG\n", "load: control token: 707 '[control_705]' is not marked as EOG\n", "load: control token: 371 '[control_369]' is not marked as EOG\n", "load: control token: 628 '[control_626]' is not marked as EOG\n", "load: control token: 15 '[control_13]' is not marked as EOG\n", "load: control token: 354 '[control_352]' is not marked as EOG\n", "load: control token: 25 '[control_23]' is not marked as EOG\n", "load: control token: 261 '[control_259]' is not marked as EOG\n", "load: control token: 435 '[control_433]' is not marked as EOG\n", "load: control token: 307 '[control_305]' is not marked as EOG\n", "load: control token: 305 '[control_303]' is not marked as EOG\n", "load: control token: 656 '[control_654]' is not marked as EOG\n", "load: control token: 702 '[control_700]' is not marked as EOG\n", "load: control token: 428 '[control_426]' is not marked as EOG\n", "load: control token: 499 '[control_497]' is not marked as EOG\n", "load: control token: 154 '[control_152]' is not marked as EOG\n", "load: control token: 308 '[control_306]' is not marked as EOG\n", "load: control token: 286 '[control_284]' is not marked as EOG\n", "load: control token: 634 '[control_632]' is not marked as EOG\n", "load: control token: 51 '[control_49]' is not marked as EOG\n", "load: control token: 599 '[control_597]' is not marked as EOG\n", "load: control token: 606 '[control_604]' is not marked as EOG\n", "load: control token: 30 '[control_28]' is not marked as EOG\n", "load: control token: 250 '[control_248]' is not marked as EOG\n", "load: control token: 714 '[control_712]' is not marked as EOG\n", "load: control token: 270 '[control_268]' is not marked as EOG\n", "load: control token: 83 '[control_81]' is not marked as EOG\n", "load: control token: 730 '[control_728]' is not marked as EOG\n", "load: control token: 743 '[control_741]' is not marked as EOG\n", "load: control token: 484 '[control_482]' is not marked as EOG\n", "load: control token: 758 '[control_756]' is not marked as EOG\n", "load: control token: 271 '[control_269]' is not marked as EOG\n", "load: control token: 279 '[control_277]' is not marked as EOG\n", "load: control token: 73 '[control_71]' is not marked as EOG\n", "load: control token: 363 '[control_361]' is not marked as EOG\n", "load: control token: 60 '[control_58]' is not marked as EOG\n", "load: control token: 405 '[control_403]' is not marked as EOG\n", "load: control token: 220 '[control_218]' is not marked as EOG\n", "load: control token: 436 '[control_434]' is not marked as EOG\n", "load: control token: 27 '[control_25]' is not marked as EOG\n", "load: control token: 652 '[control_650]' is not marked as EOG\n", "load: control token: 646 '[control_644]' is not marked as EOG\n", "load: control token: 395 '[control_393]' is not marked as EOG\n", "load: control token: 549 '[control_547]' is not marked as EOG\n", "load: control token: 12 '[control_10]' is not marked as EOG\n", "load: control token: 229 '[control_227]' is not marked as EOG\n", "load: control token: 162 '[control_160]' is not marked as EOG\n", "load: control token: 497 '[control_495]' is not marked as EOG\n", "load: control token: 31 '[control_29]' is not marked as EOG\n", "load: control token: 98 '[control_96]' is not marked as EOG\n", "load: control token: 686 '[control_684]' is not marked as EOG\n", "load: control token: 3 '[INST]' is not marked as EOG\n", "load: control token: 155 '[control_153]' is not marked as EOG\n", "load: control token: 470 '[control_468]' is not marked as EOG\n", "load: control token: 69 '[control_67]' is not marked as EOG\n", "load: control token: 93 '[control_91]' is not marked as EOG\n", "load: control token: 71 '[control_69]' is not marked as EOG\n", "load: control token: 11 '[control_9]' is not marked as EOG\n", "load: control token: 43 '[control_41]' is not marked as EOG\n", "load: control token: 22 '[control_20]' is not marked as EOG\n", "load: control token: 35 '[control_33]' is not marked as EOG\n", "load: control token: 706 '[control_704]' is not marked as EOG\n", "load: control token: 511 '[control_509]' is not marked as EOG\n", "load: control token: 491 '[control_489]' is not marked as EOG\n", "load: control token: 324 '[control_322]' is not marked as EOG\n", "load: control token: 655 '[control_653]' is not marked as EOG\n", "load: control token: 117 '[control_115]' is not marked as EOG\n", "load: control token: 665 '[control_663]' is not marked as EOG\n", "load: control token: 9 '[/TOOL_RESULTS]' is not marked as EOG\n", "load: control token: 29 '[control_27]' is not marked as EOG\n", "load: control token: 44 '[control_42]' is not marked as EOG\n", "load: control token: 310 '[control_308]' is not marked as EOG\n", "load: control token: 157 '[control_155]' is not marked as EOG\n", "load: control token: 515 '[control_513]' is not marked as EOG\n", "load: control token: 388 '[control_386]' is not marked as EOG\n", "load: control token: 438 '[control_436]' is not marked as EOG\n", "load: control token: 486 '[control_484]' is not marked as EOG\n", "load: control token: 139 '[control_137]' is not marked as EOG\n", "load: control token: 543 '[control_541]' is not marked as EOG\n", "load: control token: 622 '[control_620]' is not marked as EOG\n", "load: control token: 598 '[control_596]' is not marked as EOG\n", "load: control token: 16 '[control_14]' is not marked as EOG\n", "load: control token: 39 '[control_37]' is not marked as EOG\n", "load: control token: 102 '[control_100]' is not marked as EOG\n", "load: control token: 673 '[control_671]' is not marked as EOG\n", "load: control token: 287 '[control_285]' is not marked as EOG\n", "load: control token: 244 '[control_242]' is not marked as EOG\n", "load: control token: 446 '[control_444]' is not marked as EOG\n", "load: control token: 483 '[control_481]' is not marked as EOG\n", "load: control token: 467 '[control_465]' is not marked as EOG\n", "load: control token: 264 '[control_262]' is not marked as EOG\n", "load: control token: 176 '[control_174]' is not marked as EOG\n", "load: control token: 625 '[control_623]' is not marked as EOG\n", "load: control token: 160 '[control_158]' is not marked as EOG\n", "load: control token: 20 '[control_18]' is not marked as EOG\n", "load: control token: 641 '[control_639]' is not marked as EOG\n", "load: control token: 99 '[control_97]' is not marked as EOG\n", "load: control token: 54 '[control_52]' is not marked as EOG\n", "load: control token: 37 '[control_35]' is not marked as EOG\n", "load: control token: 401 '[control_399]' is not marked as EOG\n", "load: control token: 130 '[control_128]' is not marked as EOG\n", "load: control token: 218 '[control_216]' is not marked as EOG\n", "load: control token: 243 '[control_241]' is not marked as EOG\n", "load: control token: 753 '[control_751]' is not marked as EOG\n", "load: control token: 74 '[control_72]' is not marked as EOG\n", "load: control token: 171 '[control_169]' is not marked as EOG\n", "load: control token: 151 '[control_149]' is not marked as EOG\n", "load: control token: 364 '[control_362]' is not marked as EOG\n", "load: control token: 495 '[control_493]' is not marked as EOG\n", "load: control token: 119 '[control_117]' is not marked as EOG\n", "load: control token: 217 '[control_215]' is not marked as EOG\n", "load: control token: 84 '[control_82]' is not marked as EOG\n", "load: control token: 5 '[TOOL_CALLS]' is not marked as EOG\n", "load: control token: 531 '[control_529]' is not marked as EOG\n", "load: control token: 33 '[control_31]' is not marked as EOG\n", "load: control token: 82 '[control_80]' is not marked as EOG\n", "load: control token: 100 '[control_98]' is not marked as EOG\n", "load: control token: 125 '[control_123]' is not marked as EOG\n", "load: control token: 520 '[control_518]' is not marked as EOG\n", "load: control token: 144 '[control_142]' is not marked as EOG\n", "load: control token: 152 '[control_150]' is not marked as EOG\n", "load: control token: 592 '[control_590]' is not marked as EOG\n", "load: control token: 461 '[control_459]' is not marked as EOG\n", "load: control token: 624 '[control_622]' is not marked as EOG\n", "load: control token: 65 '[control_63]' is not marked as EOG\n", "load: control token: 172 '[control_170]' is not marked as EOG\n", "load: control token: 178 '[control_176]' is not marked as EOG\n", "load: control token: 258 '[control_256]' is not marked as EOG\n", "load: control token: 524 '[control_522]' is not marked as EOG\n", "load: control token: 657 '[control_655]' is not marked as EOG\n", "load: control token: 184 '[control_182]' is not marked as EOG\n", "load: control token: 441 '[control_439]' is not marked as EOG\n", "load: control token: 315 '[control_313]' is not marked as EOG\n", "load: control token: 662 '[control_660]' is not marked as EOG\n", "load: control token: 6 '[AVAILABLE_TOOLS]' is not marked as EOG\n", "load: control token: 517 '[control_515]' is not marked as EOG\n", "load: control token: 194 '[control_192]' is not marked as EOG\n", "load: control token: 447 '[control_445]' is not marked as EOG\n", "load: control token: 52 '[control_50]' is not marked as EOG\n", "load: control token: 718 '[control_716]' is not marked as EOG\n", "load: control token: 692 '[control_690]' is not marked as EOG\n", "load: control token: 290 '[control_288]' is not marked as EOG\n", "load: control token: 763 '[control_761]' is not marked as EOG\n", "load: control token: 362 '[control_360]' is not marked as EOG\n", "load: control token: 200 '[control_198]' is not marked as EOG\n", "load: control token: 204 '[control_202]' is not marked as EOG\n", "load: control token: 722 '[control_720]' is not marked as EOG\n", "load: control token: 351 '[control_349]' is not marked as EOG\n", "load: control token: 349 '[control_347]' is not marked as EOG\n", "load: control token: 38 '[control_36]' is not marked as EOG\n", "load: control token: 134 '[control_132]' is not marked as EOG\n", "load: control token: 67 '[control_65]' is not marked as EOG\n", "load: control token: 239 '[control_237]' is not marked as EOG\n", "load: control token: 570 '[control_568]' is not marked as EOG\n", "load: control token: 253 '[control_251]' is not marked as EOG\n", "load: control token: 221 '[control_219]' is not marked as EOG\n", "load: control token: 664 '[control_662]' is not marked as EOG\n", "load: control token: 225 '[control_223]' is not marked as EOG\n", "load: control token: 226 '[control_224]' is not marked as EOG\n", "load: control token: 553 '[control_551]' is not marked as EOG\n", "load: control token: 242 '[control_240]' is not marked as EOG\n", "load: control token: 633 '[control_631]' is not marked as EOG\n", "load: control token: 580 '[control_578]' is not marked as EOG\n", "load: control token: 245 '[control_243]' is not marked as EOG\n", "load: control token: 620 '[control_618]' is not marked as EOG\n", "load: control token: 191 '[control_189]' is not marked as EOG\n", "load: control token: 87 '[control_85]' is not marked as EOG\n", "load: control token: 189 '[control_187]' is not marked as EOG\n", "load: control token: 542 '[control_540]' is not marked as EOG\n", "load: control token: 548 '[control_546]' is not marked as EOG\n", "load: control token: 558 '[control_556]' is not marked as EOG\n", "load: control token: 560 '[control_558]' is not marked as EOG\n", "load: control token: 564 '[control_562]' is not marked as EOG\n", "load: control token: 410 '[control_408]' is not marked as EOG\n", "load: control token: 566 '[control_564]' is not marked as EOG\n", "load: control token: 649 '[control_647]' is not marked as EOG\n", "load: control token: 414 '[control_412]' is not marked as EOG\n", "load: control token: 567 '[control_565]' is not marked as EOG\n", "load: control token: 569 '[control_567]' is not marked as EOG\n", "load: control token: 761 '[control_759]' is not marked as EOG\n", "load: control token: 408 '[control_406]' is not marked as EOG\n", "load: control token: 579 '[control_577]' is not marked as EOG\n", "load: control token: 147 '[control_145]' is not marked as EOG\n", "load: control token: 236 '[control_234]' is not marked as EOG\n", "load: control token: 588 '[control_586]' is not marked as EOG\n", "load: control token: 292 '[control_290]' is not marked as EOG\n", "load: control token: 591 '[control_589]' is not marked as EOG\n", "load: control token: 645 '[control_643]' is not marked as EOG\n", "load: control token: 140 '[control_138]' is not marked as EOG\n", "load: control token: 422 '[control_420]' is not marked as EOG\n", "load: control token: 595 '[control_593]' is not marked as EOG\n", "load: control token: 597 '[control_595]' is not marked as EOG\n", "load: control token: 754 '[control_752]' is not marked as EOG\n", "load: control token: 159 '[control_157]' is not marked as EOG\n", "load: control token: 344 '[control_342]' is not marked as EOG\n", "load: control token: 393 '[control_391]' is not marked as EOG\n", "load: control token: 691 '[control_689]' is not marked as EOG\n", "load: control token: 608 '[control_606]' is not marked as EOG\n", "load: control token: 663 '[control_661]' is not marked as EOG\n", "load: control token: 613 '[control_611]' is not marked as EOG\n", "load: control token: 340 '[control_338]' is not marked as EOG\n", "load: control token: 616 '[control_614]' is not marked as EOG\n", "load: control token: 719 '[control_717]' is not marked as EOG\n", "load: control token: 521 '[control_519]' is not marked as EOG\n", "load: control token: 619 '[control_617]' is not marked as EOG\n", "load: control token: 523 '[control_521]' is not marked as EOG\n", "load: control token: 626 '[control_624]' is not marked as EOG\n", "load: control token: 695 '[control_693]' is not marked as EOG\n", "load: control token: 637 '[control_635]' is not marked as EOG\n", "load: control token: 235 '[control_233]' is not marked as EOG\n", "load: control token: 659 '[control_657]' is not marked as EOG\n", "load: control token: 268 '[control_266]' is not marked as EOG\n", "load: control token: 406 '[control_404]' is not marked as EOG\n", "load: control token: 735 '[control_733]' is not marked as EOG\n", "load: control token: 398 '[control_396]' is not marked as EOG\n", "load: control token: 674 '[control_672]' is not marked as EOG\n", "load: control token: 684 '[control_682]' is not marked as EOG\n", "load: control token: 462 '[control_460]' is not marked as EOG\n", "load: control token: 382 '[control_380]' is not marked as EOG\n", "load: control token: 697 '[control_695]' is not marked as EOG\n", "load: control token: 698 '[control_696]' is not marked as EOG\n", "load: control token: 699 '[control_697]' is not marked as EOG\n", "load: control token: 688 '[control_686]' is not marked as EOG\n", "load: control token: 704 '[control_702]' is not marked as EOG\n", "load: control token: 705 '[control_703]' is not marked as EOG\n", "load: control token: 205 '[control_203]' is not marked as EOG\n", "load: control token: 726 '[control_724]' is not marked as EOG\n", "load: control token: 180 '[control_178]' is not marked as EOG\n", "load: control token: 716 '[control_714]' is not marked as EOG\n", "load: control token: 590 '[control_588]' is not marked as EOG\n", "load: control token: 454 '[control_452]' is not marked as EOG\n", "load: control token: 728 '[control_726]' is not marked as EOG\n", "load: control token: 738 '[control_736]' is not marked as EOG\n", "load: control token: 744 '[control_742]' is not marked as EOG\n", "load: control token: 403 '[control_401]' is not marked as EOG\n", "load: control token: 764 '[control_762]' is not marked as EOG\n", "load: control token: 676 '[control_674]' is not marked as EOG\n", "load: control token: 756 '[control_754]' is not marked as EOG\n", "load: control token: 380 '[control_378]' is not marked as EOG\n", "load: control token: 319 '[control_317]' is not marked as EOG\n", "load: control token: 600 '[control_598]' is not marked as EOG\n", "load: control token: 297 '[control_295]' is not marked as EOG\n", "load: control token: 259 '[control_257]' is not marked as EOG\n", "load: control token: 639 '[control_637]' is not marked as EOG\n", "load: control token: 529 '[control_527]' is not marked as EOG\n", "load: control token: 101 '[control_99]' is not marked as EOG\n", "load: control token: 522 '[control_520]' is not marked as EOG\n", "load: control token: 723 '[control_721]' is not marked as EOG\n", "load: control token: 254 '[control_252]' is not marked as EOG\n", "load: control token: 513 '[control_511]' is not marked as EOG\n", "load: control token: 667 '[control_665]' is not marked as EOG\n", "load: control token: 267 '[control_265]' is not marked as EOG\n", "load: control token: 732 '[control_730]' is not marked as EOG\n", "load: control token: 24 '[control_22]' is not marked as EOG\n", "load: control token: 650 '[control_648]' is not marked as EOG\n", "load: control token: 528 '[control_526]' is not marked as EOG\n", "load: control token: 749 '[control_747]' is not marked as EOG\n", "load: control token: 721 '[control_719]' is not marked as EOG\n", "load: control token: 103 '[control_101]' is not marked as EOG\n", "load: control token: 294 '[control_292]' is not marked as EOG\n", "load: control token: 41 '[control_39]' is not marked as EOG\n", "load: control token: 133 '[control_131]' is not marked as EOG\n", "load: control token: 188 '[control_186]' is not marked as EOG\n", "load: control token: 276 '[control_274]' is not marked as EOG\n", "load: control token: 644 '[control_642]' is not marked as EOG\n", "load: control token: 648 '[control_646]' is not marked as EOG\n", "load: control token: 459 '[control_457]' is not marked as EOG\n", "load: control token: 112 '[control_110]' is not marked as EOG\n", "load: control token: 300 '[control_298]' is not marked as EOG\n", "load: control token: 206 '[control_204]' is not marked as EOG\n", "load: control token: 121 '[control_119]' is not marked as EOG\n", "load: control token: 750 '[control_748]' is not marked as EOG\n", "load: control token: 131 '[control_129]' is not marked as EOG\n", "load: control token: 474 '[control_472]' is not marked as EOG\n", "load: control token: 685 '[control_683]' is not marked as EOG\n", "load: control token: 251 '[control_249]' is not marked as EOG\n", "load: control token: 402 '[control_400]' is not marked as EOG\n", "load: control token: 281 '[control_279]' is not marked as EOG\n", "load: control token: 192 '[control_190]' is not marked as EOG\n", "load: control token: 387 '[control_385]' is not marked as EOG\n", "load: control token: 587 '[control_585]' is not marked as EOG\n", "load: control token: 66 '[control_64]' is not marked as EOG\n", "load: control token: 394 '[control_392]' is not marked as EOG\n", "load: control token: 49 '[control_47]' is not marked as EOG\n", "load: control token: 42 '[control_40]' is not marked as EOG\n", "load: control token: 32 '[control_30]' is not marked as EOG\n", "load: control token: 399 '[control_397]' is not marked as EOG\n", "load: control token: 448 '[control_446]' is not marked as EOG\n", "load: control token: 479 '[control_477]' is not marked as EOG\n", "load: control token: 493 '[control_491]' is not marked as EOG\n", "load: control token: 736 '[control_734]' is not marked as EOG\n", "load: control token: 348 '[control_346]' is not marked as EOG\n", "load: control token: 193 '[control_191]' is not marked as EOG\n", "load: control token: 231 '[control_229]' is not marked as EOG\n", "load: control token: 609 '[control_607]' is not marked as EOG\n", "load: control token: 213 '[control_211]' is not marked as EOG\n", "load: control token: 128 '[control_126]' is not marked as EOG\n", "load: control token: 118 '[control_116]' is not marked as EOG\n", "load: control token: 369 '[control_367]' is not marked as EOG\n", "load: control token: 630 '[control_628]' is not marked as EOG\n", "load: control token: 755 '[control_753]' is not marked as EOG\n", "load: control token: 747 '[control_745]' is not marked as EOG\n", "load: control token: 289 '[control_287]' is not marked as EOG\n", "load: control token: 26 '[control_24]' is not marked as EOG\n", "load: control token: 63 '[control_61]' is not marked as EOG\n", "load: control token: 284 '[control_282]' is not marked as EOG\n", "load: control token: 164 '[control_162]' is not marked as EOG\n", "load: control token: 404 '[control_402]' is not marked as EOG\n", "load: control token: 506 '[control_504]' is not marked as EOG\n", "load: control token: 123 '[control_121]' is not marked as EOG\n", "load: control token: 237 '[control_235]' is not marked as EOG\n", "load: control token: 62 '[control_60]' is not marked as EOG\n", "load: control token: 510 '[control_508]' is not marked as EOG\n", "load: control token: 233 '[control_231]' is not marked as EOG\n", "load: control token: 68 '[control_66]' is not marked as EOG\n", "load: control token: 487 '[control_485]' is not marked as EOG\n", "load: control token: 471 '[control_469]' is not marked as EOG\n", "load: control token: 415 '[control_413]' is not marked as EOG\n", "load: control token: 78 '[control_76]' is not marked as EOG\n", "load: control token: 610 '[control_608]' is not marked as EOG\n", "load: control token: 137 '[control_135]' is not marked as EOG\n", "load: control token: 539 '[control_537]' is not marked as EOG\n", "load: control token: 658 '[control_656]' is not marked as EOG\n", "load: control token: 158 '[control_156]' is not marked as EOG\n", "load: control token: 585 '[control_583]' is not marked as EOG\n", "load: control token: 215 '[control_213]' is not marked as EOG\n", "load: control token: 554 '[control_552]' is not marked as EOG\n", "load: control token: 762 '[control_760]' is not marked as EOG\n", "load: control token: 717 '[control_715]' is not marked as EOG\n", "load: control token: 228 '[control_226]' is not marked as EOG\n", "load: control token: 442 '[control_440]' is not marked as EOG\n", "load: control token: 494 '[control_492]' is not marked as EOG\n", "load: control token: 97 '[control_95]' is not marked as EOG\n", "load: control token: 683 '[control_681]' is not marked as EOG\n", "load: control token: 629 '[control_627]' is not marked as EOG\n", "load: control token: 725 '[control_723]' is not marked as EOG\n", "load: control token: 432 '[control_430]' is not marked as EOG\n", "load: control token: 477 '[control_475]' is not marked as EOG\n", "load: control token: 355 '[control_353]' is not marked as EOG\n", "load: control token: 36 '[control_34]' is not marked as EOG\n", "load: control token: 534 '[control_532]' is not marked as EOG\n", "load: control token: 120 '[control_118]' is not marked as EOG\n", "load: control token: 28 '[control_26]' is not marked as EOG\n", "load: control token: 306 '[control_304]' is not marked as EOG\n", "load: control token: 407 '[control_405]' is not marked as EOG\n", "load: control token: 696 '[control_694]' is not marked as EOG\n", "load: control token: 274 '[control_272]' is not marked as EOG\n", "load: control token: 142 '[control_140]' is not marked as EOG\n", "load: control token: 136 '[control_134]' is not marked as EOG\n", "load: control token: 309 '[control_307]' is not marked as EOG\n", "load: control token: 423 '[control_421]' is not marked as EOG\n", "load: control token: 724 '[control_722]' is not marked as EOG\n", "load: control token: 262 '[control_260]' is not marked as EOG\n", "load: control token: 419 '[control_417]' is not marked as EOG\n", "load: control token: 329 '[control_327]' is not marked as EOG\n", "load: control token: 427 '[control_425]' is not marked as EOG\n", "load: control token: 370 '[control_368]' is not marked as EOG\n", "load: control token: 266 '[control_264]' is not marked as EOG\n", "load: control token: 710 '[control_708]' is not marked as EOG\n", "load: control token: 602 '[control_600]' is not marked as EOG\n", "load: control token: 277 '[control_275]' is not marked as EOG\n", "load: control token: 298 '[control_296]' is not marked as EOG\n", "load: control token: 378 '[control_376]' is not marked as EOG\n", "load: control token: 198 '[control_196]' is not marked as EOG\n", "load: control token: 76 '[control_74]' is not marked as EOG\n", "load: control token: 607 '[control_605]' is not marked as EOG\n", "load: control token: 682 '[control_680]' is not marked as EOG\n", "load: control token: 333 '[control_331]' is not marked as EOG\n", "load: control token: 660 '[control_658]' is not marked as EOG\n", "load: control token: 163 '[control_161]' is not marked as EOG\n", "load: control token: 95 '[control_93]' is not marked as EOG\n", "load: control token: 377 '[control_375]' is not marked as EOG\n", "load: control token: 223 '[control_221]' is not marked as EOG\n", "load: control token: 326 '[control_324]' is not marked as EOG\n", "load: control token: 111 '[control_109]' is not marked as EOG\n", "load: control token: 187 '[control_185]' is not marked as EOG\n", "load: control token: 678 '[control_676]' is not marked as EOG\n", "load: control token: 514 '[control_512]' is not marked as EOG\n", "load: control token: 431 '[control_429]' is not marked as EOG\n", "load: control token: 4 '[/INST]' is not marked as EOG\n", "load: control token: 342 '[control_340]' is not marked as EOG\n", "load: control token: 594 '[control_592]' is not marked as EOG\n", "load: control token: 768 '[control_766]' is not marked as EOG\n", "load: control token: 143 '[control_141]' is not marked as EOG\n", "load: control token: 445 '[control_443]' is not marked as EOG\n", "load: control token: 165 '[control_163]' is not marked as EOG\n", "load: control token: 439 '[control_437]' is not marked as EOG\n", "load: control token: 181 '[control_179]' is not marked as EOG\n", "load: control token: 352 '[control_350]' is not marked as EOG\n", "load: control token: 643 '[control_641]' is not marked as EOG\n", "load: control token: 182 '[control_180]' is not marked as EOG\n", "load: control token: 19 '[control_17]' is not marked as EOG\n", "load: control token: 269 '[control_267]' is not marked as EOG\n", "load: control token: 677 '[control_675]' is not marked as EOG\n", "load: control token: 615 '[control_613]' is not marked as EOG\n", "load: control token: 170 '[control_168]' is not marked as EOG\n", "load: control token: 56 '[control_54]' is not marked as EOG\n", "load: control token: 582 '[control_580]' is not marked as EOG\n", "load: control token: 700 '[control_698]' is not marked as EOG\n", "load: control token: 561 '[control_559]' is not marked as EOG\n", "load: control token: 381 '[control_379]' is not marked as EOG\n", "load: control token: 647 '[control_645]' is not marked as EOG\n", "load: control token: 437 '[control_435]' is not marked as EOG\n", "load: control token: 507 '[control_505]' is not marked as EOG\n", "load: control token: 490 '[control_488]' is not marked as EOG\n", "load: control token: 586 '[control_584]' is not marked as EOG\n", "load: control token: 23 '[control_21]' is not marked as EOG\n", "load: control token: 452 '[control_450]' is not marked as EOG\n", "load: control token: 605 '[control_603]' is not marked as EOG\n", "load: control token: 426 '[control_424]' is not marked as EOG\n", "load: control token: 769 '[control_767]' is not marked as EOG\n", "load: control token: 81 '[control_79]' is not marked as EOG\n", "load: control token: 386 '[control_384]' is not marked as EOG\n", "load: control token: 627 '[control_625]' is not marked as EOG\n", "load: control token: 18 '[control_16]' is not marked as EOG\n", "load: control token: 741 '[control_739]' is not marked as EOG\n", "load: control token: 96 '[control_94]' is not marked as EOG\n", "load: control token: 502 '[control_500]' is not marked as EOG\n", "load: control token: 482 '[control_480]' is not marked as EOG\n", "load: control token: 480 '[control_478]' is not marked as EOG\n", "load: control token: 557 '[control_555]' is not marked as EOG\n", "load: control token: 263 '[control_261]' is not marked as EOG\n", "load: control token: 668 '[control_666]' is not marked as EOG\n", "load: control token: 466 '[control_464]' is not marked as EOG\n", "load: control token: 334 '[control_332]' is not marked as EOG\n", "load: control token: 413 '[control_411]' is not marked as EOG\n", "load: control token: 572 '[control_570]' is not marked as EOG\n", "load: control token: 295 '[control_293]' is not marked as EOG\n", "load: control token: 150 '[control_148]' is not marked as EOG\n", "load: control token: 275 '[control_273]' is not marked as EOG\n", "load: control token: 748 '[control_746]' is not marked as EOG\n", "load: control token: 135 '[control_133]' is not marked as EOG\n", "load: control token: 357 '[control_355]' is not marked as EOG\n", "load: control token: 577 '[control_575]' is not marked as EOG\n", "load: control token: 578 '[control_576]' is not marked as EOG\n", "load: control token: 556 '[control_554]' is not marked as EOG\n", "load: control token: 375 '[control_373]' is not marked as EOG\n", "load: control token: 589 '[control_587]' is not marked as EOG\n", "load: control token: 202 '[control_200]' is not marked as EOG\n", "load: control token: 85 '[control_83]' is not marked as EOG\n", "load: control token: 278 '[control_276]' is not marked as EOG\n", "load: control token: 169 '[control_167]' is not marked as EOG\n", "load: control token: 208 '[control_206]' is not marked as EOG\n", "load: control token: 488 '[control_486]' is not marked as EOG\n", "load: control token: 337 '[control_335]' is not marked as EOG\n", "load: control token: 733 '[control_731]' is not marked as EOG\n", "load: control token: 109 '[control_107]' is not marked as EOG\n", "load: control token: 547 '[control_545]' is not marked as EOG\n", "load: control token: 350 '[control_348]' is not marked as EOG\n", "load: control token: 132 '[control_130]' is not marked as EOG\n", "load: control token: 114 '[control_112]' is not marked as EOG\n", "load: control token: 248 '[control_246]' is not marked as EOG\n", "load: control token: 328 '[control_326]' is not marked as EOG\n", "load: control token: 104 '[control_102]' is not marked as EOG\n", "load: control token: 449 '[control_447]' is not marked as EOG\n", "load: control token: 711 '[control_709]' is not marked as EOG\n", "load: control token: 715 '[control_713]' is not marked as EOG\n", "load: control token: 358 '[control_356]' is not marked as EOG\n", "load: control token: 746 '[control_744]' is not marked as EOG\n", "load: control token: 385 '[control_383]' is not marked as EOG\n", "load: control token: 720 '[control_718]' is not marked as EOG\n", "load: control token: 460 '[control_458]' is not marked as EOG\n", "load: control token: 632 '[control_630]' is not marked as EOG\n", "load: control token: 296 '[control_294]' is not marked as EOG\n", "load: control token: 240 '[control_238]' is not marked as EOG\n", "load: control token: 361 '[control_359]' is not marked as EOG\n", "load: control token: 504 '[control_502]' is not marked as EOG\n", "load: control token: 149 '[control_147]' is not marked as EOG\n", "load: control token: 1 '' is not marked as EOG\n", "load: control token: 219 '[control_217]' is not marked as EOG\n", "load: control token: 106 '[control_104]' is not marked as EOG\n", "load: control token: 669 '[control_667]' is not marked as EOG\n", "load: control token: 411 '[control_409]' is not marked as EOG\n", "load: control token: 546 '[control_544]' is not marked as EOG\n", "load: control token: 379 '[control_377]' is not marked as EOG\n", "load: control token: 209 '[control_207]' is not marked as EOG\n", "load: control token: 110 '[control_108]' is not marked as EOG\n", "load: control token: 681 '[control_679]' is not marked as EOG\n", "load: control token: 153 '[control_151]' is not marked as EOG\n", "load: control token: 457 '[control_455]' is not marked as EOG\n", "load: control token: 91 '[control_89]' is not marked as EOG\n", "load: control token: 224 '[control_222]' is not marked as EOG\n", "load: control token: 653 '[control_651]' is not marked as EOG\n", "load: control token: 694 '[control_692]' is not marked as EOG\n", "load: control token: 316 '[control_314]' is not marked as EOG\n", "load: printing all EOG tokens:\n", "load: - 2 ('')\n", "load: special tokens cache size = 771\n", "load: token to piece cache size = 0.1731 MB\n", "print_info: arch = llama\n", "print_info: vocab_only = 0\n", "print_info: no_alloc = 0\n", "print_info: n_ctx_train = 32768\n", "print_info: n_embd = 4096\n", "print_info: n_embd_inp = 4096\n", "print_info: n_layer = 32\n", "print_info: n_head = 32\n", "print_info: n_head_kv = 8\n", "print_info: n_rot = 128\n", "print_info: n_swa = 0\n", "print_info: is_swa_any = 0\n", "print_info: n_embd_head_k = 128\n", "print_info: n_embd_head_v = 128\n", "print_info: n_gqa = 4\n", "print_info: n_embd_k_gqa = 1024\n", "print_info: n_embd_v_gqa = 1024\n", "print_info: f_norm_eps = 0.0e+00\n", "print_info: f_norm_rms_eps = 1.0e-05\n", "print_info: f_clamp_kqv = 0.0e+00\n", "print_info: f_max_alibi_bias = 0.0e+00\n", "print_info: f_logit_scale = 0.0e+00\n", "print_info: f_attn_scale = 0.0e+00\n", "print_info: f_attn_value_scale = 0.0000\n", "print_info: n_ff = 14336\n", "print_info: n_expert = 0\n", "print_info: n_expert_used = 0\n", "print_info: n_expert_groups = 0\n", "print_info: n_group_used = 0\n", "print_info: causal attn = 1\n", "print_info: pooling type = -1\n", "print_info: rope type = 0\n", "print_info: rope scaling = linear\n", "print_info: freq_base_train = 1000000.0\n", "print_info: freq_scale_train = 1\n", "print_info: n_ctx_orig_yarn = 32768\n", "print_info: rope_yarn_log_mul = 0.0000\n", "print_info: rope_finetuned = unknown\n", "print_info: model type = 7B\n", "print_info: model params = 7.25 B\n", "print_info: general.name = Mistral-7B-Instruct-v0.3\n", "print_info: vocab type = SPM\n", "print_info: n_vocab = 32768\n", "print_info: n_merges = 0\n", "print_info: BOS token = 1 ''\n", "print_info: EOS token = 2 ''\n", "print_info: UNK token = 0 ''\n", "print_info: LF token = 781 '<0x0A>'\n", "print_info: EOG token = 2 ''\n", "print_info: max token length = 48\n", "load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)\n", "load_tensors: layer 0 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 1 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 2 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 3 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 4 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 5 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 6 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 7 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 8 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 9 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 10 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 11 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 12 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 13 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 14 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 15 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 16 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 17 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 18 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 19 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 20 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 21 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 22 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 23 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 24 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 25 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 26 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 27 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 28 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 29 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 30 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 31 assigned to device CPU, is_swa = 0\n", "load_tensors: layer 32 assigned to device CPU, is_swa = 0\n", "create_tensor: loading tensor token_embd.weight\n", "create_tensor: loading tensor output_norm.weight\n", "create_tensor: loading tensor output.weight\n", "create_tensor: loading tensor blk.0.attn_norm.weight\n", "create_tensor: loading tensor blk.0.attn_q.weight\n", "create_tensor: loading tensor blk.0.attn_k.weight\n", "create_tensor: loading tensor blk.0.attn_v.weight\n", "create_tensor: loading tensor blk.0.attn_output.weight\n", "create_tensor: loading tensor blk.0.ffn_norm.weight\n", "create_tensor: loading tensor blk.0.ffn_gate.weight\n", "create_tensor: loading tensor blk.0.ffn_down.weight\n", "create_tensor: loading tensor blk.0.ffn_up.weight\n", "create_tensor: loading tensor blk.1.attn_norm.weight\n", "create_tensor: loading tensor blk.1.attn_q.weight\n", "create_tensor: loading tensor blk.1.attn_k.weight\n", "create_tensor: loading tensor blk.1.attn_v.weight\n", "create_tensor: loading tensor blk.1.attn_output.weight\n", "create_tensor: loading tensor blk.1.ffn_norm.weight\n", "create_tensor: loading tensor blk.1.ffn_gate.weight\n", "create_tensor: loading tensor blk.1.ffn_down.weight\n", "create_tensor: loading tensor blk.1.ffn_up.weight\n", "create_tensor: loading tensor blk.2.attn_norm.weight\n", "create_tensor: loading tensor blk.2.attn_q.weight\n", "create_tensor: loading tensor blk.2.attn_k.weight\n", "create_tensor: loading tensor blk.2.attn_v.weight\n", "create_tensor: loading tensor blk.2.attn_output.weight\n", "create_tensor: loading tensor blk.2.ffn_norm.weight\n", "create_tensor: loading tensor blk.2.ffn_gate.weight\n", "create_tensor: loading tensor blk.2.ffn_down.weight\n", "create_tensor: loading tensor blk.2.ffn_up.weight\n", "create_tensor: loading tensor blk.3.attn_norm.weight\n", "create_tensor: loading tensor blk.3.attn_q.weight\n", "create_tensor: loading tensor blk.3.attn_k.weight\n", "create_tensor: loading tensor blk.3.attn_v.weight\n", "create_tensor: loading tensor blk.3.attn_output.weight\n", "create_tensor: loading tensor blk.3.ffn_norm.weight\n", "create_tensor: loading tensor blk.3.ffn_gate.weight\n", "create_tensor: loading tensor blk.3.ffn_down.weight\n", "create_tensor: loading tensor blk.3.ffn_up.weight\n", "create_tensor: loading tensor blk.4.attn_norm.weight\n", "create_tensor: loading tensor blk.4.attn_q.weight\n", "create_tensor: loading tensor blk.4.attn_k.weight\n", "create_tensor: loading tensor blk.4.attn_v.weight\n", "create_tensor: loading tensor blk.4.attn_output.weight\n", "create_tensor: loading tensor blk.4.ffn_norm.weight\n", "create_tensor: loading tensor blk.4.ffn_gate.weight\n", "create_tensor: loading tensor blk.4.ffn_down.weight\n", "create_tensor: loading tensor blk.4.ffn_up.weight\n", "create_tensor: loading tensor blk.5.attn_norm.weight\n", "create_tensor: loading tensor blk.5.attn_q.weight\n", "create_tensor: loading tensor blk.5.attn_k.weight\n", "create_tensor: loading tensor blk.5.attn_v.weight\n", "create_tensor: loading tensor blk.5.attn_output.weight\n", "create_tensor: loading tensor blk.5.ffn_norm.weight\n", "create_tensor: loading tensor blk.5.ffn_gate.weight\n", "create_tensor: loading tensor blk.5.ffn_down.weight\n", "create_tensor: loading tensor blk.5.ffn_up.weight\n", "create_tensor: loading tensor blk.6.attn_norm.weight\n", "create_tensor: loading tensor blk.6.attn_q.weight\n", "create_tensor: loading tensor blk.6.attn_k.weight\n", "create_tensor: loading tensor blk.6.attn_v.weight\n", "create_tensor: loading tensor blk.6.attn_output.weight\n", "create_tensor: loading tensor blk.6.ffn_norm.weight\n", "create_tensor: loading tensor blk.6.ffn_gate.weight\n", "create_tensor: loading tensor blk.6.ffn_down.weight\n", "create_tensor: loading tensor blk.6.ffn_up.weight\n", "create_tensor: loading tensor blk.7.attn_norm.weight\n", "create_tensor: loading tensor blk.7.attn_q.weight\n", "create_tensor: loading tensor blk.7.attn_k.weight\n", "create_tensor: loading tensor blk.7.attn_v.weight\n", "create_tensor: loading tensor blk.7.attn_output.weight\n", "create_tensor: loading tensor blk.7.ffn_norm.weight\n", "create_tensor: loading tensor blk.7.ffn_gate.weight\n", "create_tensor: loading tensor blk.7.ffn_down.weight\n", "create_tensor: loading tensor blk.7.ffn_up.weight\n", "create_tensor: loading tensor blk.8.attn_norm.weight\n", "create_tensor: loading tensor blk.8.attn_q.weight\n", "create_tensor: loading tensor blk.8.attn_k.weight\n", "create_tensor: loading tensor blk.8.attn_v.weight\n", "create_tensor: loading tensor blk.8.attn_output.weight\n", "create_tensor: loading tensor blk.8.ffn_norm.weight\n", "create_tensor: loading tensor blk.8.ffn_gate.weight\n", "create_tensor: loading tensor blk.8.ffn_down.weight\n", "create_tensor: loading tensor blk.8.ffn_up.weight\n", "create_tensor: loading tensor blk.9.attn_norm.weight\n", "create_tensor: loading tensor blk.9.attn_q.weight\n", "create_tensor: loading tensor blk.9.attn_k.weight\n", "create_tensor: loading tensor blk.9.attn_v.weight\n", "create_tensor: loading tensor blk.9.attn_output.weight\n", "create_tensor: loading tensor blk.9.ffn_norm.weight\n", "create_tensor: loading tensor blk.9.ffn_gate.weight\n", "create_tensor: loading tensor blk.9.ffn_down.weight\n", "create_tensor: loading tensor blk.9.ffn_up.weight\n", "create_tensor: loading tensor blk.10.attn_norm.weight\n", "create_tensor: loading tensor blk.10.attn_q.weight\n", "create_tensor: loading tensor blk.10.attn_k.weight\n", "create_tensor: loading tensor blk.10.attn_v.weight\n", "create_tensor: loading tensor blk.10.attn_output.weight\n", "create_tensor: loading tensor blk.10.ffn_norm.weight\n", "create_tensor: loading tensor blk.10.ffn_gate.weight\n", "create_tensor: loading tensor blk.10.ffn_down.weight\n", "create_tensor: loading tensor blk.10.ffn_up.weight\n", "create_tensor: loading tensor blk.11.attn_norm.weight\n", "create_tensor: loading tensor blk.11.attn_q.weight\n", "create_tensor: loading tensor blk.11.attn_k.weight\n", "create_tensor: loading tensor blk.11.attn_v.weight\n", "create_tensor: loading tensor blk.11.attn_output.weight\n", "create_tensor: loading tensor blk.11.ffn_norm.weight\n", "create_tensor: loading tensor blk.11.ffn_gate.weight\n", "create_tensor: loading tensor blk.11.ffn_down.weight\n", "create_tensor: loading tensor blk.11.ffn_up.weight\n", "create_tensor: loading tensor blk.12.attn_norm.weight\n", "create_tensor: loading tensor blk.12.attn_q.weight\n", "create_tensor: loading tensor blk.12.attn_k.weight\n", "create_tensor: loading tensor blk.12.attn_v.weight\n", "create_tensor: loading tensor blk.12.attn_output.weight\n", "create_tensor: loading tensor blk.12.ffn_norm.weight\n", "create_tensor: loading tensor blk.12.ffn_gate.weight\n", "create_tensor: loading tensor blk.12.ffn_down.weight\n", "create_tensor: loading tensor blk.12.ffn_up.weight\n", "create_tensor: loading tensor blk.13.attn_norm.weight\n", "create_tensor: loading tensor blk.13.attn_q.weight\n", "create_tensor: loading tensor blk.13.attn_k.weight\n", "create_tensor: loading tensor blk.13.attn_v.weight\n", "create_tensor: loading tensor blk.13.attn_output.weight\n", "create_tensor: loading tensor blk.13.ffn_norm.weight\n", "create_tensor: loading tensor blk.13.ffn_gate.weight\n", "create_tensor: loading tensor blk.13.ffn_down.weight\n", "create_tensor: loading tensor blk.13.ffn_up.weight\n", "create_tensor: loading tensor blk.14.attn_norm.weight\n", "create_tensor: loading tensor blk.14.attn_q.weight\n", "create_tensor: loading tensor blk.14.attn_k.weight\n", "create_tensor: loading tensor blk.14.attn_v.weight\n", "create_tensor: loading tensor blk.14.attn_output.weight\n", "create_tensor: loading tensor blk.14.ffn_norm.weight\n", "create_tensor: loading tensor blk.14.ffn_gate.weight\n", "create_tensor: loading tensor blk.14.ffn_down.weight\n", "create_tensor: loading tensor blk.14.ffn_up.weight\n", "create_tensor: loading tensor blk.15.attn_norm.weight\n", "create_tensor: loading tensor blk.15.attn_q.weight\n", "create_tensor: loading tensor blk.15.attn_k.weight\n", "create_tensor: loading tensor blk.15.attn_v.weight\n", "create_tensor: loading tensor blk.15.attn_output.weight\n", "create_tensor: loading tensor blk.15.ffn_norm.weight\n", "create_tensor: loading tensor blk.15.ffn_gate.weight\n", "create_tensor: loading tensor blk.15.ffn_down.weight\n", "create_tensor: loading tensor blk.15.ffn_up.weight\n", "create_tensor: loading tensor blk.16.attn_norm.weight\n", "create_tensor: loading tensor blk.16.attn_q.weight\n", "create_tensor: loading tensor blk.16.attn_k.weight\n", "create_tensor: loading tensor blk.16.attn_v.weight\n", "create_tensor: loading tensor blk.16.attn_output.weight\n", "create_tensor: loading tensor blk.16.ffn_norm.weight\n", "create_tensor: loading tensor blk.16.ffn_gate.weight\n", "create_tensor: loading tensor blk.16.ffn_down.weight\n", "create_tensor: loading tensor blk.16.ffn_up.weight\n", "create_tensor: loading tensor blk.17.attn_norm.weight\n", "create_tensor: loading tensor blk.17.attn_q.weight\n", "create_tensor: loading tensor blk.17.attn_k.weight\n", "create_tensor: loading tensor blk.17.attn_v.weight\n", "create_tensor: loading tensor blk.17.attn_output.weight\n", "create_tensor: loading tensor blk.17.ffn_norm.weight\n", "create_tensor: loading tensor blk.17.ffn_gate.weight\n", "create_tensor: loading tensor blk.17.ffn_down.weight\n", "create_tensor: loading tensor blk.17.ffn_up.weight\n", "create_tensor: loading tensor blk.18.attn_norm.weight\n", "create_tensor: loading tensor blk.18.attn_q.weight\n", "create_tensor: loading tensor blk.18.attn_k.weight\n", "create_tensor: loading tensor blk.18.attn_v.weight\n", "create_tensor: loading tensor blk.18.attn_output.weight\n", "create_tensor: loading tensor blk.18.ffn_norm.weight\n", "create_tensor: loading tensor blk.18.ffn_gate.weight\n", "create_tensor: loading tensor blk.18.ffn_down.weight\n", "create_tensor: loading tensor blk.18.ffn_up.weight\n", "create_tensor: loading tensor blk.19.attn_norm.weight\n", "create_tensor: loading tensor blk.19.attn_q.weight\n", "create_tensor: loading tensor blk.19.attn_k.weight\n", "create_tensor: loading tensor blk.19.attn_v.weight\n", "create_tensor: loading tensor blk.19.attn_output.weight\n", "create_tensor: loading tensor blk.19.ffn_norm.weight\n", "create_tensor: loading tensor blk.19.ffn_gate.weight\n", "create_tensor: loading tensor blk.19.ffn_down.weight\n", "create_tensor: loading tensor blk.19.ffn_up.weight\n", "create_tensor: loading tensor blk.20.attn_norm.weight\n", "create_tensor: loading tensor blk.20.attn_q.weight\n", "create_tensor: loading tensor blk.20.attn_k.weight\n", "create_tensor: loading tensor blk.20.attn_v.weight\n", "create_tensor: loading tensor blk.20.attn_output.weight\n", "create_tensor: loading tensor blk.20.ffn_norm.weight\n", "create_tensor: loading tensor blk.20.ffn_gate.weight\n", "create_tensor: loading tensor blk.20.ffn_down.weight\n", "create_tensor: loading tensor blk.20.ffn_up.weight\n", "create_tensor: loading tensor blk.21.attn_norm.weight\n", "create_tensor: loading tensor blk.21.attn_q.weight\n", "create_tensor: loading tensor blk.21.attn_k.weight\n", "create_tensor: loading tensor blk.21.attn_v.weight\n", "create_tensor: loading tensor blk.21.attn_output.weight\n", "create_tensor: loading tensor blk.21.ffn_norm.weight\n", "create_tensor: loading tensor blk.21.ffn_gate.weight\n", "create_tensor: loading tensor blk.21.ffn_down.weight\n", "create_tensor: loading tensor blk.21.ffn_up.weight\n", "create_tensor: loading tensor blk.22.attn_norm.weight\n", "create_tensor: loading tensor blk.22.attn_q.weight\n", "create_tensor: loading tensor blk.22.attn_k.weight\n", "create_tensor: loading tensor blk.22.attn_v.weight\n", "create_tensor: loading tensor blk.22.attn_output.weight\n", "create_tensor: loading tensor blk.22.ffn_norm.weight\n", "create_tensor: loading tensor blk.22.ffn_gate.weight\n", "create_tensor: loading tensor blk.22.ffn_down.weight\n", "create_tensor: loading tensor blk.22.ffn_up.weight\n", "create_tensor: loading tensor blk.23.attn_norm.weight\n", "create_tensor: loading tensor blk.23.attn_q.weight\n", "create_tensor: loading tensor blk.23.attn_k.weight\n", "create_tensor: loading tensor blk.23.attn_v.weight\n", "create_tensor: loading tensor blk.23.attn_output.weight\n", "create_tensor: loading tensor blk.23.ffn_norm.weight\n", "create_tensor: loading tensor blk.23.ffn_gate.weight\n", "create_tensor: loading tensor blk.23.ffn_down.weight\n", "create_tensor: loading tensor blk.23.ffn_up.weight\n", "create_tensor: loading tensor blk.24.attn_norm.weight\n", "create_tensor: loading tensor blk.24.attn_q.weight\n", "create_tensor: loading tensor blk.24.attn_k.weight\n", "create_tensor: loading tensor blk.24.attn_v.weight\n", "create_tensor: loading tensor blk.24.attn_output.weight\n", "create_tensor: loading tensor blk.24.ffn_norm.weight\n", "create_tensor: loading tensor blk.24.ffn_gate.weight\n", "create_tensor: loading tensor blk.24.ffn_down.weight\n", "create_tensor: loading tensor blk.24.ffn_up.weight\n", "create_tensor: loading tensor blk.25.attn_norm.weight\n", "create_tensor: loading tensor blk.25.attn_q.weight\n", "create_tensor: loading tensor blk.25.attn_k.weight\n", "create_tensor: loading tensor blk.25.attn_v.weight\n", "create_tensor: loading tensor blk.25.attn_output.weight\n", "create_tensor: loading tensor blk.25.ffn_norm.weight\n", "create_tensor: loading tensor blk.25.ffn_gate.weight\n", "create_tensor: loading tensor blk.25.ffn_down.weight\n", "create_tensor: loading tensor blk.25.ffn_up.weight\n", "create_tensor: loading tensor blk.26.attn_norm.weight\n", "create_tensor: loading tensor blk.26.attn_q.weight\n", "create_tensor: loading tensor blk.26.attn_k.weight\n", "create_tensor: loading tensor blk.26.attn_v.weight\n", "create_tensor: loading tensor blk.26.attn_output.weight\n", "create_tensor: loading tensor blk.26.ffn_norm.weight\n", "create_tensor: loading tensor blk.26.ffn_gate.weight\n", "create_tensor: loading tensor blk.26.ffn_down.weight\n", "create_tensor: loading tensor blk.26.ffn_up.weight\n", "create_tensor: loading tensor blk.27.attn_norm.weight\n", "create_tensor: loading tensor blk.27.attn_q.weight\n", "create_tensor: loading tensor blk.27.attn_k.weight\n", "create_tensor: loading tensor blk.27.attn_v.weight\n", "create_tensor: loading tensor blk.27.attn_output.weight\n", "create_tensor: loading tensor blk.27.ffn_norm.weight\n", "create_tensor: loading tensor blk.27.ffn_gate.weight\n", "create_tensor: loading tensor blk.27.ffn_down.weight\n", "create_tensor: loading tensor blk.27.ffn_up.weight\n", "create_tensor: loading tensor blk.28.attn_norm.weight\n", "create_tensor: loading tensor blk.28.attn_q.weight\n", "create_tensor: loading tensor blk.28.attn_k.weight\n", "create_tensor: loading tensor blk.28.attn_v.weight\n", "create_tensor: loading tensor blk.28.attn_output.weight\n", "create_tensor: loading tensor blk.28.ffn_norm.weight\n", "create_tensor: loading tensor blk.28.ffn_gate.weight\n", "create_tensor: loading tensor blk.28.ffn_down.weight\n", "create_tensor: loading tensor blk.28.ffn_up.weight\n", "create_tensor: loading tensor blk.29.attn_norm.weight\n", "create_tensor: loading tensor blk.29.attn_q.weight\n", "create_tensor: loading tensor blk.29.attn_k.weight\n", "create_tensor: loading tensor blk.29.attn_v.weight\n", "create_tensor: loading tensor blk.29.attn_output.weight\n", "create_tensor: loading tensor blk.29.ffn_norm.weight\n", "create_tensor: loading tensor blk.29.ffn_gate.weight\n", "create_tensor: loading tensor blk.29.ffn_down.weight\n", "create_tensor: loading tensor blk.29.ffn_up.weight\n", "create_tensor: loading tensor blk.30.attn_norm.weight\n", "create_tensor: loading tensor blk.30.attn_q.weight\n", "create_tensor: loading tensor blk.30.attn_k.weight\n", "create_tensor: loading tensor blk.30.attn_v.weight\n", "create_tensor: loading tensor blk.30.attn_output.weight\n", "create_tensor: loading tensor blk.30.ffn_norm.weight\n", "create_tensor: loading tensor blk.30.ffn_gate.weight\n", "create_tensor: loading tensor blk.30.ffn_down.weight\n", "create_tensor: loading tensor blk.30.ffn_up.weight\n", "create_tensor: loading tensor blk.31.attn_norm.weight\n", "create_tensor: loading tensor blk.31.attn_q.weight\n", "create_tensor: loading tensor blk.31.attn_k.weight\n", "create_tensor: loading tensor blk.31.attn_v.weight\n", "create_tensor: loading tensor blk.31.attn_output.weight\n", "create_tensor: loading tensor blk.31.ffn_norm.weight\n", "create_tensor: loading tensor blk.31.ffn_gate.weight\n", "create_tensor: loading tensor blk.31.ffn_down.weight\n", "create_tensor: loading tensor blk.31.ffn_up.weight\n", "done_getting_tensors: tensor 'token_embd.weight' (q4_K) (and 290 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead\n", "load_tensors: offloading 0 repeating layers to GPU\n", "load_tensors: offloaded 0/33 layers to GPU\n", "load_tensors: CPU_Mapped model buffer size = 4169.52 MiB\n", "................................................................................................\n", "llama_context: constructing llama_context\n", "llama_context: n_seq_max = 1\n", "llama_context: n_ctx = 512\n", "llama_context: n_ctx_seq = 512\n", "llama_context: n_batch = 512\n", "llama_context: n_ubatch = 512\n", "llama_context: causal_attn = 1\n", "llama_context: flash_attn = disabled\n", "llama_context: kv_unified = false\n", "llama_context: freq_base = 1000000.0\n", "llama_context: freq_scale = 1\n", "llama_context: n_ctx_seq (512) < n_ctx_train (32768) -- the full capacity of the model will not be utilized\n", "set_abort_callback: call\n", "llama_context: CPU output buffer size = 0.12 MiB\n", "llama_kv_cache: layer 0: dev = CPU\n", "llama_kv_cache: layer 1: dev = CPU\n", "llama_kv_cache: layer 2: dev = CPU\n", "llama_kv_cache: layer 3: dev = CPU\n", "llama_kv_cache: layer 4: dev = CPU\n", "llama_kv_cache: layer 5: dev = CPU\n", "llama_kv_cache: layer 6: dev = CPU\n", "llama_kv_cache: layer 7: dev = CPU\n", "llama_kv_cache: layer 8: dev = CPU\n", "llama_kv_cache: layer 9: dev = CPU\n", "llama_kv_cache: layer 10: dev = CPU\n", "llama_kv_cache: layer 11: dev = CPU\n", "llama_kv_cache: layer 12: dev = CPU\n", "llama_kv_cache: layer 13: dev = CPU\n", "llama_kv_cache: layer 14: dev = CPU\n", "llama_kv_cache: layer 15: dev = CPU\n", "llama_kv_cache: layer 16: dev = CPU\n", "llama_kv_cache: layer 17: dev = CPU\n", "llama_kv_cache: layer 18: dev = CPU\n", "llama_kv_cache: layer 19: dev = CPU\n", "llama_kv_cache: layer 20: dev = CPU\n", "llama_kv_cache: layer 21: dev = CPU\n", "llama_kv_cache: layer 22: dev = CPU\n", "llama_kv_cache: layer 23: dev = CPU\n", "llama_kv_cache: layer 24: dev = CPU\n", "llama_kv_cache: layer 25: dev = CPU\n", "llama_kv_cache: layer 26: dev = CPU\n", "llama_kv_cache: layer 27: dev = CPU\n", "llama_kv_cache: layer 28: dev = CPU\n", "llama_kv_cache: layer 29: dev = CPU\n", "llama_kv_cache: layer 30: dev = CPU\n", "llama_kv_cache: layer 31: dev = CPU\n", "llama_kv_cache: CPU KV buffer size = 64.00 MiB\n", "llama_kv_cache: size = 64.00 MiB ( 512 cells, 32 layers, 1/1 seqs), K (f16): 32.00 MiB, V (f16): 32.00 MiB\n", "llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 128\n", "llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 128\n", "llama_context: enumerating backends\n", "llama_context: backend_ptrs.size() = 2\n", "sched_reserve: reserving ...\n", "sched_reserve: max_nodes = 2328\n", "sched_reserve: reserving full memory module\n", "sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1\n", "sched_reserve: resolving fused Gated Delta Net support:\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 1, n_seqs = 1, n_outputs = 1\n", "sched_reserve: fused Gated Delta Net (autoregressive) enabled\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 16, n_seqs = 1, n_outputs = 16\n", "sched_reserve: fused Gated Delta Net (chunked) enabled\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 512, n_seqs = 1, n_outputs = 512\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 1, n_seqs = 1, n_outputs = 1\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 512, n_seqs = 1, n_outputs = 512\n", "sched_reserve: CUDA0 compute buffer size = 194.94 MiB\n", "sched_reserve: CUDA_Host compute buffer size = 21.01 MiB\n", "sched_reserve: graph nodes = 1158\n", "sched_reserve: graph splits = 356 (with bs=512), 1 (with bs=1)\n", "sched_reserve: reserve took 26.86 ms, sched copies = 1\n", "CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | \n", "Model metadata: {'quantize.imatrix.entries_count': '224', 'quantize.imatrix.dataset': '/training_data/calibration_data.txt', 'quantize.imatrix.chunks_count': '228', 'quantize.imatrix.file': '/models/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3.imatrix', 'tokenizer.chat_template': \"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.architecture': 'llama', 'llama.rope.freq_base': '1000000.000000', 'tokenizer.ggml.pre': 'default', 'llama.context_length': '32768', 'general.name': 'Mistral-7B-Instruct-v0.3', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.file_type': '15', 'llama.vocab_size': '32768', 'llama.rope.dimension_count': '128'}\n", "Available chat formats from metadata: chat_template.default\n", "Guessed chat format: mistral-instruct\n" ] } ], "source": [ "# 指定本地模型的路径\n", "model_path = \"./Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf\"\n", "\n", "# 加载模型\n", "llm = Llama(\n", " model_path=model_path,\n", " # n_gpu_layers=-1, # 取消注释使用 GPU 加速\n", " # verbose=False, # 取消注释减少模型方面的信息打印\n", ")" ] }, { "cell_type": "markdown", "id": "94dfa7fc", "metadata": {}, "source": [ "#### 自动下载并导入\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "e4d1c736", "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "llama_model_loader: loaded meta data with 29 key-value pairs and 291 tensors from /root/.cache/huggingface/hub/models--bartowski--Mistral-7B-Instruct-v0.3-GGUF/snapshots/61fd4167fff3ab01ee1cfe0da183fa27a944db48/./Mistral-7B-Instruct-v0.3-Q4_K_M.gguf (version GGUF V3 (latest))\n", "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n", "llama_model_loader: - kv 0: general.architecture str = llama\n", "llama_model_loader: - kv 1: general.name str = Mistral-7B-Instruct-v0.3\n", "llama_model_loader: - kv 2: llama.block_count u32 = 32\n", "llama_model_loader: - kv 3: llama.context_length u32 = 32768\n", "llama_model_loader: - kv 4: llama.embedding_length u32 = 4096\n", "llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336\n", "llama_model_loader: - kv 6: llama.attention.head_count u32 = 32\n", "llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8\n", "llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000\n", "llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010\n", "llama_model_loader: - kv 10: general.file_type u32 = 15\n", "llama_model_loader: - kv 11: llama.vocab_size u32 = 32768\n", "llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128\n", "llama_model_loader: - kv 13: tokenizer.ggml.model str = llama\n", "llama_model_loader: - kv 14: tokenizer.ggml.pre str = default\n", "llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32768] = [\"\", \"\", \"\", \"[INST]\", \"[...\n", "llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32768] = [0.000000, 0.000000, 0.000000, 0.0000...\n", "llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32768] = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n", "llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1\n", "llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 2\n", "llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0\n", "llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true\n", "llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false\n", "llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...\n", "llama_model_loader: - kv 24: general.quantization_version u32 = 2\n", "llama_model_loader: - kv 25: quantize.imatrix.file str = /models/Mistral-7B-Instruct-v0.3-GGUF...\n", "llama_model_loader: - kv 26: quantize.imatrix.dataset str = /training_data/calibration_data.txt\n", "llama_model_loader: - kv 27: quantize.imatrix.entries_count i32 = 224\n", "llama_model_loader: - kv 28: quantize.imatrix.chunks_count i32 = 228\n", "llama_model_loader: - type f32: 65 tensors\n", "llama_model_loader: - type q4_K: 193 tensors\n", "llama_model_loader: - type q6_K: 33 tensors\n", "print_info: file format = GGUF V3 (latest)\n", "print_info: file type = Q4_K - Medium\n", "print_info: file size = 4.07 GiB (4.83 BPW) \n", "llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 4090 D) (0000:d6:00.0) - 23528 MiB free\n", "init_tokenizer: initializing tokenizer for type 1\n", "load: 0 unused tokens\n", "load: control token: 468 '[control_466]' is not marked as EOG\n", "load: control token: 464 '[control_462]' is not marked as EOG\n", "load: control token: 727 '[control_725]' is not marked as EOG\n", "load: control token: 343 '[control_341]' is not marked as EOG\n", "load: control token: 603 '[control_601]' is not marked as EOG\n", "load: control token: 332 '[control_330]' is not marked as EOG\n", "load: control token: 34 '[control_32]' is not marked as EOG\n", "load: control token: 412 '[control_410]' is not marked as EOG\n", "load: control token: 675 '[control_673]' is not marked as EOG\n", "load: control token: 177 '[control_175]' is not marked as EOG\n", "load: control token: 434 '[control_432]' is not marked as EOG\n", "load: control token: 166 '[control_164]' is not marked as EOG\n", "load: control token: 199 '[control_197]' is not marked as EOG\n", "load: control token: 252 '[control_250]' is not marked as EOG\n", "load: control token: 742 '[control_740]' is not marked as EOG\n", "load: control token: 611 '[control_609]' is not marked as EOG\n", "load: control token: 573 '[control_571]' is not marked as EOG\n", "load: control token: 455 '[control_453]' is not marked as EOG\n", "load: control token: 701 '[control_699]' is not marked as EOG\n", "load: control token: 583 '[control_581]' is not marked as EOG\n", "load: control token: 433 '[control_431]' is not marked as EOG\n", "load: control token: 301 '[control_299]' is not marked as EOG\n", "load: control token: 617 '[control_615]' is not marked as EOG\n", "load: control token: 536 '[control_534]' is not marked as EOG\n", "load: control token: 285 '[control_283]' is not marked as EOG\n", "load: control token: 376 '[control_374]' is not marked as EOG\n", "load: control token: 79 '[control_77]' is not marked as EOG\n", "load: control token: 671 '[control_669]' is not marked as EOG\n", "load: control token: 391 '[control_389]' is not marked as EOG\n", "load: control token: 174 '[control_172]' is not marked as EOG\n", "load: control token: 708 '[control_706]' is not marked as EOG\n", "load: control token: 729 '[control_727]' is not marked as EOG\n", "load: control token: 311 '[control_309]' is not marked as EOG\n", "load: control token: 384 '[control_382]' is not marked as EOG\n", "load: control token: 766 '[control_764]' is not marked as EOG\n", "load: control token: 94 '[control_92]' is not marked as EOG\n", "load: control token: 304 '[control_302]' is not marked as EOG\n", "load: control token: 197 '[control_195]' is not marked as EOG\n", "load: control token: 17 '[control_15]' is not marked as EOG\n", "load: control token: 604 '[control_602]' is not marked as EOG\n", "load: control token: 576 '[control_574]' is not marked as EOG\n", "load: control token: 359 '[control_357]' is not marked as EOG\n", "load: control token: 302 '[control_300]' is not marked as EOG\n", "load: control token: 453 '[control_451]' is not marked as EOG\n", "load: control token: 374 '[control_372]' is not marked as EOG\n", "load: control token: 207 '[control_205]' is not marked as EOG\n", "load: control token: 739 '[control_737]' is not marked as EOG\n", "load: control token: 689 '[control_687]' is not marked as EOG\n", "load: control token: 335 '[control_333]' is not marked as EOG\n", "load: control token: 485 '[control_483]' is not marked as EOG\n", "load: control token: 283 '[control_281]' is not marked as EOG\n", "load: control token: 249 '[control_247]' is not marked as EOG\n", "load: control token: 752 '[control_750]' is not marked as EOG\n", "load: control token: 330 '[control_328]' is not marked as EOG\n", "load: control token: 635 '[control_633]' is not marked as EOG\n", "load: control token: 161 '[control_159]' is not marked as EOG\n", "load: control token: 731 '[control_729]' is not marked as EOG\n", "load: control token: 631 '[control_629]' is not marked as EOG\n", "load: control token: 323 '[control_321]' is not marked as EOG\n", "load: control token: 48 '[control_46]' is not marked as EOG\n", "load: control token: 544 '[control_542]' is not marked as EOG\n", "load: control token: 687 '[control_685]' is not marked as EOG\n", "load: control token: 463 '[control_461]' is not marked as EOG\n", "load: control token: 47 '[control_45]' is not marked as EOG\n", "load: control token: 86 '[control_84]' is not marked as EOG\n", "load: control token: 331 '[control_329]' is not marked as EOG\n", "load: control token: 13 '[control_11]' is not marked as EOG\n", "load: control token: 760 '[control_758]' is not marked as EOG\n", "load: control token: 612 '[control_610]' is not marked as EOG\n", "load: control token: 473 '[control_471]' is not marked as EOG\n", "load: control token: 601 '[control_599]' is not marked as EOG\n", "load: control token: 146 '[control_144]' is not marked as EOG\n", "load: control token: 734 '[control_732]' is not marked as EOG\n", "load: control token: 141 '[control_139]' is not marked as EOG\n", "load: control token: 138 '[control_136]' is not marked as EOG\n", "load: control token: 61 '[control_59]' is not marked as EOG\n", "load: control token: 46 '[control_44]' is not marked as EOG\n", "load: control token: 173 '[control_171]' is not marked as EOG\n", "load: control token: 367 '[control_365]' is not marked as EOG\n", "load: control token: 417 '[control_415]' is not marked as EOG\n", "load: control token: 740 '[control_738]' is not marked as EOG\n", "load: control token: 216 '[control_214]' is not marked as EOG\n", "load: control token: 498 '[control_496]' is not marked as EOG\n", "load: control token: 512 '[control_510]' is not marked as EOG\n", "load: control token: 122 '[control_120]' is not marked as EOG\n", "load: control token: 246 '[control_244]' is not marked as EOG\n", "load: control token: 745 '[control_743]' is not marked as EOG\n", "load: control token: 451 '[control_449]' is not marked as EOG\n", "load: control token: 280 '[control_278]' is not marked as EOG\n", "load: control token: 654 '[control_652]' is not marked as EOG\n", "load: control token: 679 '[control_677]' is not marked as EOG\n", "load: control token: 77 '[control_75]' is not marked as EOG\n", "load: control token: 638 '[control_636]' is not marked as EOG\n", "load: control token: 472 '[control_470]' is not marked as EOG\n", "load: control token: 59 '[control_57]' is not marked as EOG\n", "load: control token: 145 '[control_143]' is not marked as EOG\n", "load: control token: 318 '[control_316]' is not marked as EOG\n", "load: control token: 640 '[control_638]' is not marked as EOG\n", "load: control token: 690 '[control_688]' is not marked as EOG\n", "load: control token: 256 '[control_254]' is not marked as EOG\n", "load: control token: 476 '[control_474]' is not marked as EOG\n", "load: control token: 21 '[control_19]' is not marked as EOG\n", "load: control token: 288 '[control_286]' is not marked as EOG\n", "load: control token: 255 '[control_253]' is not marked as EOG\n", "load: control token: 113 '[control_111]' is not marked as EOG\n", "load: control token: 190 '[control_188]' is not marked as EOG\n", "load: control token: 108 '[control_106]' is not marked as EOG\n", "load: control token: 211 '[control_209]' is not marked as EOG\n", "load: control token: 551 '[control_549]' is not marked as EOG\n", "load: control token: 14 '[control_12]' is not marked as EOG\n", "load: control token: 737 '[control_735]' is not marked as EOG\n", "load: control token: 555 '[control_553]' is not marked as EOG\n", "load: control token: 227 '[control_225]' is not marked as EOG\n", "load: control token: 210 '[control_208]' is not marked as EOG\n", "load: control token: 230 '[control_228]' is not marked as EOG\n", "load: control token: 7 '[/AVAILABLE_TOOLS]' is not marked as EOG\n", "load: control token: 55 '[control_53]' is not marked as EOG\n", "load: control token: 525 '[control_523]' is not marked as EOG\n", "load: control token: 533 '[control_531]' is not marked as EOG\n", "load: control token: 195 '[control_193]' is not marked as EOG\n", "load: control token: 584 '[control_582]' is not marked as EOG\n", "load: control token: 203 '[control_201]' is not marked as EOG\n", "load: control token: 322 '[control_320]' is not marked as EOG\n", "load: control token: 757 '[control_755]' is not marked as EOG\n", "load: control token: 642 '[control_640]' is not marked as EOG\n", "load: control token: 168 '[control_166]' is not marked as EOG\n", "load: control token: 389 '[control_387]' is not marked as EOG\n", "load: control token: 368 '[control_366]' is not marked as EOG\n", "load: control token: 767 '[control_765]' is not marked as EOG\n", "load: control token: 222 '[control_220]' is not marked as EOG\n", "load: control token: 420 '[control_418]' is not marked as EOG\n", "load: control token: 530 '[control_528]' is not marked as EOG\n", "load: control token: 535 '[control_533]' is not marked as EOG\n", "load: control token: 765 '[control_763]' is not marked as EOG\n", "load: control token: 661 '[control_659]' is not marked as EOG\n", "load: control token: 88 '[control_86]' is not marked as EOG\n", "load: control token: 581 '[control_579]' is not marked as EOG\n", "load: control token: 327 '[control_325]' is not marked as EOG\n", "load: control token: 201 '[control_199]' is not marked as EOG\n", "load: control token: 115 '[control_113]' is not marked as EOG\n", "load: control token: 709 '[control_707]' is not marked as EOG\n", "load: control token: 291 '[control_289]' is not marked as EOG\n", "load: control token: 265 '[control_263]' is not marked as EOG\n", "load: control token: 148 '[control_146]' is not marked as EOG\n", "load: control token: 185 '[control_183]' is not marked as EOG\n", "load: control token: 574 '[control_572]' is not marked as EOG\n", "load: control token: 360 '[control_358]' is not marked as EOG\n", "load: control token: 127 '[control_125]' is not marked as EOG\n", "load: control token: 325 '[control_323]' is not marked as EOG\n", "load: control token: 183 '[control_181]' is not marked as EOG\n", "load: control token: 116 '[control_114]' is not marked as EOG\n", "load: control token: 540 '[control_538]' is not marked as EOG\n", "load: control token: 293 '[control_291]' is not marked as EOG\n", "load: control token: 559 '[control_557]' is not marked as EOG\n", "load: control token: 527 '[control_525]' is not marked as EOG\n", "load: control token: 156 '[control_154]' is not marked as EOG\n", "load: control token: 338 '[control_336]' is not marked as EOG\n", "load: control token: 519 '[control_517]' is not marked as EOG\n", "load: control token: 516 '[control_514]' is not marked as EOG\n", "load: control token: 10 '[control_8]' is not marked as EOG\n", "load: control token: 8 '[TOOL_RESULTS]' is not marked as EOG\n", "load: control token: 505 '[control_503]' is not marked as EOG\n", "load: control token: 503 '[control_501]' is not marked as EOG\n", "load: control token: 500 '[control_498]' is not marked as EOG\n", "load: control token: 496 '[control_494]' is not marked as EOG\n", "load: control token: 492 '[control_490]' is not marked as EOG\n", "load: control token: 489 '[control_487]' is not marked as EOG\n", "load: control token: 538 '[control_536]' is not marked as EOG\n", "load: control token: 596 '[control_594]' is not marked as EOG\n", "load: control token: 481 '[control_479]' is not marked as EOG\n", "load: control token: 475 '[control_473]' is not marked as EOG\n", "load: control token: 336 '[control_334]' is not marked as EOG\n", "load: control token: 670 '[control_668]' is not marked as EOG\n", "load: control token: 50 '[control_48]' is not marked as EOG\n", "load: control token: 456 '[control_454]' is not marked as EOG\n", "load: control token: 105 '[control_103]' is not marked as EOG\n", "load: control token: 421 '[control_419]' is not marked as EOG\n", "load: control token: 430 '[control_428]' is not marked as EOG\n", "load: control token: 429 '[control_427]' is not marked as EOG\n", "load: control token: 575 '[control_573]' is not marked as EOG\n", "load: control token: 425 '[control_423]' is not marked as EOG\n", "load: control token: 424 '[control_422]' is not marked as EOG\n", "load: control token: 680 '[control_678]' is not marked as EOG\n", "load: control token: 57 '[control_55]' is not marked as EOG\n", "load: control token: 356 '[control_354]' is not marked as EOG\n", "load: control token: 458 '[control_456]' is not marked as EOG\n", "load: control token: 313 '[control_311]' is not marked as EOG\n", "load: control token: 418 '[control_416]' is not marked as EOG\n", "load: control token: 70 '[control_68]' is not marked as EOG\n", "load: control token: 759 '[control_757]' is not marked as EOG\n", "load: control token: 416 '[control_414]' is not marked as EOG\n", "load: control token: 238 '[control_236]' is not marked as EOG\n", "load: control token: 568 '[control_566]' is not marked as EOG\n", "load: control token: 409 '[control_407]' is not marked as EOG\n", "load: control token: 550 '[control_548]' is not marked as EOG\n", "load: control token: 571 '[control_569]' is not marked as EOG\n", "load: control token: 618 '[control_616]' is not marked as EOG\n", "load: control token: 623 '[control_621]' is not marked as EOG\n", "load: control token: 247 '[control_245]' is not marked as EOG\n", "load: control token: 400 '[control_398]' is not marked as EOG\n", "load: control token: 396 '[control_394]' is not marked as EOG\n", "load: control token: 392 '[control_390]' is not marked as EOG\n", "load: control token: 552 '[control_550]' is not marked as EOG\n", "load: control token: 651 '[control_649]' is not marked as EOG\n", "load: control token: 390 '[control_388]' is not marked as EOG\n", "load: control token: 186 '[control_184]' is not marked as EOG\n", "load: control token: 565 '[control_563]' is not marked as EOG\n", "load: control token: 92 '[control_90]' is not marked as EOG\n", "load: control token: 508 '[control_506]' is not marked as EOG\n", "load: control token: 373 '[control_371]' is not marked as EOG\n", "load: control token: 299 '[control_297]' is not marked as EOG\n", "load: control token: 562 '[control_560]' is not marked as EOG\n", "load: control token: 537 '[control_535]' is not marked as EOG\n", "load: control token: 372 '[control_370]' is not marked as EOG\n", "load: control token: 366 '[control_364]' is not marked as EOG\n", "load: control token: 196 '[control_194]' is not marked as EOG\n", "load: control token: 478 '[control_476]' is not marked as EOG\n", "load: control token: 621 '[control_619]' is not marked as EOG\n", "load: control token: 53 '[control_51]' is not marked as EOG\n", "load: control token: 40 '[control_38]' is not marked as EOG\n", "load: control token: 347 '[control_345]' is not marked as EOG\n", "load: control token: 345 '[control_343]' is not marked as EOG\n", "load: control token: 339 '[control_337]' is not marked as EOG\n", "load: control token: 234 '[control_232]' is not marked as EOG\n", "load: control token: 770 '[control_768]' is not marked as EOG\n", "load: control token: 444 '[control_442]' is not marked as EOG\n", "load: control token: 317 '[control_315]' is not marked as EOG\n", "load: control token: 693 '[control_691]' is not marked as EOG\n", "load: control token: 321 '[control_319]' is not marked as EOG\n", "load: control token: 320 '[control_318]' is not marked as EOG\n", "load: control token: 179 '[control_177]' is not marked as EOG\n", "load: control token: 214 '[control_212]' is not marked as EOG\n", "load: control token: 666 '[control_664]' is not marked as EOG\n", "load: control token: 712 '[control_710]' is not marked as EOG\n", "load: control token: 303 '[control_301]' is not marked as EOG\n", "load: control token: 440 '[control_438]' is not marked as EOG\n", "load: control token: 314 '[control_312]' is not marked as EOG\n", "load: control token: 397 '[control_395]' is not marked as EOG\n", "load: control token: 312 '[control_310]' is not marked as EOG\n", "load: control token: 129 '[control_127]' is not marked as EOG\n", "load: control token: 545 '[control_543]' is not marked as EOG\n", "load: control token: 58 '[control_56]' is not marked as EOG\n", "load: control token: 509 '[control_507]' is not marked as EOG\n", "load: control token: 541 '[control_539]' is not marked as EOG\n", "load: control token: 443 '[control_441]' is not marked as EOG\n", "load: control token: 45 '[control_43]' is not marked as EOG\n", "load: control token: 469 '[control_467]' is not marked as EOG\n", "load: control token: 532 '[control_530]' is not marked as EOG\n", "load: control token: 64 '[control_62]' is not marked as EOG\n", "load: control token: 365 '[control_363]' is not marked as EOG\n", "load: control token: 563 '[control_561]' is not marked as EOG\n", "load: control token: 346 '[control_344]' is not marked as EOG\n", "load: control token: 282 '[control_280]' is not marked as EOG\n", "load: control token: 450 '[control_448]' is not marked as EOG\n", "load: control token: 526 '[control_524]' is not marked as EOG\n", "load: control token: 672 '[control_670]' is not marked as EOG\n", "load: control token: 232 '[control_230]' is not marked as EOG\n", "load: control token: 273 '[control_271]' is not marked as EOG\n", "load: control token: 272 '[control_270]' is not marked as EOG\n", "load: control token: 636 '[control_634]' is not marked as EOG\n", "load: control token: 260 '[control_258]' is not marked as EOG\n", "load: control token: 518 '[control_516]' is not marked as EOG\n", "load: control token: 353 '[control_351]' is not marked as EOG\n", "load: control token: 257 '[control_255]' is not marked as EOG\n", "load: control token: 126 '[control_124]' is not marked as EOG\n", "load: control token: 124 '[control_122]' is not marked as EOG\n", "load: control token: 751 '[control_749]' is not marked as EOG\n", "load: control token: 383 '[control_381]' is not marked as EOG\n", "load: control token: 175 '[control_173]' is not marked as EOG\n", "load: control token: 75 '[control_73]' is not marked as EOG\n", "load: control token: 72 '[control_70]' is not marked as EOG\n", "load: control token: 614 '[control_612]' is not marked as EOG\n", "load: control token: 167 '[control_165]' is not marked as EOG\n", "load: control token: 465 '[control_463]' is not marked as EOG\n", "load: control token: 341 '[control_339]' is not marked as EOG\n", "load: control token: 703 '[control_701]' is not marked as EOG\n", "load: control token: 501 '[control_499]' is not marked as EOG\n", "load: control token: 90 '[control_88]' is not marked as EOG\n", "load: control token: 89 '[control_87]' is not marked as EOG\n", "load: control token: 593 '[control_591]' is not marked as EOG\n", "load: control token: 107 '[control_105]' is not marked as EOG\n", "load: control token: 241 '[control_239]' is not marked as EOG\n", "load: control token: 713 '[control_711]' is not marked as EOG\n", "load: control token: 212 '[control_210]' is not marked as EOG\n", "load: control token: 80 '[control_78]' is not marked as EOG\n", "load: control token: 707 '[control_705]' is not marked as EOG\n", "load: control token: 371 '[control_369]' is not marked as EOG\n", "load: control token: 628 '[control_626]' is not marked as EOG\n", "load: control token: 15 '[control_13]' is not marked as EOG\n", "load: control token: 354 '[control_352]' is not marked as EOG\n", "load: control token: 25 '[control_23]' is not marked as EOG\n", "load: control token: 261 '[control_259]' is not marked as EOG\n", "load: control token: 435 '[control_433]' is not marked as EOG\n", "load: control token: 307 '[control_305]' is not marked as EOG\n", "load: control token: 305 '[control_303]' is not marked as EOG\n", "load: control token: 656 '[control_654]' is not marked as EOG\n", "load: control token: 702 '[control_700]' is not marked as EOG\n", "load: control token: 428 '[control_426]' is not marked as EOG\n", "load: control token: 499 '[control_497]' is not marked as EOG\n", "load: control token: 154 '[control_152]' is not marked as EOG\n", "load: control token: 308 '[control_306]' is not marked as EOG\n", "load: control token: 286 '[control_284]' is not marked as EOG\n", "load: control token: 634 '[control_632]' is not marked as EOG\n", "load: control token: 51 '[control_49]' is not marked as EOG\n", "load: control token: 599 '[control_597]' is not marked as EOG\n", "load: control token: 606 '[control_604]' is not marked as EOG\n", "load: control token: 30 '[control_28]' is not marked as EOG\n", "load: control token: 250 '[control_248]' is not marked as EOG\n", "load: control token: 714 '[control_712]' is not marked as EOG\n", "load: control token: 270 '[control_268]' is not marked as EOG\n", "load: control token: 83 '[control_81]' is not marked as EOG\n", "load: control token: 730 '[control_728]' is not marked as EOG\n", "load: control token: 743 '[control_741]' is not marked as EOG\n", "load: control token: 484 '[control_482]' is not marked as EOG\n", "load: control token: 758 '[control_756]' is not marked as EOG\n", "load: control token: 271 '[control_269]' is not marked as EOG\n", "load: control token: 279 '[control_277]' is not marked as EOG\n", "load: control token: 73 '[control_71]' is not marked as EOG\n", "load: control token: 363 '[control_361]' is not marked as EOG\n", "load: control token: 60 '[control_58]' is not marked as EOG\n", "load: control token: 405 '[control_403]' is not marked as EOG\n", "load: control token: 220 '[control_218]' is not marked as EOG\n", "load: control token: 436 '[control_434]' is not marked as EOG\n", "load: control token: 27 '[control_25]' is not marked as EOG\n", "load: control token: 652 '[control_650]' is not marked as EOG\n", "load: control token: 646 '[control_644]' is not marked as EOG\n", "load: control token: 395 '[control_393]' is not marked as EOG\n", "load: control token: 549 '[control_547]' is not marked as EOG\n", "load: control token: 12 '[control_10]' is not marked as EOG\n", "load: control token: 229 '[control_227]' is not marked as EOG\n", "load: control token: 162 '[control_160]' is not marked as EOG\n", "load: control token: 497 '[control_495]' is not marked as EOG\n", "load: control token: 31 '[control_29]' is not marked as EOG\n", "load: control token: 98 '[control_96]' is not marked as EOG\n", "load: control token: 686 '[control_684]' is not marked as EOG\n", "load: control token: 3 '[INST]' is not marked as EOG\n", "load: control token: 155 '[control_153]' is not marked as EOG\n", "load: control token: 470 '[control_468]' is not marked as EOG\n", "load: control token: 69 '[control_67]' is not marked as EOG\n", "load: control token: 93 '[control_91]' is not marked as EOG\n", "load: control token: 71 '[control_69]' is not marked as EOG\n", "load: control token: 11 '[control_9]' is not marked as EOG\n", "load: control token: 43 '[control_41]' is not marked as EOG\n", "load: control token: 22 '[control_20]' is not marked as EOG\n", "load: control token: 35 '[control_33]' is not marked as EOG\n", "load: control token: 706 '[control_704]' is not marked as EOG\n", "load: control token: 511 '[control_509]' is not marked as EOG\n", "load: control token: 491 '[control_489]' is not marked as EOG\n", "load: control token: 324 '[control_322]' is not marked as EOG\n", "load: control token: 655 '[control_653]' is not marked as EOG\n", "load: control token: 117 '[control_115]' is not marked as EOG\n", "load: control token: 665 '[control_663]' is not marked as EOG\n", "load: control token: 9 '[/TOOL_RESULTS]' is not marked as EOG\n", "load: control token: 29 '[control_27]' is not marked as EOG\n", "load: control token: 44 '[control_42]' is not marked as EOG\n", "load: control token: 310 '[control_308]' is not marked as EOG\n", "load: control token: 157 '[control_155]' is not marked as EOG\n", "load: control token: 515 '[control_513]' is not marked as EOG\n", "load: control token: 388 '[control_386]' is not marked as EOG\n", "load: control token: 438 '[control_436]' is not marked as EOG\n", "load: control token: 486 '[control_484]' is not marked as EOG\n", "load: control token: 139 '[control_137]' is not marked as EOG\n", "load: control token: 543 '[control_541]' is not marked as EOG\n", "load: control token: 622 '[control_620]' is not marked as EOG\n", "load: control token: 598 '[control_596]' is not marked as EOG\n", "load: control token: 16 '[control_14]' is not marked as EOG\n", "load: control token: 39 '[control_37]' is not marked as EOG\n", "load: control token: 102 '[control_100]' is not marked as EOG\n", "load: control token: 673 '[control_671]' is not marked as EOG\n", "load: control token: 287 '[control_285]' is not marked as EOG\n", "load: control token: 244 '[control_242]' is not marked as EOG\n", "load: control token: 446 '[control_444]' is not marked as EOG\n", "load: control token: 483 '[control_481]' is not marked as EOG\n", "load: control token: 467 '[control_465]' is not marked as EOG\n", "load: control token: 264 '[control_262]' is not marked as EOG\n", "load: control token: 176 '[control_174]' is not marked as EOG\n", "load: control token: 625 '[control_623]' is not marked as EOG\n", "load: control token: 160 '[control_158]' is not marked as EOG\n", "load: control token: 20 '[control_18]' is not marked as EOG\n", "load: control token: 641 '[control_639]' is not marked as EOG\n", "load: control token: 99 '[control_97]' is not marked as EOG\n", "load: control token: 54 '[control_52]' is not marked as EOG\n", "load: control token: 37 '[control_35]' is not marked as EOG\n", "load: control token: 401 '[control_399]' is not marked as EOG\n", "load: control token: 130 '[control_128]' is not marked as EOG\n", "load: control token: 218 '[control_216]' is not marked as EOG\n", "load: control token: 243 '[control_241]' is not marked as EOG\n", "load: control token: 753 '[control_751]' is not marked as EOG\n", "load: control token: 74 '[control_72]' is not marked as EOG\n", "load: control token: 171 '[control_169]' is not marked as EOG\n", "load: control token: 151 '[control_149]' is not marked as EOG\n", "load: control token: 364 '[control_362]' is not marked as EOG\n", "load: control token: 495 '[control_493]' is not marked as EOG\n", "load: control token: 119 '[control_117]' is not marked as EOG\n", "load: control token: 217 '[control_215]' is not marked as EOG\n", "load: control token: 84 '[control_82]' is not marked as EOG\n", "load: control token: 5 '[TOOL_CALLS]' is not marked as EOG\n", "load: control token: 531 '[control_529]' is not marked as EOG\n", "load: control token: 33 '[control_31]' is not marked as EOG\n", "load: control token: 82 '[control_80]' is not marked as EOG\n", "load: control token: 100 '[control_98]' is not marked as EOG\n", "load: control token: 125 '[control_123]' is not marked as EOG\n", "load: control token: 520 '[control_518]' is not marked as EOG\n", "load: control token: 144 '[control_142]' is not marked as EOG\n", "load: control token: 152 '[control_150]' is not marked as EOG\n", "load: control token: 592 '[control_590]' is not marked as EOG\n", "load: control token: 461 '[control_459]' is not marked as EOG\n", "load: control token: 624 '[control_622]' is not marked as EOG\n", "load: control token: 65 '[control_63]' is not marked as EOG\n", "load: control token: 172 '[control_170]' is not marked as EOG\n", "load: control token: 178 '[control_176]' is not marked as EOG\n", "load: control token: 258 '[control_256]' is not marked as EOG\n", "load: control token: 524 '[control_522]' is not marked as EOG\n", "load: control token: 657 '[control_655]' is not marked as EOG\n", "load: control token: 184 '[control_182]' is not marked as EOG\n", "load: control token: 441 '[control_439]' is not marked as EOG\n", "load: control token: 315 '[control_313]' is not marked as EOG\n", "load: control token: 662 '[control_660]' is not marked as EOG\n", "load: control token: 6 '[AVAILABLE_TOOLS]' is not marked as EOG\n", "load: control token: 517 '[control_515]' is not marked as EOG\n", "load: control token: 194 '[control_192]' is not marked as EOG\n", "load: control token: 447 '[control_445]' is not marked as EOG\n", "load: control token: 52 '[control_50]' is not marked as EOG\n", "load: control token: 718 '[control_716]' is not marked as EOG\n", "load: control token: 692 '[control_690]' is not marked as EOG\n", "load: control token: 290 '[control_288]' is not marked as EOG\n", "load: control token: 763 '[control_761]' is not marked as EOG\n", "load: control token: 362 '[control_360]' is not marked as EOG\n", "load: control token: 200 '[control_198]' is not marked as EOG\n", "load: control token: 204 '[control_202]' is not marked as EOG\n", "load: control token: 722 '[control_720]' is not marked as EOG\n", "load: control token: 351 '[control_349]' is not marked as EOG\n", "load: control token: 349 '[control_347]' is not marked as EOG\n", "load: control token: 38 '[control_36]' is not marked as EOG\n", "load: control token: 134 '[control_132]' is not marked as EOG\n", "load: control token: 67 '[control_65]' is not marked as EOG\n", "load: control token: 239 '[control_237]' is not marked as EOG\n", "load: control token: 570 '[control_568]' is not marked as EOG\n", "load: control token: 253 '[control_251]' is not marked as EOG\n", "load: control token: 221 '[control_219]' is not marked as EOG\n", "load: control token: 664 '[control_662]' is not marked as EOG\n", "load: control token: 225 '[control_223]' is not marked as EOG\n", "load: control token: 226 '[control_224]' is not marked as EOG\n", "load: control token: 553 '[control_551]' is not marked as EOG\n", "load: control token: 242 '[control_240]' is not marked as EOG\n", "load: control token: 633 '[control_631]' is not marked as EOG\n", "load: control token: 580 '[control_578]' is not marked as EOG\n", "load: control token: 245 '[control_243]' is not marked as EOG\n", "load: control token: 620 '[control_618]' is not marked as EOG\n", "load: control token: 191 '[control_189]' is not marked as EOG\n", "load: control token: 87 '[control_85]' is not marked as EOG\n", "load: control token: 189 '[control_187]' is not marked as EOG\n", "load: control token: 542 '[control_540]' is not marked as EOG\n", "load: control token: 548 '[control_546]' is not marked as EOG\n", "load: control token: 558 '[control_556]' is not marked as EOG\n", "load: control token: 560 '[control_558]' is not marked as EOG\n", "load: control token: 564 '[control_562]' is not marked as EOG\n", "load: control token: 410 '[control_408]' is not marked as EOG\n", "load: control token: 566 '[control_564]' is not marked as EOG\n", "load: control token: 649 '[control_647]' is not marked as EOG\n", "load: control token: 414 '[control_412]' is not marked as EOG\n", "load: control token: 567 '[control_565]' is not marked as EOG\n", "load: control token: 569 '[control_567]' is not marked as EOG\n", "load: control token: 761 '[control_759]' is not marked as EOG\n", "load: control token: 408 '[control_406]' is not marked as EOG\n", "load: control token: 579 '[control_577]' is not marked as EOG\n", "load: control token: 147 '[control_145]' is not marked as EOG\n", "load: control token: 236 '[control_234]' is not marked as EOG\n", "load: control token: 588 '[control_586]' is not marked as EOG\n", "load: control token: 292 '[control_290]' is not marked as EOG\n", "load: control token: 591 '[control_589]' is not marked as EOG\n", "load: control token: 645 '[control_643]' is not marked as EOG\n", "load: control token: 140 '[control_138]' is not marked as EOG\n", "load: control token: 422 '[control_420]' is not marked as EOG\n", "load: control token: 595 '[control_593]' is not marked as EOG\n", "load: control token: 597 '[control_595]' is not marked as EOG\n", "load: control token: 754 '[control_752]' is not marked as EOG\n", "load: control token: 159 '[control_157]' is not marked as EOG\n", "load: control token: 344 '[control_342]' is not marked as EOG\n", "load: control token: 393 '[control_391]' is not marked as EOG\n", "load: control token: 691 '[control_689]' is not marked as EOG\n", "load: control token: 608 '[control_606]' is not marked as EOG\n", "load: control token: 663 '[control_661]' is not marked as EOG\n", "load: control token: 613 '[control_611]' is not marked as EOG\n", "load: control token: 340 '[control_338]' is not marked as EOG\n", "load: control token: 616 '[control_614]' is not marked as EOG\n", "load: control token: 719 '[control_717]' is not marked as EOG\n", "load: control token: 521 '[control_519]' is not marked as EOG\n", "load: control token: 619 '[control_617]' is not marked as EOG\n", "load: control token: 523 '[control_521]' is not marked as EOG\n", "load: control token: 626 '[control_624]' is not marked as EOG\n", "load: control token: 695 '[control_693]' is not marked as EOG\n", "load: control token: 637 '[control_635]' is not marked as EOG\n", "load: control token: 235 '[control_233]' is not marked as EOG\n", "load: control token: 659 '[control_657]' is not marked as EOG\n", "load: control token: 268 '[control_266]' is not marked as EOG\n", "load: control token: 406 '[control_404]' is not marked as EOG\n", "load: control token: 735 '[control_733]' is not marked as EOG\n", "load: control token: 398 '[control_396]' is not marked as EOG\n", "load: control token: 674 '[control_672]' is not marked as EOG\n", "load: control token: 684 '[control_682]' is not marked as EOG\n", "load: control token: 462 '[control_460]' is not marked as EOG\n", "load: control token: 382 '[control_380]' is not marked as EOG\n", "load: control token: 697 '[control_695]' is not marked as EOG\n", "load: control token: 698 '[control_696]' is not marked as EOG\n", "load: control token: 699 '[control_697]' is not marked as EOG\n", "load: control token: 688 '[control_686]' is not marked as EOG\n", "load: control token: 704 '[control_702]' is not marked as EOG\n", "load: control token: 705 '[control_703]' is not marked as EOG\n", "load: control token: 205 '[control_203]' is not marked as EOG\n", "load: control token: 726 '[control_724]' is not marked as EOG\n", "load: control token: 180 '[control_178]' is not marked as EOG\n", "load: control token: 716 '[control_714]' is not marked as EOG\n", "load: control token: 590 '[control_588]' is not marked as EOG\n", "load: control token: 454 '[control_452]' is not marked as EOG\n", "load: control token: 728 '[control_726]' is not marked as EOG\n", "load: control token: 738 '[control_736]' is not marked as EOG\n", "load: control token: 744 '[control_742]' is not marked as EOG\n", "load: control token: 403 '[control_401]' is not marked as EOG\n", "load: control token: 764 '[control_762]' is not marked as EOG\n", "load: control token: 676 '[control_674]' is not marked as EOG\n", "load: control token: 756 '[control_754]' is not marked as EOG\n", "load: control token: 380 '[control_378]' is not marked as EOG\n", "load: control token: 319 '[control_317]' is not marked as EOG\n", "load: control token: 600 '[control_598]' is not marked as EOG\n", "load: control token: 297 '[control_295]' is not marked as EOG\n", "load: control token: 259 '[control_257]' is not marked as EOG\n", "load: control token: 639 '[control_637]' is not marked as EOG\n", "load: control token: 529 '[control_527]' is not marked as EOG\n", "load: control token: 101 '[control_99]' is not marked as EOG\n", "load: control token: 522 '[control_520]' is not marked as EOG\n", "load: control token: 723 '[control_721]' is not marked as EOG\n", "load: control token: 254 '[control_252]' is not marked as EOG\n", "load: control token: 513 '[control_511]' is not marked as EOG\n", "load: control token: 667 '[control_665]' is not marked as EOG\n", "load: control token: 267 '[control_265]' is not marked as EOG\n", "load: control token: 732 '[control_730]' is not marked as EOG\n", "load: control token: 24 '[control_22]' is not marked as EOG\n", "load: control token: 650 '[control_648]' is not marked as EOG\n", "load: control token: 528 '[control_526]' is not marked as EOG\n", "load: control token: 749 '[control_747]' is not marked as EOG\n", "load: control token: 721 '[control_719]' is not marked as EOG\n", "load: control token: 103 '[control_101]' is not marked as EOG\n", "load: control token: 294 '[control_292]' is not marked as EOG\n", "load: control token: 41 '[control_39]' is not marked as EOG\n", "load: control token: 133 '[control_131]' is not marked as EOG\n", "load: control token: 188 '[control_186]' is not marked as EOG\n", "load: control token: 276 '[control_274]' is not marked as EOG\n", "load: control token: 644 '[control_642]' is not marked as EOG\n", "load: control token: 648 '[control_646]' is not marked as EOG\n", "load: control token: 459 '[control_457]' is not marked as EOG\n", "load: control token: 112 '[control_110]' is not marked as EOG\n", "load: control token: 300 '[control_298]' is not marked as EOG\n", "load: control token: 206 '[control_204]' is not marked as EOG\n", "load: control token: 121 '[control_119]' is not marked as EOG\n", "load: control token: 750 '[control_748]' is not marked as EOG\n", "load: control token: 131 '[control_129]' is not marked as EOG\n", "load: control token: 474 '[control_472]' is not marked as EOG\n", "load: control token: 685 '[control_683]' is not marked as EOG\n", "load: control token: 251 '[control_249]' is not marked as EOG\n", "load: control token: 402 '[control_400]' is not marked as EOG\n", "load: control token: 281 '[control_279]' is not marked as EOG\n", "load: control token: 192 '[control_190]' is not marked as EOG\n", "load: control token: 387 '[control_385]' is not marked as EOG\n", "load: control token: 587 '[control_585]' is not marked as EOG\n", "load: control token: 66 '[control_64]' is not marked as EOG\n", "load: control token: 394 '[control_392]' is not marked as EOG\n", "load: control token: 49 '[control_47]' is not marked as EOG\n", "load: control token: 42 '[control_40]' is not marked as EOG\n", "load: control token: 32 '[control_30]' is not marked as EOG\n", "load: control token: 399 '[control_397]' is not marked as EOG\n", "load: control token: 448 '[control_446]' is not marked as EOG\n", "load: control token: 479 '[control_477]' is not marked as EOG\n", "load: control token: 493 '[control_491]' is not marked as EOG\n", "load: control token: 736 '[control_734]' is not marked as EOG\n", "load: control token: 348 '[control_346]' is not marked as EOG\n", "load: control token: 193 '[control_191]' is not marked as EOG\n", "load: control token: 231 '[control_229]' is not marked as EOG\n", "load: control token: 609 '[control_607]' is not marked as EOG\n", "load: control token: 213 '[control_211]' is not marked as EOG\n", "load: control token: 128 '[control_126]' is not marked as EOG\n", "load: control token: 118 '[control_116]' is not marked as EOG\n", "load: control token: 369 '[control_367]' is not marked as EOG\n", "load: control token: 630 '[control_628]' is not marked as EOG\n", "load: control token: 755 '[control_753]' is not marked as EOG\n", "load: control token: 747 '[control_745]' is not marked as EOG\n", "load: control token: 289 '[control_287]' is not marked as EOG\n", "load: control token: 26 '[control_24]' is not marked as EOG\n", "load: control token: 63 '[control_61]' is not marked as EOG\n", "load: control token: 284 '[control_282]' is not marked as EOG\n", "load: control token: 164 '[control_162]' is not marked as EOG\n", "load: control token: 404 '[control_402]' is not marked as EOG\n", "load: control token: 506 '[control_504]' is not marked as EOG\n", "load: control token: 123 '[control_121]' is not marked as EOG\n", "load: control token: 237 '[control_235]' is not marked as EOG\n", "load: control token: 62 '[control_60]' is not marked as EOG\n", "load: control token: 510 '[control_508]' is not marked as EOG\n", "load: control token: 233 '[control_231]' is not marked as EOG\n", "load: control token: 68 '[control_66]' is not marked as EOG\n", "load: control token: 487 '[control_485]' is not marked as EOG\n", "load: control token: 471 '[control_469]' is not marked as EOG\n", "load: control token: 415 '[control_413]' is not marked as EOG\n", "load: control token: 78 '[control_76]' is not marked as EOG\n", "load: control token: 610 '[control_608]' is not marked as EOG\n", "load: control token: 137 '[control_135]' is not marked as EOG\n", "load: control token: 539 '[control_537]' is not marked as EOG\n", "load: control token: 658 '[control_656]' is not marked as EOG\n", "load: control token: 158 '[control_156]' is not marked as EOG\n", "load: control token: 585 '[control_583]' is not marked as EOG\n", "load: control token: 215 '[control_213]' is not marked as EOG\n", "load: control token: 554 '[control_552]' is not marked as EOG\n", "load: control token: 762 '[control_760]' is not marked as EOG\n", "load: control token: 717 '[control_715]' is not marked as EOG\n", "load: control token: 228 '[control_226]' is not marked as EOG\n", "load: control token: 442 '[control_440]' is not marked as EOG\n", "load: control token: 494 '[control_492]' is not marked as EOG\n", "load: control token: 97 '[control_95]' is not marked as EOG\n", "load: control token: 683 '[control_681]' is not marked as EOG\n", "load: control token: 629 '[control_627]' is not marked as EOG\n", "load: control token: 725 '[control_723]' is not marked as EOG\n", "load: control token: 432 '[control_430]' is not marked as EOG\n", "load: control token: 477 '[control_475]' is not marked as EOG\n", "load: control token: 355 '[control_353]' is not marked as EOG\n", "load: control token: 36 '[control_34]' is not marked as EOG\n", "load: control token: 534 '[control_532]' is not marked as EOG\n", "load: control token: 120 '[control_118]' is not marked as EOG\n", "load: control token: 28 '[control_26]' is not marked as EOG\n", "load: control token: 306 '[control_304]' is not marked as EOG\n", "load: control token: 407 '[control_405]' is not marked as EOG\n", "load: control token: 696 '[control_694]' is not marked as EOG\n", "load: control token: 274 '[control_272]' is not marked as EOG\n", "load: control token: 142 '[control_140]' is not marked as EOG\n", "load: control token: 136 '[control_134]' is not marked as EOG\n", "load: control token: 309 '[control_307]' is not marked as EOG\n", "load: control token: 423 '[control_421]' is not marked as EOG\n", "load: control token: 724 '[control_722]' is not marked as EOG\n", "load: control token: 262 '[control_260]' is not marked as EOG\n", "load: control token: 419 '[control_417]' is not marked as EOG\n", "load: control token: 329 '[control_327]' is not marked as EOG\n", "load: control token: 427 '[control_425]' is not marked as EOG\n", "load: control token: 370 '[control_368]' is not marked as EOG\n", "load: control token: 266 '[control_264]' is not marked as EOG\n", "load: control token: 710 '[control_708]' is not marked as EOG\n", "load: control token: 602 '[control_600]' is not marked as EOG\n", "load: control token: 277 '[control_275]' is not marked as EOG\n", "load: control token: 298 '[control_296]' is not marked as EOG\n", "load: control token: 378 '[control_376]' is not marked as EOG\n", "load: control token: 198 '[control_196]' is not marked as EOG\n", "load: control token: 76 '[control_74]' is not marked as EOG\n", "load: control token: 607 '[control_605]' is not marked as EOG\n", "load: control token: 682 '[control_680]' is not marked as EOG\n", "load: control token: 333 '[control_331]' is not marked as EOG\n", "load: control token: 660 '[control_658]' is not marked as EOG\n", "load: control token: 163 '[control_161]' is not marked as EOG\n", "load: control token: 95 '[control_93]' is not marked as EOG\n", "load: control token: 377 '[control_375]' is not marked as EOG\n", "load: control token: 223 '[control_221]' is not marked as EOG\n", "load: control token: 326 '[control_324]' is not marked as EOG\n", "load: control token: 111 '[control_109]' is not marked as EOG\n", "load: control token: 187 '[control_185]' is not marked as EOG\n", "load: control token: 678 '[control_676]' is not marked as EOG\n", "load: control token: 514 '[control_512]' is not marked as EOG\n", "load: control token: 431 '[control_429]' is not marked as EOG\n", "load: control token: 4 '[/INST]' is not marked as EOG\n", "load: control token: 342 '[control_340]' is not marked as EOG\n", "load: control token: 594 '[control_592]' is not marked as EOG\n", "load: control token: 768 '[control_766]' is not marked as EOG\n", "load: control token: 143 '[control_141]' is not marked as EOG\n", "load: control token: 445 '[control_443]' is not marked as EOG\n", "load: control token: 165 '[control_163]' is not marked as EOG\n", "load: control token: 439 '[control_437]' is not marked as EOG\n", "load: control token: 181 '[control_179]' is not marked as EOG\n", "load: control token: 352 '[control_350]' is not marked as EOG\n", "load: control token: 643 '[control_641]' is not marked as EOG\n", "load: control token: 182 '[control_180]' is not marked as EOG\n", "load: control token: 19 '[control_17]' is not marked as EOG\n", "load: control token: 269 '[control_267]' is not marked as EOG\n", "load: control token: 677 '[control_675]' is not marked as EOG\n", "load: control token: 615 '[control_613]' is not marked as EOG\n", "load: control token: 170 '[control_168]' is not marked as EOG\n", "load: control token: 56 '[control_54]' is not marked as EOG\n", "load: control token: 582 '[control_580]' is not marked as EOG\n", "load: control token: 700 '[control_698]' is not marked as EOG\n", "load: control token: 561 '[control_559]' is not marked as EOG\n", "load: control token: 381 '[control_379]' is not marked as EOG\n", "load: control token: 647 '[control_645]' is not marked as EOG\n", "load: control token: 437 '[control_435]' is not marked as EOG\n", "load: control token: 507 '[control_505]' is not marked as EOG\n", "load: control token: 490 '[control_488]' is not marked as EOG\n", "load: control token: 586 '[control_584]' is not marked as EOG\n", "load: control token: 23 '[control_21]' is not marked as EOG\n", "load: control token: 452 '[control_450]' is not marked as EOG\n", "load: control token: 605 '[control_603]' is not marked as EOG\n", "load: control token: 426 '[control_424]' is not marked as EOG\n", "load: control token: 769 '[control_767]' is not marked as EOG\n", "load: control token: 81 '[control_79]' is not marked as EOG\n", "load: control token: 386 '[control_384]' is not marked as EOG\n", "load: control token: 627 '[control_625]' is not marked as EOG\n", "load: control token: 18 '[control_16]' is not marked as EOG\n", "load: control token: 741 '[control_739]' is not marked as EOG\n", "load: control token: 96 '[control_94]' is not marked as EOG\n", "load: control token: 502 '[control_500]' is not marked as EOG\n", "load: control token: 482 '[control_480]' is not marked as EOG\n", "load: control token: 480 '[control_478]' is not marked as EOG\n", "load: control token: 557 '[control_555]' is not marked as EOG\n", "load: control token: 263 '[control_261]' is not marked as EOG\n", "load: control token: 668 '[control_666]' is not marked as EOG\n", "load: control token: 466 '[control_464]' is not marked as EOG\n", "load: control token: 334 '[control_332]' is not marked as EOG\n", "load: control token: 413 '[control_411]' is not marked as EOG\n", "load: control token: 572 '[control_570]' is not marked as EOG\n", "load: control token: 295 '[control_293]' is not marked as EOG\n", "load: control token: 150 '[control_148]' is not marked as EOG\n", "load: control token: 275 '[control_273]' is not marked as EOG\n", "load: control token: 748 '[control_746]' is not marked as EOG\n", "load: control token: 135 '[control_133]' is not marked as EOG\n", "load: control token: 357 '[control_355]' is not marked as EOG\n", "load: control token: 577 '[control_575]' is not marked as EOG\n", "load: control token: 578 '[control_576]' is not marked as EOG\n", "load: control token: 556 '[control_554]' is not marked as EOG\n", "load: control token: 375 '[control_373]' is not marked as EOG\n", "load: control token: 589 '[control_587]' is not marked as EOG\n", "load: control token: 202 '[control_200]' is not marked as EOG\n", "load: control token: 85 '[control_83]' is not marked as EOG\n", "load: control token: 278 '[control_276]' is not marked as EOG\n", "load: control token: 169 '[control_167]' is not marked as EOG\n", "load: control token: 208 '[control_206]' is not marked as EOG\n", "load: control token: 488 '[control_486]' is not marked as EOG\n", "load: control token: 337 '[control_335]' is not marked as EOG\n", "load: control token: 733 '[control_731]' is not marked as EOG\n", "load: control token: 109 '[control_107]' is not marked as EOG\n", "load: control token: 547 '[control_545]' is not marked as EOG\n", "load: control token: 350 '[control_348]' is not marked as EOG\n", "load: control token: 132 '[control_130]' is not marked as EOG\n", "load: control token: 114 '[control_112]' is not marked as EOG\n", "load: control token: 248 '[control_246]' is not marked as EOG\n", "load: control token: 328 '[control_326]' is not marked as EOG\n", "load: control token: 104 '[control_102]' is not marked as EOG\n", "load: control token: 449 '[control_447]' is not marked as EOG\n", "load: control token: 711 '[control_709]' is not marked as EOG\n", "load: control token: 715 '[control_713]' is not marked as EOG\n", "load: control token: 358 '[control_356]' is not marked as EOG\n", "load: control token: 746 '[control_744]' is not marked as EOG\n", "load: control token: 385 '[control_383]' is not marked as EOG\n", "load: control token: 720 '[control_718]' is not marked as EOG\n", "load: control token: 460 '[control_458]' is not marked as EOG\n", "load: control token: 632 '[control_630]' is not marked as EOG\n", "load: control token: 296 '[control_294]' is not marked as EOG\n", "load: control token: 240 '[control_238]' is not marked as EOG\n", "load: control token: 361 '[control_359]' is not marked as EOG\n", "load: control token: 504 '[control_502]' is not marked as EOG\n", "load: control token: 149 '[control_147]' is not marked as EOG\n", "load: control token: 1 '' is not marked as EOG\n", "load: control token: 219 '[control_217]' is not marked as EOG\n", "load: control token: 106 '[control_104]' is not marked as EOG\n", "load: control token: 669 '[control_667]' is not marked as EOG\n", "load: control token: 411 '[control_409]' is not marked as EOG\n", "load: control token: 546 '[control_544]' is not marked as EOG\n", "load: control token: 379 '[control_377]' is not marked as EOG\n", "load: control token: 209 '[control_207]' is not marked as EOG\n", "load: control token: 110 '[control_108]' is not marked as EOG\n", "load: control token: 681 '[control_679]' is not marked as EOG\n", "load: control token: 153 '[control_151]' is not marked as EOG\n", "load: control token: 457 '[control_455]' is not marked as EOG\n", "load: control token: 91 '[control_89]' is not marked as EOG\n", "load: control token: 224 '[control_222]' is not marked as EOG\n", "load: control token: 653 '[control_651]' is not marked as EOG\n", "load: control token: 694 '[control_692]' is not marked as EOG\n", "load: control token: 316 '[control_314]' is not marked as EOG\n", "load: printing all EOG tokens:\n", "load: - 2 ('')\n", "load: special tokens cache size = 771\n", "load: token to piece cache size = 0.1731 MB\n", "print_info: arch = llama\n", "print_info: vocab_only = 0\n", "print_info: no_alloc = 0\n", "print_info: n_ctx_train = 32768\n", "print_info: n_embd = 4096\n", "print_info: n_embd_inp = 4096\n", "print_info: n_layer = 32\n", "print_info: n_head = 32\n", "print_info: n_head_kv = 8\n", "print_info: n_rot = 128\n", "print_info: n_swa = 0\n", "print_info: is_swa_any = 0\n", "print_info: n_embd_head_k = 128\n", "print_info: n_embd_head_v = 128\n", "print_info: n_gqa = 4\n", "print_info: n_embd_k_gqa = 1024\n", "print_info: n_embd_v_gqa = 1024\n", "print_info: f_norm_eps = 0.0e+00\n", "print_info: f_norm_rms_eps = 1.0e-05\n", "print_info: f_clamp_kqv = 0.0e+00\n", "print_info: f_max_alibi_bias = 0.0e+00\n", "print_info: f_logit_scale = 0.0e+00\n", "print_info: f_attn_scale = 0.0e+00\n", "print_info: f_attn_value_scale = 0.0000\n", "print_info: n_ff = 14336\n", "print_info: n_expert = 0\n", "print_info: n_expert_used = 0\n", "print_info: n_expert_groups = 0\n", "print_info: n_group_used = 0\n", "print_info: causal attn = 1\n", "print_info: pooling type = -1\n", "print_info: rope type = 0\n", "print_info: rope scaling = linear\n", "print_info: freq_base_train = 1000000.0\n", "print_info: freq_scale_train = 1\n", "print_info: n_ctx_orig_yarn = 32768\n", "print_info: rope_yarn_log_mul = 0.0000\n", "print_info: rope_finetuned = unknown\n", "print_info: model type = 7B\n", "print_info: model params = 7.25 B\n", "print_info: general.name = Mistral-7B-Instruct-v0.3\n", "print_info: vocab type = SPM\n", "print_info: n_vocab = 32768\n", "print_info: n_merges = 0\n", "print_info: BOS token = 1 ''\n", "print_info: EOS token = 2 ''\n", "print_info: UNK token = 0 ''\n", "print_info: LF token = 781 '<0x0A>'\n", "print_info: EOG token = 2 ''\n", "print_info: max token length = 48\n", "load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)\n", "load_tensors: layer 0 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 1 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 2 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 3 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 4 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 5 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 6 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 7 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 8 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 9 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 10 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 11 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 12 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 13 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 14 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 15 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 16 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 17 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 18 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 19 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 20 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 21 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 22 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 23 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 24 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 25 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 26 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 27 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 28 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 29 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 30 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 31 assigned to device CUDA0, is_swa = 0\n", "load_tensors: layer 32 assigned to device CUDA0, is_swa = 0\n", "create_tensor: loading tensor token_embd.weight\n", "create_tensor: loading tensor output_norm.weight\n", "create_tensor: loading tensor output.weight\n", "create_tensor: loading tensor blk.0.attn_norm.weight\n", "create_tensor: loading tensor blk.0.attn_q.weight\n", "create_tensor: loading tensor blk.0.attn_k.weight\n", "create_tensor: loading tensor blk.0.attn_v.weight\n", "create_tensor: loading tensor blk.0.attn_output.weight\n", "create_tensor: loading tensor blk.0.ffn_norm.weight\n", "create_tensor: loading tensor blk.0.ffn_gate.weight\n", "create_tensor: loading tensor blk.0.ffn_down.weight\n", "create_tensor: loading tensor blk.0.ffn_up.weight\n", "create_tensor: loading tensor blk.1.attn_norm.weight\n", "create_tensor: loading tensor blk.1.attn_q.weight\n", "create_tensor: loading tensor blk.1.attn_k.weight\n", "create_tensor: loading tensor blk.1.attn_v.weight\n", "create_tensor: loading tensor blk.1.attn_output.weight\n", "create_tensor: loading tensor blk.1.ffn_norm.weight\n", "create_tensor: loading tensor blk.1.ffn_gate.weight\n", "create_tensor: loading tensor blk.1.ffn_down.weight\n", "create_tensor: loading tensor blk.1.ffn_up.weight\n", "create_tensor: loading tensor blk.2.attn_norm.weight\n", "create_tensor: loading tensor blk.2.attn_q.weight\n", "create_tensor: loading tensor blk.2.attn_k.weight\n", "create_tensor: loading tensor blk.2.attn_v.weight\n", "create_tensor: loading tensor blk.2.attn_output.weight\n", "create_tensor: loading tensor blk.2.ffn_norm.weight\n", "create_tensor: loading tensor blk.2.ffn_gate.weight\n", "create_tensor: loading tensor blk.2.ffn_down.weight\n", "create_tensor: loading tensor blk.2.ffn_up.weight\n", "create_tensor: loading tensor blk.3.attn_norm.weight\n", "create_tensor: loading tensor blk.3.attn_q.weight\n", "create_tensor: loading tensor blk.3.attn_k.weight\n", "create_tensor: loading tensor blk.3.attn_v.weight\n", "create_tensor: loading tensor blk.3.attn_output.weight\n", "create_tensor: loading tensor blk.3.ffn_norm.weight\n", "create_tensor: loading tensor blk.3.ffn_gate.weight\n", "create_tensor: loading tensor blk.3.ffn_down.weight\n", "create_tensor: loading tensor blk.3.ffn_up.weight\n", "create_tensor: loading tensor blk.4.attn_norm.weight\n", "create_tensor: loading tensor blk.4.attn_q.weight\n", "create_tensor: loading tensor blk.4.attn_k.weight\n", "create_tensor: loading tensor blk.4.attn_v.weight\n", "create_tensor: loading tensor blk.4.attn_output.weight\n", "create_tensor: loading tensor blk.4.ffn_norm.weight\n", "create_tensor: loading tensor blk.4.ffn_gate.weight\n", "create_tensor: loading tensor blk.4.ffn_down.weight\n", "create_tensor: loading tensor blk.4.ffn_up.weight\n", "create_tensor: loading tensor blk.5.attn_norm.weight\n", "create_tensor: loading tensor blk.5.attn_q.weight\n", "create_tensor: loading tensor blk.5.attn_k.weight\n", "create_tensor: loading tensor blk.5.attn_v.weight\n", "create_tensor: loading tensor blk.5.attn_output.weight\n", "create_tensor: loading tensor blk.5.ffn_norm.weight\n", "create_tensor: loading tensor blk.5.ffn_gate.weight\n", "create_tensor: loading tensor blk.5.ffn_down.weight\n", "create_tensor: loading tensor blk.5.ffn_up.weight\n", "create_tensor: loading tensor blk.6.attn_norm.weight\n", "create_tensor: loading tensor blk.6.attn_q.weight\n", "create_tensor: loading tensor blk.6.attn_k.weight\n", "create_tensor: loading tensor blk.6.attn_v.weight\n", "create_tensor: loading tensor blk.6.attn_output.weight\n", "create_tensor: loading tensor blk.6.ffn_norm.weight\n", "create_tensor: loading tensor blk.6.ffn_gate.weight\n", "create_tensor: loading tensor blk.6.ffn_down.weight\n", "create_tensor: loading tensor blk.6.ffn_up.weight\n", "create_tensor: loading tensor blk.7.attn_norm.weight\n", "create_tensor: loading tensor blk.7.attn_q.weight\n", "create_tensor: loading tensor blk.7.attn_k.weight\n", "create_tensor: loading tensor blk.7.attn_v.weight\n", "create_tensor: loading tensor blk.7.attn_output.weight\n", "create_tensor: loading tensor blk.7.ffn_norm.weight\n", "create_tensor: loading tensor blk.7.ffn_gate.weight\n", "create_tensor: loading tensor blk.7.ffn_down.weight\n", "create_tensor: loading tensor blk.7.ffn_up.weight\n", "create_tensor: loading tensor blk.8.attn_norm.weight\n", "create_tensor: loading tensor blk.8.attn_q.weight\n", "create_tensor: loading tensor blk.8.attn_k.weight\n", "create_tensor: loading tensor blk.8.attn_v.weight\n", "create_tensor: loading tensor blk.8.attn_output.weight\n", "create_tensor: loading tensor blk.8.ffn_norm.weight\n", "create_tensor: loading tensor blk.8.ffn_gate.weight\n", "create_tensor: loading tensor blk.8.ffn_down.weight\n", "create_tensor: loading tensor blk.8.ffn_up.weight\n", "create_tensor: loading tensor blk.9.attn_norm.weight\n", "create_tensor: loading tensor blk.9.attn_q.weight\n", "create_tensor: loading tensor blk.9.attn_k.weight\n", "create_tensor: loading tensor blk.9.attn_v.weight\n", "create_tensor: loading tensor blk.9.attn_output.weight\n", "create_tensor: loading tensor blk.9.ffn_norm.weight\n", "create_tensor: loading tensor blk.9.ffn_gate.weight\n", "create_tensor: loading tensor blk.9.ffn_down.weight\n", "create_tensor: loading tensor blk.9.ffn_up.weight\n", "create_tensor: loading tensor blk.10.attn_norm.weight\n", "create_tensor: loading tensor blk.10.attn_q.weight\n", "create_tensor: loading tensor blk.10.attn_k.weight\n", "create_tensor: loading tensor blk.10.attn_v.weight\n", "create_tensor: loading tensor blk.10.attn_output.weight\n", "create_tensor: loading tensor blk.10.ffn_norm.weight\n", "create_tensor: loading tensor blk.10.ffn_gate.weight\n", "create_tensor: loading tensor blk.10.ffn_down.weight\n", "create_tensor: loading tensor blk.10.ffn_up.weight\n", "create_tensor: loading tensor blk.11.attn_norm.weight\n", "create_tensor: loading tensor blk.11.attn_q.weight\n", "create_tensor: loading tensor blk.11.attn_k.weight\n", "create_tensor: loading tensor blk.11.attn_v.weight\n", "create_tensor: loading tensor blk.11.attn_output.weight\n", "create_tensor: loading tensor blk.11.ffn_norm.weight\n", "create_tensor: loading tensor blk.11.ffn_gate.weight\n", "create_tensor: loading tensor blk.11.ffn_down.weight\n", "create_tensor: loading tensor blk.11.ffn_up.weight\n", "create_tensor: loading tensor blk.12.attn_norm.weight\n", "create_tensor: loading tensor blk.12.attn_q.weight\n", "create_tensor: loading tensor blk.12.attn_k.weight\n", "create_tensor: loading tensor blk.12.attn_v.weight\n", "create_tensor: loading tensor blk.12.attn_output.weight\n", "create_tensor: loading tensor blk.12.ffn_norm.weight\n", "create_tensor: loading tensor blk.12.ffn_gate.weight\n", "create_tensor: loading tensor blk.12.ffn_down.weight\n", "create_tensor: loading tensor blk.12.ffn_up.weight\n", "create_tensor: loading tensor blk.13.attn_norm.weight\n", "create_tensor: loading tensor blk.13.attn_q.weight\n", "create_tensor: loading tensor blk.13.attn_k.weight\n", "create_tensor: loading tensor blk.13.attn_v.weight\n", "create_tensor: loading tensor blk.13.attn_output.weight\n", "create_tensor: loading tensor blk.13.ffn_norm.weight\n", "create_tensor: loading tensor blk.13.ffn_gate.weight\n", "create_tensor: loading tensor blk.13.ffn_down.weight\n", "create_tensor: loading tensor blk.13.ffn_up.weight\n", "create_tensor: loading tensor blk.14.attn_norm.weight\n", "create_tensor: loading tensor blk.14.attn_q.weight\n", "create_tensor: loading tensor blk.14.attn_k.weight\n", "create_tensor: loading tensor blk.14.attn_v.weight\n", "create_tensor: loading tensor blk.14.attn_output.weight\n", "create_tensor: loading tensor blk.14.ffn_norm.weight\n", "create_tensor: loading tensor blk.14.ffn_gate.weight\n", "create_tensor: loading tensor blk.14.ffn_down.weight\n", "create_tensor: loading tensor blk.14.ffn_up.weight\n", "create_tensor: loading tensor blk.15.attn_norm.weight\n", "create_tensor: loading tensor blk.15.attn_q.weight\n", "create_tensor: loading tensor blk.15.attn_k.weight\n", "create_tensor: loading tensor blk.15.attn_v.weight\n", "create_tensor: loading tensor blk.15.attn_output.weight\n", "create_tensor: loading tensor blk.15.ffn_norm.weight\n", "create_tensor: loading tensor blk.15.ffn_gate.weight\n", "create_tensor: loading tensor blk.15.ffn_down.weight\n", "create_tensor: loading tensor blk.15.ffn_up.weight\n", "create_tensor: loading tensor blk.16.attn_norm.weight\n", "create_tensor: loading tensor blk.16.attn_q.weight\n", "create_tensor: loading tensor blk.16.attn_k.weight\n", "create_tensor: loading tensor blk.16.attn_v.weight\n", "create_tensor: loading tensor blk.16.attn_output.weight\n", "create_tensor: loading tensor blk.16.ffn_norm.weight\n", "create_tensor: loading tensor blk.16.ffn_gate.weight\n", "create_tensor: loading tensor blk.16.ffn_down.weight\n", "create_tensor: loading tensor blk.16.ffn_up.weight\n", "create_tensor: loading tensor blk.17.attn_norm.weight\n", "create_tensor: loading tensor blk.17.attn_q.weight\n", "create_tensor: loading tensor blk.17.attn_k.weight\n", "create_tensor: loading tensor blk.17.attn_v.weight\n", "create_tensor: loading tensor blk.17.attn_output.weight\n", "create_tensor: loading tensor blk.17.ffn_norm.weight\n", "create_tensor: loading tensor blk.17.ffn_gate.weight\n", "create_tensor: loading tensor blk.17.ffn_down.weight\n", "create_tensor: loading tensor blk.17.ffn_up.weight\n", "create_tensor: loading tensor blk.18.attn_norm.weight\n", "create_tensor: loading tensor blk.18.attn_q.weight\n", "create_tensor: loading tensor blk.18.attn_k.weight\n", "create_tensor: loading tensor blk.18.attn_v.weight\n", "create_tensor: loading tensor blk.18.attn_output.weight\n", "create_tensor: loading tensor blk.18.ffn_norm.weight\n", "create_tensor: loading tensor blk.18.ffn_gate.weight\n", "create_tensor: loading tensor blk.18.ffn_down.weight\n", "create_tensor: loading tensor blk.18.ffn_up.weight\n", "create_tensor: loading tensor blk.19.attn_norm.weight\n", "create_tensor: loading tensor blk.19.attn_q.weight\n", "create_tensor: loading tensor blk.19.attn_k.weight\n", "create_tensor: loading tensor blk.19.attn_v.weight\n", "create_tensor: loading tensor blk.19.attn_output.weight\n", "create_tensor: loading tensor blk.19.ffn_norm.weight\n", "create_tensor: loading tensor blk.19.ffn_gate.weight\n", "create_tensor: loading tensor blk.19.ffn_down.weight\n", "create_tensor: loading tensor blk.19.ffn_up.weight\n", "create_tensor: loading tensor blk.20.attn_norm.weight\n", "create_tensor: loading tensor blk.20.attn_q.weight\n", "create_tensor: loading tensor blk.20.attn_k.weight\n", "create_tensor: loading tensor blk.20.attn_v.weight\n", "create_tensor: loading tensor blk.20.attn_output.weight\n", "create_tensor: loading tensor blk.20.ffn_norm.weight\n", "create_tensor: loading tensor blk.20.ffn_gate.weight\n", "create_tensor: loading tensor blk.20.ffn_down.weight\n", "create_tensor: loading tensor blk.20.ffn_up.weight\n", "create_tensor: loading tensor blk.21.attn_norm.weight\n", "create_tensor: loading tensor blk.21.attn_q.weight\n", "create_tensor: loading tensor blk.21.attn_k.weight\n", "create_tensor: loading tensor blk.21.attn_v.weight\n", "create_tensor: loading tensor blk.21.attn_output.weight\n", "create_tensor: loading tensor blk.21.ffn_norm.weight\n", "create_tensor: loading tensor blk.21.ffn_gate.weight\n", "create_tensor: loading tensor blk.21.ffn_down.weight\n", "create_tensor: loading tensor blk.21.ffn_up.weight\n", "create_tensor: loading tensor blk.22.attn_norm.weight\n", "create_tensor: loading tensor blk.22.attn_q.weight\n", "create_tensor: loading tensor blk.22.attn_k.weight\n", "create_tensor: loading tensor blk.22.attn_v.weight\n", "create_tensor: loading tensor blk.22.attn_output.weight\n", "create_tensor: loading tensor blk.22.ffn_norm.weight\n", "create_tensor: loading tensor blk.22.ffn_gate.weight\n", "create_tensor: loading tensor blk.22.ffn_down.weight\n", "create_tensor: loading tensor blk.22.ffn_up.weight\n", "create_tensor: loading tensor blk.23.attn_norm.weight\n", "create_tensor: loading tensor blk.23.attn_q.weight\n", "create_tensor: loading tensor blk.23.attn_k.weight\n", "create_tensor: loading tensor blk.23.attn_v.weight\n", "create_tensor: loading tensor blk.23.attn_output.weight\n", "create_tensor: loading tensor blk.23.ffn_norm.weight\n", "create_tensor: loading tensor blk.23.ffn_gate.weight\n", "create_tensor: loading tensor blk.23.ffn_down.weight\n", "create_tensor: loading tensor blk.23.ffn_up.weight\n", "create_tensor: loading tensor blk.24.attn_norm.weight\n", "create_tensor: loading tensor blk.24.attn_q.weight\n", "create_tensor: loading tensor blk.24.attn_k.weight\n", "create_tensor: loading tensor blk.24.attn_v.weight\n", "create_tensor: loading tensor blk.24.attn_output.weight\n", "create_tensor: loading tensor blk.24.ffn_norm.weight\n", "create_tensor: loading tensor blk.24.ffn_gate.weight\n", "create_tensor: loading tensor blk.24.ffn_down.weight\n", "create_tensor: loading tensor blk.24.ffn_up.weight\n", "create_tensor: loading tensor blk.25.attn_norm.weight\n", "create_tensor: loading tensor blk.25.attn_q.weight\n", "create_tensor: loading tensor blk.25.attn_k.weight\n", "create_tensor: loading tensor blk.25.attn_v.weight\n", "create_tensor: loading tensor blk.25.attn_output.weight\n", "create_tensor: loading tensor blk.25.ffn_norm.weight\n", "create_tensor: loading tensor blk.25.ffn_gate.weight\n", "create_tensor: loading tensor blk.25.ffn_down.weight\n", "create_tensor: loading tensor blk.25.ffn_up.weight\n", "create_tensor: loading tensor blk.26.attn_norm.weight\n", "create_tensor: loading tensor blk.26.attn_q.weight\n", "create_tensor: loading tensor blk.26.attn_k.weight\n", "create_tensor: loading tensor blk.26.attn_v.weight\n", "create_tensor: loading tensor blk.26.attn_output.weight\n", "create_tensor: loading tensor blk.26.ffn_norm.weight\n", "create_tensor: loading tensor blk.26.ffn_gate.weight\n", "create_tensor: loading tensor blk.26.ffn_down.weight\n", "create_tensor: loading tensor blk.26.ffn_up.weight\n", "create_tensor: loading tensor blk.27.attn_norm.weight\n", "create_tensor: loading tensor blk.27.attn_q.weight\n", "create_tensor: loading tensor blk.27.attn_k.weight\n", "create_tensor: loading tensor blk.27.attn_v.weight\n", "create_tensor: loading tensor blk.27.attn_output.weight\n", "create_tensor: loading tensor blk.27.ffn_norm.weight\n", "create_tensor: loading tensor blk.27.ffn_gate.weight\n", "create_tensor: loading tensor blk.27.ffn_down.weight\n", "create_tensor: loading tensor blk.27.ffn_up.weight\n", "create_tensor: loading tensor blk.28.attn_norm.weight\n", "create_tensor: loading tensor blk.28.attn_q.weight\n", "create_tensor: loading tensor blk.28.attn_k.weight\n", "create_tensor: loading tensor blk.28.attn_v.weight\n", "create_tensor: loading tensor blk.28.attn_output.weight\n", "create_tensor: loading tensor blk.28.ffn_norm.weight\n", "create_tensor: loading tensor blk.28.ffn_gate.weight\n", "create_tensor: loading tensor blk.28.ffn_down.weight\n", "create_tensor: loading tensor blk.28.ffn_up.weight\n", "create_tensor: loading tensor blk.29.attn_norm.weight\n", "create_tensor: loading tensor blk.29.attn_q.weight\n", "create_tensor: loading tensor blk.29.attn_k.weight\n", "create_tensor: loading tensor blk.29.attn_v.weight\n", "create_tensor: loading tensor blk.29.attn_output.weight\n", "create_tensor: loading tensor blk.29.ffn_norm.weight\n", "create_tensor: loading tensor blk.29.ffn_gate.weight\n", "create_tensor: loading tensor blk.29.ffn_down.weight\n", "create_tensor: loading tensor blk.29.ffn_up.weight\n", "create_tensor: loading tensor blk.30.attn_norm.weight\n", "create_tensor: loading tensor blk.30.attn_q.weight\n", "create_tensor: loading tensor blk.30.attn_k.weight\n", "create_tensor: loading tensor blk.30.attn_v.weight\n", "create_tensor: loading tensor blk.30.attn_output.weight\n", "create_tensor: loading tensor blk.30.ffn_norm.weight\n", "create_tensor: loading tensor blk.30.ffn_gate.weight\n", "create_tensor: loading tensor blk.30.ffn_down.weight\n", "create_tensor: loading tensor blk.30.ffn_up.weight\n", "create_tensor: loading tensor blk.31.attn_norm.weight\n", "create_tensor: loading tensor blk.31.attn_q.weight\n", "create_tensor: loading tensor blk.31.attn_k.weight\n", "create_tensor: loading tensor blk.31.attn_v.weight\n", "create_tensor: loading tensor blk.31.attn_output.weight\n", "create_tensor: loading tensor blk.31.ffn_norm.weight\n", "create_tensor: loading tensor blk.31.ffn_gate.weight\n", "create_tensor: loading tensor blk.31.ffn_down.weight\n", "create_tensor: loading tensor blk.31.ffn_up.weight\n", "done_getting_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead\n", "load_tensors: offloading output layer to GPU\n", "load_tensors: offloading 31 repeating layers to GPU\n", "load_tensors: offloaded 33/33 layers to GPU\n", "load_tensors: CPU_Mapped model buffer size = 72.00 MiB\n", "load_tensors: CUDA0 model buffer size = 4097.52 MiB\n", "................................................................................................\n", "llama_context: constructing llama_context\n", "llama_context: n_seq_max = 1\n", "llama_context: n_ctx = 512\n", "llama_context: n_ctx_seq = 512\n", "llama_context: n_batch = 512\n", "llama_context: n_ubatch = 512\n", "llama_context: causal_attn = 1\n", "llama_context: flash_attn = disabled\n", "llama_context: kv_unified = false\n", "llama_context: freq_base = 1000000.0\n", "llama_context: freq_scale = 1\n", "llama_context: n_ctx_seq (512) < n_ctx_train (32768) -- the full capacity of the model will not be utilized\n", "set_abort_callback: call\n", "llama_context: CUDA_Host output buffer size = 0.12 MiB\n", "llama_kv_cache: layer 0: dev = CUDA0\n", "llama_kv_cache: layer 1: dev = CUDA0\n", "llama_kv_cache: layer 2: dev = CUDA0\n", "llama_kv_cache: layer 3: dev = CUDA0\n", "llama_kv_cache: layer 4: dev = CUDA0\n", "llama_kv_cache: layer 5: dev = CUDA0\n", "llama_kv_cache: layer 6: dev = CUDA0\n", "llama_kv_cache: layer 7: dev = CUDA0\n", "llama_kv_cache: layer 8: dev = CUDA0\n", "llama_kv_cache: layer 9: dev = CUDA0\n", "llama_kv_cache: layer 10: dev = CUDA0\n", "llama_kv_cache: layer 11: dev = CUDA0\n", "llama_kv_cache: layer 12: dev = CUDA0\n", "llama_kv_cache: layer 13: dev = CUDA0\n", "llama_kv_cache: layer 14: dev = CUDA0\n", "llama_kv_cache: layer 15: dev = CUDA0\n", "llama_kv_cache: layer 16: dev = CUDA0\n", "llama_kv_cache: layer 17: dev = CUDA0\n", "llama_kv_cache: layer 18: dev = CUDA0\n", "llama_kv_cache: layer 19: dev = CUDA0\n", "llama_kv_cache: layer 20: dev = CUDA0\n", "llama_kv_cache: layer 21: dev = CUDA0\n", "llama_kv_cache: layer 22: dev = CUDA0\n", "llama_kv_cache: layer 23: dev = CUDA0\n", "llama_kv_cache: layer 24: dev = CUDA0\n", "llama_kv_cache: layer 25: dev = CUDA0\n", "llama_kv_cache: layer 26: dev = CUDA0\n", "llama_kv_cache: layer 27: dev = CUDA0\n", "llama_kv_cache: layer 28: dev = CUDA0\n", "llama_kv_cache: layer 29: dev = CUDA0\n", "llama_kv_cache: layer 30: dev = CUDA0\n", "llama_kv_cache: layer 31: dev = CUDA0\n", "llama_kv_cache: CUDA0 KV buffer size = 64.00 MiB\n", "llama_kv_cache: size = 64.00 MiB ( 512 cells, 32 layers, 1/1 seqs), K (f16): 32.00 MiB, V (f16): 32.00 MiB\n", "llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 128\n", "llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 128\n", "llama_context: enumerating backends\n", "llama_context: backend_ptrs.size() = 2\n", "sched_reserve: reserving ...\n", "sched_reserve: max_nodes = 2328\n", "sched_reserve: reserving full memory module\n", "sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1\n", "sched_reserve: resolving fused Gated Delta Net support:\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 1, n_seqs = 1, n_outputs = 1\n", "sched_reserve: fused Gated Delta Net (autoregressive) enabled\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 16, n_seqs = 1, n_outputs = 16\n", "sched_reserve: fused Gated Delta Net (chunked) enabled\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 512, n_seqs = 1, n_outputs = 512\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 1, n_seqs = 1, n_outputs = 1\n", "graph_reserve: reserving a graph for ubatch with n_tokens = 512, n_seqs = 1, n_outputs = 512\n", "sched_reserve: CUDA0 compute buffer size = 113.01 MiB\n", "sched_reserve: CUDA_Host compute buffer size = 21.01 MiB\n", "sched_reserve: graph nodes = 1158\n", "sched_reserve: graph splits = 2\n", "sched_reserve: reserve took 32.45 ms, sched copies = 1\n", "CUDA : ARCHS = 890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | \n", "Model metadata: {'quantize.imatrix.entries_count': '224', 'quantize.imatrix.dataset': '/training_data/calibration_data.txt', 'quantize.imatrix.chunks_count': '228', 'quantize.imatrix.file': '/models/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3.imatrix', 'tokenizer.chat_template': \"{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.architecture': 'llama', 'llama.rope.freq_base': '1000000.000000', 'tokenizer.ggml.pre': 'default', 'llama.context_length': '32768', 'general.name': 'Mistral-7B-Instruct-v0.3', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.file_type': '15', 'llama.vocab_size': '32768', 'llama.rope.dimension_count': '128'}\n", "Available chat formats from metadata: chat_template.default\n", "Guessed chat format: mistral-instruct\n", "~llama_context: CUDA0 compute buffer size is 194.9375 MiB, matches expectation of 194.9375 MiB\n", "~llama_context: CUDA_Host compute buffer size is 21.0098 MiB, matches expectation of 21.0098 MiB\n" ] } ], "source": [ "# 指定仓库的名称和文件名\n", "repo_id = \"bartowski/Mistral-7B-Instruct-v0.3-GGUF\"\n", "filename = \"Mistral-7B-Instruct-v0.3-Q4_K_M.gguf\"\n", "#filename = \"*Q4_K_M.gguf\" # 使用通配符也是可以的\n", "\n", "\n", "# 下载并加载模型\n", "llm = Llama.from_pretrained(\n", " repo_id=repo_id,\n", " filename=filename,\n", " n_gpu_layers=-1, # 注释后在 CPU 推理\n", " #verbose=False, # 取消注释减少模型方面的信息打印\n", ")" ] }, { "cell_type": "markdown", "id": "4d7f7cf7", "metadata": {}, "source": [ "#### 推理测试\n", "\n", "简单使用以下命令进行推理测试：" ] }, { "cell_type": "code", "execution_count": 6, "id": "b4e190a3", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 741 reused\n", "ggml_backend_cuda_graph_compute: CUDA graph warmup complete\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "CUDA Graph id 741 reused\n", "llama_perf_context_print: load time = 35.66 ms\n", "llama_perf_context_print: prompt eval time = 34.94 ms / 5 tokens ( 6.99 ms per token, 143.09 tokens per second)\n", "llama_perf_context_print: eval time = 329.46 ms / 49 runs ( 6.72 ms per token, 148.73 tokens per second)\n", "llama_perf_context_print: total time = 376.32 ms / 54 tokens\n", "llama_perf_context_print: graphs reused = 48\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " I’m back with a new post!\n", "\n", "For this post, I will be reviewing the 2019 drama, “When the Camellia Blooms.”\n", "\n", "I found this drama on Netflix, and I thought it\n" ] } ], "source": [ "# 输入文本\n", "input_text = \"Hello, World!\"\n", "\n", "# 生成输出\n", "output = llm(input_text, max_tokens=50)\n", "\n", "# 打印生成的文本\n", "print(output['choices'][0]['text'])" ] }, { "cell_type": "markdown", "id": "314951dd-5b1e-4dc1-ac87-d920a4ab9e48", "metadata": {}, "source": [ "### 流式输出\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "5355e797-8025-4019-890f-4d5b10bf49d3", "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Llama.generate: 1 prefix-match hit, remaining 19 prompt tokens to eval\n", "ggml_backend_cuda_graph_compute: CUDA graph warmup reset\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "assistant: 人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n", "ggml_backend_cuda_graph_compute: CUDA graph warmup complete\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "工" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "智" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "能" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "未" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "来" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "发" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "展" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "方" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "向" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "包" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "括" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "但" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "不" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "限" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "于" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "以" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "下" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "几" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "个" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "方" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "面" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "1" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "自" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "然" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "语" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "言" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "处" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "理" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "N" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "LP" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "将" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "更" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "好" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "地" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "理" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "解" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "生" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "成" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "类" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "语" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "言" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "从" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "而" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "提" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "高" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "机" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "交" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "互" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "效" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "率" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "准" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "确" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "性" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "机" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "器" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "ML" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "将" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "更" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "好" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "地" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "分" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "析" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "大" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "量" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "数据" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "从" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "而" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "提" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "高" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "预" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "测" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "决" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "策" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "能" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "力" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "3" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "工" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "智" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "能" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "算" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "法" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "优" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "化" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "将" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "更" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "好" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "地" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "优" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "化" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "算" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "法" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "从" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "而" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "提" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "高" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "计" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "算" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "效" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "率" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "准" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "确" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "性" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "4" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "深" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "度" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "将" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "更" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "好" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "地" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "使" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "用" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "深" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "度" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "技" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "术" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "如" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "卷" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "积" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "神" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "经" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "网" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "络" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "C" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "NN" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "递" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "归" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "神" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "经" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "网" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "络" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "R" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "NN" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "从" ] }, { "name": "stderr", "output_type": "stream", "text": [ "llama_perf_context_print: load time = 35.66 ms\n", "llama_perf_context_print: prompt eval time = 190.34 ms / 19 tokens ( 10.02 ms per token, 99.82 tokens per second)\n", "llama_perf_context_print: eval time = 1288.71 ms / 199 runs ( 6.48 ms per token, 154.42 tokens per second)\n", "llama_perf_context_print: total time = 1664.62 ms / 218 tokens\n", "llama_perf_context_print: graphs reused = 198\n" ] } ], "source": [ "prompt = \"人工智能的未来发展方向是什么？\"\n", "\n", "output = llm.create_chat_completion(\n", " messages=[{\n", " \"role\": \"user\",\n", " \"content\": prompt\n", " }],\n", " max_tokens=200,\n", " stream=True\n", ")\n", "\n", "for chunk in output:\n", " delta = chunk['choices'][0]['delta']\n", " if 'role' in delta:\n", " print(delta['role'], end=': ', flush=True)\n", " elif 'content' in delta:\n", " print(delta['content'], end='', flush=True)" ] }, { "cell_type": "markdown", "id": "ec7f36bc-82f8-47da-84b4-785ef804ad61", "metadata": {}, "source": [ "查看 output 的构造：" ] }, { "cell_type": "code", "execution_count": 8, "id": "81f847f8-73dd-4ba8-b524-7b4a85e96665", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Llama.generate: 19 prefix-match hit, remaining 1 prompt tokens to eval\n", "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n" ] }, { "data": { "text/plain": [ "[{'id': 'chatcmpl-479fe970-e48e-4a82-9be6-0433ccdf5d22',\n", " 'model': '/root/.cache/huggingface/hub/models--bartowski--Mistral-7B-Instruct-v0.3-GGUF/snapshots/61fd4167fff3ab01ee1cfe0da183fa27a944db48/./Mistral-7B-Instruct-v0.3-Q4_K_M.gguf',\n", " 'created': 1779036911,\n", " 'object': 'chat.completion.chunk',\n", " 'choices': [{'index': 0,\n", " 'delta': {'role': 'assistant'},\n", " 'logprobs': None,\n", " 'finish_reason': None}]},\n", " {'id': 'chatcmpl-479fe970-e48e-4a82-9be6-0433ccdf5d22',\n", " 'model': '/root/.cache/huggingface/hub/models--bartowski--Mistral-7B-Instruct-v0.3-GGUF/snapshots/61fd4167fff3ab01ee1cfe0da183fa27a944db48/./Mistral-7B-Instruct-v0.3-Q4_K_M.gguf',\n", " 'created': 1779036911,\n", " 'object': 'chat.completion.chunk',\n", " 'choices': [{'index': 0,\n", " 'delta': {'content': ' '},\n", " 'logprobs': None,\n", " 'finish_reason': None}]},\n", " {'id': 'chatcmpl-479fe970-e48e-4a82-9be6-0433ccdf5d22',\n", " 'model': '/root/.cache/huggingface/hub/models--bartowski--Mistral-7B-Instruct-v0.3-GGUF/snapshots/61fd4167fff3ab01ee1cfe0da183fa27a944db48/./Mistral-7B-Instruct-v0.3-Q4_K_M.gguf',\n", " 'created': 1779036911,\n", " 'object': 'chat.completion.chunk',\n", " 'choices': [{'index': 0,\n", " 'delta': {'content': '人'},\n", " 'logprobs': None,\n", " 'finish_reason': None}]}]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from itertools import islice\n", "\n", "prompt = \"人工智能的未来发展方向是什么？\"\n", "\n", "output = llm.create_chat_completion(\n", " messages=[{\n", " \"role\": \"user\",\n", " \"content\": prompt\n", " }],\n", " max_tokens=200,\n", " stream=True\n", ")\n", "\n", "print(type(output))\n", "\n", "# 将生成器转换为列表\n", "output_list = list(islice(output, 3))\n", "\n", "# 获取前 3 个条目\n", "output_list[:3]" ] }, { "cell_type": "markdown", "id": "6d1806f8-9509-4dbf-9662-c1bf0934fef1", "metadata": {}, "source": [ "将刚刚对于流式输出的处理抽象为函数便于后续调用：" ] }, { "cell_type": "code", "execution_count": 9, "id": "b9145b07-1cd3-43df-8eba-21794ac91fbb", "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Llama.generate: 19 prefix-match hit, remaining 1 prompt tokens to eval\n", "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "assistant: " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "工" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "智" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "能" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "未" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "来" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "发" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "展" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "方" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "向" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "包" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "括" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "但" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "不" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "限" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "于" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "以" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "下" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "几" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "个" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "方" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "面" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "1" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "强" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "化" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "（" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Re" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "in" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "force" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "ment" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Learning" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "RL" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "）" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "RL" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "是" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "一" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "种" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "机" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "器" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "方" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "法" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "它" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "允" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "许" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "机" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "器" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "通" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "过" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "试" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "错" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "来" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "学" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "习" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "以" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "最" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "小" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "化" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "某" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "种" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "成" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "本" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "函" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "数" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "或" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "最" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "大" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "化" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "某" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "种" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "奖" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "励" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "函" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "数" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "RL" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "已" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "被" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "应" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "用" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "于" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "游" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "戏" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "、" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "自" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "动" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "驾" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "驶" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "、" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "健" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "康" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "保" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "健" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "等" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "领" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "域" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " " ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "工" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "智" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "能" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "安" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "全" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "隐" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "私" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "：" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "发" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "展" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "也" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "带" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "来" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "了" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "新" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n", "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "挑" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "战" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "包" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "括" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "数据" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "隐" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "私" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "保" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "护" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "系" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "统" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "的" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "安" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "全" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "性" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "研" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "究" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "人" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "员" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "正" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "在" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "开" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "发" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "各" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "种" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "技" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "术" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "来" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "保" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "护" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "数据" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "和" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "系" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "统" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "，" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "同" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "时" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "又" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "能" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "够" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "有" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "效" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "地" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "使" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "用" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "AI" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "。" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "3" ] }, { "name": "stderr", "output_type": "stream", "text": [ "CUDA Graph id 747 reused\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "." ] }, { "name": "stderr", "output_type": "stream", "text": [ "llama_perf_context_print: load time = 35.66 ms\n", "llama_perf_context_print: prompt eval time = 0.00 ms / 1 tokens ( 0.00 ms per token, inf tokens per second)\n", "llama_perf_context_print: eval time = 1286.86 ms / 200 runs ( 6.43 ms per token, 155.42 tokens per second)\n", "llama_perf_context_print: total time = 1462.38 ms / 201 tokens\n", "llama_perf_context_print: graphs reused = 200\n" ] } ], "source": [ "def handle_stream_output(output):\n", " \"\"\"\n", " 处理流式输出，将生成的内容逐步打印出来。\n", " \n", " 参数:\n", " output: 生成器对象，来自 create_chat_completion 的流式输出\n", " \"\"\"\n", " for chunk in output:\n", " delta = chunk['choices'][0]['delta']\n", " if 'role' in delta:\n", " print(f\"{delta['role']}: \", end='', flush=True)\n", " elif 'content' in delta:\n", " print(delta['content'], end='', flush=True)\n", " \n", "# 使用示例\n", "prompt = \"人工智能的未来发展方向是什么？\"\n", "\n", "output = llm.create_chat_completion(\n", " messages=[{\n", " \"role\": \"user\",\n", " \"content\": prompt\n", " }],\n", " max_tokens=200,\n", " stream=True\n", ")\n", "\n", "handle_stream_output(output)" ] }, { "cell_type": "markdown", "id": "554686f8-2b6f-422a-8701-e8af93f1c2b5", "metadata": {}, "source": [ "### 多轮对话\n", "\n", "让我们自定义一个交互的对话类（需要注意到 handle_stream_output() 有所修改）。\n" ] }, { "cell_type": "code", "execution_count": 10, "id": "e5c8d8f6-2656-43af-a53e-b49cca1ec166", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "llama_context: n_ctx_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ "User: 如果你是大模型面试官，你会怎么出面试题\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "assistant: 如果我是大模型面试官，我会出以下几类面试题来评估候选人的技能和能力：\n", "\n", "1. 基础知识：\n", " - 问题理解：给出一个问题，要求候选人根据问题描述和提供的信息，提出最佳解决方案。\n", " - 算法和数据结构：要求候选人解释常见算法和数据结构，并讨论其优缺点和适用场景。\n", " - 数学和统计学：要求候选人解释常见数学和统计学概念，并讨论其在机器学习中的应用。\n", "\n", "2. 机器学习和深度学习：\n", " - 机器学习算法：要求候选人讨论常见机器学习算法，并讨论其优缺点和适用场景。\n", " - 深度学习框架：要求候选人讨论常见深度学习框架，并讨论其优缺点和使用场景。\n", " - 模型训练和优化：要求候选人讨论模型训练过程中的常见问题和解决方案，并讨论模型优化的方法和技术。\n", "\n", "3. 实践能力：\n", " - 项目经验：要求候选人讨论过去的项目经验，并讨论自己在项目中所做的贡献。\n", " - 代码编写：要求候选人编写代码来解决给定的问题，并讨论代码的设计和实现过程。\n", " - 数据分析：要求候选人分析给定的数据集，并提出数据的洞察和建议。\n", "\n", "4. 沟通能力：\n", " - 口头表达：要求候选人给出解决问题的方案，并讨论其优缺点和实际应用。\n", " - 写作能力：要求候选人写一篇文章来解释一个机器学习概念或技术，并讨论其在实际应用中的作用。\n", "\n", "5. 问题解决能力：\n", " - 问题分析：要求候选人分析给定的问题，并提出可能的解决方案。\n", " - 问题调试：要求候选人分析和解决给定的问题，并讨论解决过程中遇到的障碍和解决方案。\n", "\n", "6. 团队合作能力：\n", " - 团队协作：要求候选人讨论过去的团队协作经验，并讨论自己在团队中所做的贡献。\n", " - 领导力：要求候选人讨论自己的领导力经验，并讨论自己如何 Motivate 和管理团队成员。\n", "\n", "7. 自我学习能力：\n", " - 自我学习：要求候选人讨论自己的学习方法和习惯，并讨论自己如何在工作中不断学习和提高自己的能力。\n", " - 自我反思：要求候选人讨论自己的成长和失败经验，并讨论自己如何从失败中学习和改进。\n", "\n", "8. 职业目标：\n", " - 职业目标：要求候选人讨论自己的职业目标，并讨论自己如何实现这些目标。\n", " - 职业发展：要求候选人讨论自己的职业发展计划，并讨论自己如何在职业生涯中不断发展和成长。\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ "User: 对于第二个场景能否进一步深入\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "assistant: 当然可以！在第二个场景中，我会进一步深入以下几个方面：\n", "\n", "1. 深度学习框架：\n", " - 深度学习框架的选择：要求候选人讨论选择深度学习框架时的因素，并讨论常见的深度学习框架，如 TensorFlow、PyTorch、MXNet 等，以及它们的优缺点和适用场景。\n", " - 深度学习框架的使用：要求候选人讨论如何使用深度学习框架，包括数据预处理、模型定义、训练和评估等步骤。\n", " - 深度学习框架的优化：要求候选人讨论如何优化深度学习模型的训练过程，包括数据增广、学习率调整、正则化等方法。\n", "\n", "2. 模型训练和优化：\n", " - 模型训练：要求候选人讨论模型训练过程中的常见问题和解决方案，包括数据泛化、过拟合、梯度消失等问题。\n", " - 模型优化：要求候选人讨论模型优化的方法和技术，包括超参数调整、学习率衰减、批量正则化等方法。\n", " - 模型部署：要求候选人讨论如何将训练好的模型部署到生产环境中，包括模型压缩、模型部署工具等方面。\n", "\n", "3. 深度学习应用：\n", " - 自然语言处理：要求候选人讨论自然语言处理领域中的深度学习应用，包括语言模型、文本分类、机器翻译等。\n", " - 计算机视觉：要求候选人讨论计算机视觉领域中的深度学习应用，包括图像分类、目标检测、对象识别等。\n", " - 其他应用：要求候选人讨论深度学习在其他领域中的应用，例如音频处理、生物医学等。\n", "\n", "4. 深度学习研究：\n", " - 最新研究：要求候选人讨论最近几年在深度学习领域发生的重要研究，并讨论这些研究的影响和应用。\n", " - 深度学习的挑战：要求候选人讨论深度学习在实际应用中面临的挑战，例如数据泛化、模型复杂性、计算成本等。\n", " - 深度学习的未来：要求候选人讨论深度学习在未来几年中的发展趋势，并讨论自己对深度学习的兴趣和研究方向。\n" ] }, { "name": "stdin", "output_type": "stream", "text": [ "User: bye\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Goodbye!\n" ] } ], "source": [ "from llama_cpp import Llama\n", "\n", "def handle_stream_output(output):\n", " \"\"\"\n", " 处理流式输出，将生成的内容逐步打印出来，并收集完整的回复。\n", "\n", " 参数:\n", " output: 生成器对象，来自 create_chat_completion 的流式输出\n", "\n", " 返回:\n", " response: 完整的回复文本\n", " \"\"\"\n", " response = \"\"\n", " for chunk in output:\n", " delta = chunk['choices'][0]['delta']\n", " if 'role' in delta:\n", " print(f\"{delta['role']}: \", end='', flush=True)\n", " elif 'content' in delta:\n", " content = delta['content']\n", " print(content, end='', flush=True)\n", " response += content\n", " return response\n", "\n", "class ChatSession:\n", " def __init__(self, llm):\n", " self.llm = llm\n", " self.messages = []\n", "\n", " def add_message(self, role, content):\n", " \"\"\"\n", " 添加一条消息到会话中。\n", "\n", " 参数:\n", " role: 消息角色，通常为 'user' 或 'assistant'\n", " content: 消息内容\n", " \"\"\"\n", " self.messages.append({\"role\": role, \"content\": content})\n", "\n", " def get_response_stream(self, user_input):\n", " \"\"\"\n", " 获取模型对用户输入的响应（流式输出）。\n", "\n", " 参数:\n", " user_input: 用户输入的文本\n", "\n", " 返回:\n", " response: 完整的回复文本\n", " \"\"\"\n", " self.add_message(\"user\", user_input)\n", " \n", " try:\n", " output = self.llm.create_chat_completion(\n", " messages=self.messages,\n", " stream=True # 开启流式输出\n", " )\n", " \n", " response = handle_stream_output(output) # 同时打印和收集回复\n", " \n", " self.add_message(\"assistant\", response.strip())\n", " # print(len(response),len(response.strip()))\n", " return response.strip()\n", " except Exception as e:\n", " print(f\"\\n发生错误: {e}\")\n", " return \"\"\n", "\n", "# 初始化模型（假设使用本地路径）\n", "model_path = \"./Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf\"\n", "llm = Llama(\n", " model_path=model_path,\n", " n_gpu_layers=-1, # 根据需要卸载到 GPU\n", " n_ctx=4096, # 设置上下文窗口大小，尝试注释这行进行多轮对话，看看会发生什么\n", " verbose=False, # 禁用详细日志输出\n", ")\n", "\n", "# 创建会话实例\n", "chat = ChatSession(llm)\n", " \n", "# 开始对话\n", "while True:\n", " prompt = input(\"User: \")\n", " # 退出对话条件（当然，你也可以直接终止代码块）\n", " if prompt.lower() in [\"exit\", \"quit\", \"bye\"]:\n", " print(\"Goodbye!\")\n", " break\n", " chat.get_response_stream(prompt)\n", " print() # 换行以便下一次输入，这是因为之前的 print 都设置了 end=''" ] }, { "cell_type": "code", "execution_count": 11, "id": "62026caf-d39c-4e86-80f9-526d49765c47", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[{'role': 'user', 'content': '如果你是大模型面试官，你会怎么出面试题'}, {'role': 'assistant', 'content': '如果我是大模型面试官，我会出以下几类面试题来评估候选人的技能和能力：\\n\\n1. 基础知识：\\n - 问题理解：给出一个问题，要求候选人根据问题描述和提供的信息，提出最佳解决方案。\\n - 算法和数据结构：要求候选人解释常见算法和数据结构，并讨论其优缺点和适用场景。\\n - 数学和统计学：要求候选人解释常见数学和统计学概念，并讨论其在机器学习中的应用。\\n\\n2. 机器学习和深度学习：\\n - 机器学习算法：要求候选人讨论常见机器学习算法，并讨论其优缺点和适用场景。\\n - 深度学习框架：要求候选人讨论常见深度学习框架，并讨论其优缺点和使用场景。\\n - 模型训练和优化：要求候选人讨论模型训练过程中的常见问题和解决方案，并讨论模型优化的方法和技术。\\n\\n3. 实践能力：\\n - 项目经验：要求候选人讨论过去的项目经验，并讨论自己在项目中所做的贡献。\\n - 代码编写：要求候选人编写代码来解决给定的问题，并讨论代码的设计和实现过程。\\n - 数据分析：要求候选人分析给定的数据集，并提出数据的洞察和建议。\\n\\n4. 沟通能力：\\n - 口头表达：要求候选人给出解决问题的方案，并讨论其优缺点和实际应用。\\n - 写作能力：要求候选人写一篇文章来解释一个机器学习概念或技术，并讨论其在实际应用中的作用。\\n\\n5. 问题解决能力：\\n - 问题分析：要求候选人分析给定的问题，并提出可能的解决方案。\\n - 问题调试：要求候选人分析和解决给定的问题，并讨论解决过程中遇到的障碍和解决方案。\\n\\n6. 团队合作能力：\\n - 团队协作：要求候选人讨论过去的团队协作经验，并讨论自己在团队中所做的贡献。\\n - 领导力：要求候选人讨论自己的领导力经验，并讨论自己如何 Motivate 和管理团队成员。\\n\\n7. 自我学习能力：\\n - 自我学习：要求候选人讨论自己的学习方法和习惯，并讨论自己如何在工作中不断学习和提高自己的能力。\\n - 自我反思：要求候选人讨论自己的成长和失败经验，并讨论自己如何从失败中学习和改进。\\n\\n8. 职业目标：\\n - 职业目标：要求候选人讨论自己的职业目标，并讨论自己如何实现这些目标。\\n - 职业发展：要求候选人讨论自己的职业发展计划，并讨论自己如何在职业生涯中不断发展和成长。'}, {'role': 'user', 'content': '对于第二个场景能否进一步深入'}, {'role': 'assistant', 'content': '当然可以！在第二个场景中，我会进一步深入以下几个方面：\\n\\n1. 深度学习框架：\\n - 深度学习框架的选择：要求候选人讨论选择深度学习框架时的因素，并讨论常见的深度学习框架，如 TensorFlow、PyTorch、MXNet 等，以及它们的优缺点和适用场景。\\n - 深度学习框架的使用：要求候选人讨论如何使用深度学习框架，包括数据预处理、模型定义、训练和评估等步骤。\\n - 深度学习框架的优化：要求候选人讨论如何优化深度学习模型的训练过程，包括数据增广、学习率调整、正则化等方法。\\n\\n2. 模型训练和优化：\\n - 模型训练：要求候选人讨论模型训练过程中的常见问题和解决方案，包括数据泛化、过拟合、梯度消失等问题。\\n - 模型优化：要求候选人讨论模型优化的方法和技术，包括超参数调整、学习率衰减、批量正则化等方法。\\n - 模型部署：要求候选人讨论如何将训练好的模型部署到生产环境中，包括模型压缩、模型部署工具等方面。\\n\\n3. 深度学习应用：\\n - 自然语言处理：要求候选人讨论自然语言处理领域中的深度学习应用，包括语言模型、文本分类、机器翻译等。\\n - 计算机视觉：要求候选人讨论计算机视觉领域中的深度学习应用，包括图像分类、目标检测、对象识别等。\\n - 其他应用：要求候选人讨论深度学习在其他领域中的应用，例如音频处理、生物医学等。\\n\\n4. 深度学习研究：\\n - 最新研究：要求候选人讨论最近几年在深度学习领域发生的重要研究，并讨论这些研究的影响和应用。\\n - 深度学习的挑战：要求候选人讨论深度学习在实际应用中面临的挑战，例如数据泛化、模型复杂性、计算成本等。\\n - 深度学习的未来：要求候选人讨论深度学习在未来几年中的发展趋势，并讨论自己对深度学习的兴趣和研究方向。'}]\n" ] } ], "source": [ "# 使用 bye 退出上面的对话后打印查看\n", "print(chat.messages)" ] }, { "cell_type": "code", "execution_count": null, "id": "7c516583-336c-411b-afc3-0e53f3b20ddd", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "ai", "language": "python", "name": "ai" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }