\n", "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", "
汉化的库: https://github.com/GoatCsu/CN-LLMs-from-scratch.git\n", "\n", "
![](\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\")
\n",
"| \n",
"\n",
"Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka \n", " Code repository: https://github.com/rasbt/LLMs-from-scratch\n", " 汉化的库: https://github.com/GoatCsu/CN-LLMs-from-scratch.git\n", "\n", " | \n",
"\n",
"\n",
" | \n",
"
![](\"https://raw.githubusercontent.com/MLNLP-World/LLMs-from-scratch-CN/main/imgs/ch7/21.png\")
\n",
"\n",
"- **确保 Ollama 运行后**，在 **另一个终端窗口** 执行以下命令，尝试 **8B 参数的 LLaMA 3 模型**（首次执行时，模型将自动下载，占用 **4.7GB 存储空间**）： \n",
"\n",
"\n",
"```bash\n",
"# 8B model\n",
"ollama run llama3\n",
"```\n",
"\n",
"\n",
"他的输出如下所示\n",
"\n",
"```\n",
"$ ollama run llama3\n",
"pulling manifest \n",
"pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB \n",
"pulling 4fa551d4f938... 100% ▕████████████████▏ 12 KB \n",
"pulling 8ab4849b038c... 100% ▕████████████████▏ 254 B \n",
"pulling 577073ffcc6c... 100% ▕████████████████▏ 110 B \n",
"pulling 3f8eb4da87fa... 100% ▕████████████████▏ 485 B \n",
"verifying sha256 digest \n",
"writing manifest \n",
"removing any unused layers \n",
"success \n",
"```\n",
"\n",
"- **注意**：`llama3` 指的是 **指令微调后的 80 亿参数 LLaMA 3 模型**。 \n",
"\n",
"- **如果您的设备支持**，可以将 `llama3` 替换为 **`llama3:70b`**，以使用 **更大的 700 亿参数 LLaMA 3 模型**。 \n",
"\n",
"- **下载完成后**，您将看到 **命令行交互界面**，可以在其中与模型进行对话。 \n",
"\n",
"- **尝试输入以下提示**：\"What do llamas eat?\"（羊驼吃什么？），模型的输出应类似如下： \n",
"\n",
"```\n",
">>> What do llamas eat?\n",
"Llamas are ruminant animals, which means they have a four-chambered \n",
"stomach and eat plants that are high in fiber. In the wild, llamas \n",
"typically feed on:\n",
"1. Grasses: They love to graze on various types of grasses, including tall \n",
"grasses, wheat, oats, and barley.\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "0b5addcb-fc7d-455d-bee9-6cc7a0d684c7",
"metadata": {},
"source": [
"- 通过输入`/bye`终止程序"
]
},
{
"cell_type": "markdown",
"id": "dda155ee-cf36-44d3-b634-20ba8e1ca38a",
"metadata": {},
"source": [
"## 使用Ollama's REST API"
]
},
{
"cell_type": "markdown",
"id": "89343a84-0ddc-42fc-bf50-298a342b93c0",
"metadata": {},
"source": [
"- 另一种与模型交互的方式是 **通过 Python 调用其 REST API**，可以使用以下函数实现。 \n",
"- **在运行本笔记本中的代码前**，请确保 **Ollama 仍在运行**，可以通过以下方式启动：\n",
" - 在终端中执行 `ollama serve`\n",
" - 使用 **Ollama 应用程序** \n",
"\n",
"- 接下来，运行下面的代码单元 **查询模型**。 \n",
"\n",
"- **首先，我们用一个简单的示例测试 API**，以确保其 **正常运行**： \n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "65b0ba76-1fb1-4306-a7c2-8f3bb637ccdb",
"metadata": {},
"outputs": [],
"source": [
"import urllib.request\n",
"import json\n",
"\n",
"def query_model(prompt, model=\"llama3\", url=\"http://localhost:11434/api/chat\", role=\"user\"):\n",
" # 创建数据负载作为字典\n",
" data = {\n",
" \"model\": model,\n",
" \"seed\": 123, # 用于生成确定性响应\n",
" \"temperature\": 1., # 用于生成确定性响应\n",
" \"top_p\": 1, \n",
" \"messages\": [\n",
" {\"role\": role, \"content\": prompt}\n",
" ]\n",
" }\n",
"\n",
" # 将字典转换为JSON格式的字符串并编码为字节\n",
" payload = json.dumps(data).encode(\"utf-8\")\n",
"\n",
" # 创建请求对象，设置方法为POST并添加必要的头信息\n",
" request = urllib.request.Request(url, data=payload, method=\"POST\")\n",
" request.add_header(\"Content-Type\", \"application/json\")\n",
"\n",
" # 发送请求并捕获响应\n",
" response_data = \"\"\n",
" with urllib.request.urlopen(request) as response:\n",
" # 读取并解码响应\n",
" while True:\n",
" line = response.readline().decode(\"utf-8\")\n",
" if not line:\n",
" break\n",
" response_json = json.loads(line)\n",
" response_data += response_json[\"message\"][\"content\"]\n",
"\n",
" return response_data"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "4fb61a4e-2706-431a-835e-7e472b42989e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Llamas are herbivores, which means they primarily eat plants and plant-based foods. Their diet typically consists of:\n",
"\n",
"1. Grasses: Llamas love to graze on various types of grasses, including tall grasses, short grasses, and even weeds.\n",
"2. Hay: They enjoy eating hay, such as alfalfa or timothy hay, which provides them with fiber, protein, and other essential nutrients.\n",
"3. Grains: Llamas may eat grains like oats, barley, or corn as a supplement to their diet.\n",
"4. Leaves: They will also munch on leaves from trees and shrubs, including clover, alfalfa, and various types of leaves.\n",
"5. Fruits and vegetables: In the wild, llamas might eat fruits and vegetables that grow in their natural habitat, such as apples, carrots, or potatoes.\n",
"\n",
"In general, a llama's diet should consist of:\n",
"\n",
"* 50% grasses and hay\n",
"* 20% grains (like oats or corn)\n",
"* 10% leaves and other plant material\n",
"* 5% fruits and vegetables (as treats)\n",
"\n",
"It's essential to provide llamas with a balanced diet that meets their nutritional needs, as they can be prone to health issues if they don't receive the right combination of nutrients.\n"
]
}
],
"source": [
"result = query_model(\"What do Llamas eat?\")\n",
"print(result)"
]
},
{
"cell_type": "markdown",
"id": "8c079c6c-5845-4b31-a648-060d0273cd1d",
"metadata": {},
"source": [
"## 提取指令（Extract Instructions）\n",
"\n",
"- 现在，让我们使用论文中提出的 **“巧妙方法”**：提供 **空的提示模板** `\"<|begin_of_text|><|start_header_id|>user<|end_header_id|>\"`， \n",
" 这样 **指令微调后的 LLaMA 3 模型** 就会 **自动生成一条指令**。 "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "d7320a41-ed86-49e9-8eb1-5d609a82ad74",
"metadata": {},
"outputs": [],
"source": [
"def extract_instruction(text):\n",
" for content in text.split(\"\\n\"):\n",
" if content:\n",
" return content.strip()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "bc41b72f-a8cf-4367-b0ca-0bf8c1f094fd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"I am trying to find a way to make my child's birthday party more special and unique. What are some creative ideas you have?\n"
]
}
],
"source": [
"query = \"<|begin_of_text|><|start_header_id|>user<|end_header_id|>\"\n",
"\n",
"result = query_model(query, role=\"assistant\")\n",
"instruction = extract_instruction(result)\n",
"print(instruction)"
]
},
{
"cell_type": "markdown",
"id": "04d04ba7-bffc-47f0-87dc-d60fc676b14a",
"metadata": {},
"source": [
"- 如上所示，模型 **准确地生成了一条指令**。 "
]
},
{
"cell_type": "markdown",
"id": "162a4739-6f03-4092-a5c2-f57a0b6a4c4d",
"metadata": {},
"source": [
"## 响应生成"
]
},
{
"cell_type": "markdown",
"id": "2542f8d3-2db2-4a89-ae50-8825eb19d3b6",
"metadata": {},
"source": [
"- 接下来，我们需要 **生成对应的响应**\n",
"- 这可以 **直接将指令作为输入** 传递给模型完成。 "
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "2349eb06-710f-4459-8a03-1a3b2e1e8905",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"What an exciting question! I'd be delighted to help you come up with some creative and unique ideas to make your child's birthday party truly special!\n",
"\n",
"Here are a few ideas to get you started:\n",
"\n",
"1. **Themed Scavenger Hunt**: Plan a scavenger hunt based on the birthday child's favorite theme (e.g., superheroes, animals, or princesses). Hide clues and challenges throughout the party area, leading up to a final surprise.\n",
"2. **DIY Crafts Station**: Set up a craft station where kids can create their own party favors, such as customized t-shirts, crowns, or jewelry. This activity encourages creativity and makes for a memorable keepsake.\n",
"3. **Mystery Box Challenge**: Fill mystery boxes with different textures, smells, and sounds. Have the kids guess what's inside each box without looking. This game promotes problem-solving and teamwork.\n",
"4. **Indoor Camping Adventure**: Set up a cozy indoor \"camping\" area with sleeping bags, flashlights, and s'mores-making stations. Kids can enjoy a camping experience without leaving the party location.\n",
"5. **Personalized Photo Booth**: Create a customized photo booth with props and backdrops that match the birthday child's theme. This activity allows kids to take home special memories and share them on social media.\n",
"6. **Foodie Fun**: Plan a cooking or baking station where kids can make their own treats, such as cupcakes, pizzas, or trail mix. This activity teaches valuable skills and lets kids enjoy their creations.\n",
"7. **Outdoor Movie Night**: Set up an outdoor movie screen (or projector) with cozy seating and snacks. Screen the birthday child's favorite film or a classic kid-friendly movie.\n",
"8. **Science Experiments**: Host a science-themed party where kids can conduct fun experiments, such as making slime, creating lava lamps, or growing crystals.\n",
"9. **Karaoke Contest**: Set up a karaoke machine with popular kids' songs and have a singing competition. Offer prizes for the best performances, and provide fun props like microphones and costumes.\n",
"10. **Time Capsule Ceremony**: Have each guest bring a small item that represents their favorite memory or something they're looking forward to in the future. Bury the time capsule together as a group, with instructions to open it on a specific date (e.g., next year's birthday party).\n",
"11. **Special Guest Appearance**: Arrange for a special guest, such as a superhero, princess, or even a real-life animal (if feasible), to make an appearance at the party.\n",
"12. **Customized Games**: Design custom games and activities that fit the birthday child's interests and personality. This could include a customized version of a favorite game or a new game altogether.\n",
"\n",
"Remember, the key to making your child's birthday party unique is to incorporate elements that reflect their personality and interests. Mix and match these ideas or come up with something entirely new – the possibilities are endless!\n",
"\n",
"What do you think? Is there anything in particular that resonates with you, or would you like more suggestions?\n"
]
}
],
"source": [
"response = query_model(instruction, role=\"user\")\n",
"print(response)"
]
},
{
"cell_type": "markdown",
"id": "b12cf92c-3272-4b36-ae30-d1135af56328",
"metadata": {},
"source": [
"## 生成数据集"
]
},
{
"cell_type": "markdown",
"id": "470037f3-64f4-4465-9f00-55b69e883a04",
"metadata": {},
"source": [
"- 我们可以 **扩展此方法** 以处理 **任意数量的数据样本**（建议使用 **额外的筛选机制**，例如通过 **另一个 LLM 评估数据质量** 或 **限制数据长度**）。 \n",
"- **下面的示例** 生成 **5 组合成的指令-响应对**，在 **M3 MacBook Air** 上 **约需 3 分钟**。 \n",
"- **如果要构建可用于指令微调的数据集**，建议将数据量扩展至 **1k-50k**，并 **使用 GPU 加速数据生成**。 \n",
"\n",
"**提示（Tip）** \n",
"- 您可以通过将 `model=\"llama3\"` **更改为** `model=\"llama3:70b\"` 来生成 **更高质量的响应**，但这将 **需要更多的计算资源**。 "
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "3b9e94ab-02ef-4372-91cd-60128159fd83",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████████████████████████████████████████| 5/5 [02:37<00:00, 31.41s/it]\n"
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"dataset_size = 5\n",
"dataset = []\n",
"\n",
"for i in tqdm(range(dataset_size)):\n",
"\n",
" result = query_model(query, role=\"assistant\")\n",
" instruction = extract_instruction(result)\n",
" response = query_model(instruction, role=\"user\")\n",
" entry = {\n",
" \"instruction\": instruction,\n",
" \"output\": response\n",
" }\n",
" dataset.append(entry)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "5fdbc194-c12a-4138-96d1-51bf66ca1574",
"metadata": {},
"outputs": [],
"source": [
"with open(\"instruction-data-llama3-7b.json\", \"w\") as file:\n",
" json.dump(dataset, file, indent=4)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b4027ead-bba4-49b7-9965-47532c3fdeee",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\n",
" {\n",
" \"instruction\": \"What is the significance of the number 7 in various cultures and religions?\",\n",
" \"output\": \"The number 7 has been a significant and recurring theme across many cultures and religions, often imbuing it with special meaning and symbolism. Here are some examples:\\n\\n1. **Numerology**: In numerology, the number 7 is considered sacred and mystical, associated with spiritual awakening, introspection, and enlightenment.\\n2. **Judaism**: The Torah has seven days of creation, seven weeks in the wilderness, and seven years of rest (Sabbatical year). Seven is also a symbol of completion or perfection.\\n3. **Christianity**: In Christianity, there are seven deadly sins, seven virtues, and seven sacraments. Jesus was said to have spoken seven sermons, and the number 7 appears in various biblical accounts, such as the seven days of creation and the seven angels who appear before God.\\n4. **Islam**: In Islamic tradition, there are seven heavens, seven earths, and seven veils that separate the physical world from the divine realm. The Quran mentions seven verses (Ayats) that were revealed to Muhammad in a single revelation.\\n5. **Hinduism**: In Hindu mythology, there are seven chakras (energy centers) in the human body, seven colors of the rainbow, and seven planets (including the sun and moon). The number 7 is also associated with the seven aspects of the divine feminine (Saptamatrikas).\\n6. **Buddhism**: In Buddhist cosmology, there are seven levels of existence (dhatus) and seven stages of enlightenment (bodhisattva path). The seven days of meditation retreats in Tibetan Buddhism are also a nod to this number.\\n7. **Ancient Greek and Roman cultures**: The number 7 was considered perfect or complete, with seven being the number of planets known at the time (including Earth, Sun, Moon, Mars, Mercury, Jupiter, and Venus). Seven days were also the typical duration of a week in these cultures.\\n8. **Chinese culture**: In Chinese astrology, there are seven celestial stems and 12 earthly branches, creating a system of cyclical time reckoning. The number 7 is also associated with good luck and prosperity.\\n9. **African cultures**: In some African traditions, the number 7 is associated with spiritual powers, such as divination and healing. Seven is also considered a sacred number in Yoruba culture, representing the seven orishas (deities).\\n10. **Esoteric traditions**: Many esoteric traditions, including Hermeticism, Gnosticism, and Theosophy, have employed the number 7 to represent spiritual principles, such as the seven planes of existence, seven chakras, or seven sacred seals.\\n\\nThese examples illustrate the widespread significance of the number 7 across various cultures and religions. While the specific meanings may vary, the number 7 often represents completion, perfection, spiritual growth, and mystical connections.\"\n",
" },\n",
" {\n",
" \"instruction\": \"I've been wondering, what is the most important thing you can learn from a child?\",\n",
" \"output\": \"What a great question!\\n\\nChildren have so much to teach us, and I think one of the most important things we can learn from them is the power of simplicity, curiosity, and joy. Children are often unencumbered by the complexities and worries that come with adulthood, and they approach life with a fresh perspective.\\n\\nHere are some specific things children can teach us:\\n\\n1. **Simplicity**: Children are masters of simplifying things. They don't get bogged down in details or worry about what others think. They just enjoy the moment and find joy in simple things like playing outside, drawing, or reading.\\n2. **Curiosity**: Kids are naturally curious and love to explore their world. They ask questions, investigate, and learn from experience. This curiosity can be a powerful reminder to us adults to stay open-minded and seek new knowledge and experiences.\\n3. **Joy and playfulness**: Children have an amazing capacity for joy and playfulness. They find happiness in the simplest things, like playing with blocks or having a water balloon fight. We could all benefit from embracing our inner child and finding more joy in life's simple pleasures.\\n4. **Unconditional love**: Children are capable of loving unconditionally, without judgment or expectation. This can be a powerful reminder to us adults to practice self-love, self-acceptance, and kindness towards others.\\n5. **Creativity and imagination**: Kids are incredibly creative and imaginative, often thinking outside the box and coming up with innovative solutions. This creativity can inspire us adults to think differently, try new things, and approach problems from unique angles.\\n6. **Resilience**: Children are surprisingly resilient in the face of challenges and setbacks. They learn to adapt, cope, and move forward, teaching us valuable lessons about perseverance and bouncing back from adversity.\\n\\nIn summary, learning from children can help us regain a sense of simplicity, curiosity, joy, and playfulness, while also reminding us of the importance of unconditional love, creativity, and resilience.\"\n",
" },\n",
" {\n",
" \"instruction\": null,\n",
" \"output\": \"\"\n",
" },\n",
" {\n",
" \"instruction\": \"What is the best way to deal with a difficult person?\",\n",
" \"output\": \"Dealing with a difficult person can be challenging, but there are strategies that can help you navigate the situation effectively. Here are some tips:\\n\\n1. **Stay calm**: Take a deep breath and try not to take their behavior personally. A calm demeanor can help de-escalate tensions and prevent misunderstandings.\\n2. **Listen actively**: Sometimes, people act out because they feel unheard or misunderstood. Make an effort to listen carefully to what they're saying, and respond thoughtfully.\\n3. **Set boundaries**: Establish clear limits on what you are and aren't willing to engage in. Be firm but respectful when communicating your needs.\\n4. **Avoid taking the bait**: Don't let their provocations get under your skin. Stay focused on the issue at hand and avoid getting drawn into an argument or debate.\\n5. **Use \\\"I\\\" statements**: When expressing yourself, use \\\"I\\\" statements instead of \\\"you\\\" statements, which can come across as accusatory. This helps to reduce defensiveness and promotes a more constructive conversation.\\n6. **Practice empathy**: Try to understand where the other person is coming from, even if you don't agree with their perspective. Showing that you care about their feelings can help diffuse tension.\\n7. **Don't take it personally**: Remember that the difficult person's behavior is often a reflection of themselves, not you. Keep your self-worth and confidence intact.\\n8. **Seek common ground**: Look for areas of agreement or shared interests. This can help to build bridges and create a more positive atmosphere.\\n9. **Use humor (carefully)**: A well-timed, lighthearted joke or witty remark can help diffuse tension and lighten the mood. However, be cautious not to offend or make light of serious issues.\\n10. **Know when to disengage**: If the situation becomes too heated or toxic, it may be necessary to take a step back and re-engage at a later time when emotions have cooled down.\\n11. **Seek support**: Don't be afraid to ask for help from friends, family, or a professional if you're struggling to manage your interactions with a difficult person.\\n12. **Practice self-care**: Take care of yourself physically, emotionally, and mentally. Engage in activities that bring you joy and help you maintain your energy and resilience.\\n\\nRemember, dealing with a difficult person is not about winning an argument or changing their behavior; it's about maintaining your own emotional well-being and responding constructively to the situation.\"\n",
" },\n",
" {\n",
" \"instruction\": \"I'm looking for a way to get my cat's attention when they're hiding under the bed or in a closet.\",\n",
" \"output\": \"The classic \\\"where'd my cat go?\\\" conundrum! Don't worry, I've got some tips to help you coax your kitty out from their hiding spots:\\n\\n1. **Use their favorite treats**: Cats love food, and familiar treats can be a powerful lure. Try calling your cat's name and saying \\\"treat time\\\" in a playful tone. This might encourage them to emerge and investigate.\\n2. **Make some noise**: Cats have poor eyesight but excellent hearing. Gently knock on the bed frame or closet door with your knuckles, making a soft, rhythmic sound. This can help your cat pinpoint where you are and entice them to come out.\\n3. **Speak softly and calmly**: When speaking to your cat, use a gentle, soothing tone. Avoid loud or harsh voices, as these might scare them even further into hiding.\\n4. **Use verbal cues**: Establish a consistent verbal cue, like \\\"come on out\\\" or \\\"let's play,\\\" which can become associated with the idea of leaving their hiding spot.\\n5. **Create a \\\"safe zone\\\"**: If your cat is hiding due to fear or anxiety (e.g., from loud noises or other pets), try creating a safe, comfortable space for them to emerge into. This could be a cozy blanket or a familiar toy.\\n6. **Patiently wait it out**: Sometimes, cats just need time and space to feel secure enough to come out. Give your cat the opportunity to leave their hiding spot at their own pace.\\n7. **Use a flashlight (carefully)**: If your cat is hiding in a dark space, try using a flashlight to create a gentle beam of light. Be cautious not to shine it directly into their eyes, as this could startle them further.\\n8. **Offer a familiar object**: Place a familiar toy or blanket near the entrance to the hiding spot, which can help your cat feel more comfortable coming out.\\n9. **Make the space inviting**: If your cat is hiding under the bed, try moving any clutter or dust bunnies away from the area. Make the space underneath the bed a pleasant place for them to emerge into.\\n10. **Be patient and don't force it**: Respect your cat's boundaries and allow them to come out when they're ready. Forcing them to leave their hiding spot can create negative associations and make them more likely to hide in the future.\\n\\nRemember, every cat is different, so try a combination of these methods to see what works best for your feline friend.\"\n",
" }\n",
"]"
]
}
],
"source": [
"!cat instruction-data-llama3-7b.json"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}