{ "cells": [ { "cell_type": "markdown", "id": "08f4321d-d32a-4a90-bfc7-e923f316b2f8", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka
\n", "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", "
汉化的库: https://github.com/GoatCsu/CN-LLMs-from-scratch.git\n", "\n", "		\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "ce9295b2-182b-490b-8325-83a67c4a001d", "metadata": {}, "source": [ "# 第四章: 从零开始构建 GPT 模型" ] }, { "cell_type": "code", "execution_count": 1, "id": "f9eac223-a125-40f7-bacc-bd0d890450c7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "matplotlib version: 3.9.0\n", "torch version: 2.4.0\n", "tiktoken version: 0.7.0\n" ] } ], "source": [ "from importlib.metadata import version\n", "\n", "import matplotlib\n", "import tiktoken\n", "import torch\n", "\n", "print(\"matplotlib version:\", version(\"matplotlib\"))\n", "print(\"torch version:\", version(\"torch\"))\n", "print(\"tiktoken version:\", version(\"tiktoken\"))\n", "#加载并确认版本" ] }, { "cell_type": "markdown", "id": "e7da97ed-e02f-4d7f-b68e-a0eba3716e02", "metadata": {}, "source": [ "- 在这一章我们要用类GPT LLM架构\n", "- 下一章就是训练LLM了" ] }, { "cell_type": "markdown", "id": "7d4f11e0-4434-4979-9dee-e1207df0eb01", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "53fe99ab-0bcf-4778-a6b5-6db81fb826ef", "metadata": {}, "source": [ "## 4.1 LLM架构" ] }, { "cell_type": "markdown", "id": "ad72d1ff-d82d-4e33-a88e-3c1a8831797b", "metadata": {}, "source": [ "- 第 1 章讨论了 GPT 和 Llama 等模型，这些模型基于原始 Transformer 架构的解码器部分，按顺序生成单词。\n", "- 因此，这些大语言模型（LLM）通常被称为“类解码器”的 LLM。\n", "- 与传统的深度学习模型相比，LLM 的规模更大，这主要是由于其参数数量庞大，而非代码量的增加。\n", "- 我们将看到，在 LLM 的架构中，许多元素是重复的。" ] }, { "cell_type": "markdown", "id": "5c5213e9-bd1c-437e-aee8-f5e8fb717251", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "0d43f5e2-fb51-434a-b9be-abeef6b98d99", "metadata": {}, "source": [ "- 在前几章中，为了便于说明，我们使用了较小的嵌入维度对标记输入和输出进行处理，以确保内容能够显示在单页内。\n", "- 本章将讨论与小型 GPT-2 模型类似的嵌入和模型规模。\n", "- 我们将具体实现最小的 GPT-2 模型架构（1.24 亿参数）。该架构来源于 Radford 等人的报告 [《Language Models are Unsupervised Multitask Learners》](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)（需要注意的是，初始报告中参数数量被列为 1.17 亿，但这一错误在模型权重库中已被更正）。\n", "- 第 6 章将展示如何将预训练权重加载到我们的实现中，这些权重可兼容 3.45 亿、7.62 亿和 15.42 亿参数规模的模型。" ] }, { "cell_type": "markdown", "id": "21baa14d-24b8-4820-8191-a2808f7fbabc", "metadata": {}, "source": [ "- 123million参数的GPT-2配置如下:" ] }, { "cell_type": "code", "execution_count": 2, "id": "5ed66875-1f24-445d-add6-006aae3c5707", "metadata": {}, "outputs": [], "source": [ "GPT_CONFIG_124M = {\n", " \"vocab_size\": 50257, # Vocabulary size\n", " \"context_length\": 1024, # Context length\n", " \"emb_dim\": 768, # Embedding dimension\n", " \"n_heads\": 12, # Number of attention heads\n", " \"n_layers\": 12, # Number of layers\n", " \"drop_rate\": 0.1, # Dropout rate\n", " \"qkv_bias\": False # Query-Key-Value bias\n", "}\n", "#初始化定义需要的各种超参数" ] }, { "cell_type": "markdown", "id": "c12fcd28-d210-4c57-8be6-06cfcd5d73a4", "metadata": {}, "source": [ "- 我们使用简短的变量名，以避免代行过长。\n", "- `\"vocab_size\"` 表示词汇表大小，共 50,257 个单词，由第 2 章介绍的 BPE 分词器支持。\n", "- `\"context_length\"` 表示模型的最大输入标记数，依赖于第 2 章的位置信息嵌入。\n", "- `\"emb_dim\"` 是标记输入的嵌入维度，将每个标记转换为 768 维向量。\n", "- `\"n_heads\"` 是多头注意力机制中的注意力头数量，详见第 3 章。\n", "- `\"n_layers\"` 是模型中 Transformer 块的数量，后续章节会详细实现。\n", "- `\"drop_rate\"` 是 dropout 机制的强度，设置为 0.1，表示训练时丢弃 10% 的隐藏单元以防止过拟合（第 3 章讨论）。\n", "- `\"qkv_bias\"` 决定多头注意力机制中的 `Linear` 层是否包含偏置向量。现代 LLM 通常禁用此选项，但在第 5 章加载 OpenAI 的 GPT-2 预训练权重时，会重新启用以保持兼容性。" ] }, { "cell_type": "markdown", "id": "4adce779-857b-4418-9501-12a7f3818d88", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": 3, "id": "619c2eed-f8ea-4ff5-92c3-feda0f29b227", "metadata": {}, "outputs": [], "source": [ "import torch\n", "import torch.nn as nn\n", "\n", "\n", "class DummyGPTModel(nn.Module):\n", " def __init__(self, cfg):\n", " super().__init__()\n", " self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n", " # 词嵌入层，将输入索引转换为词向量，词表大小由字典大小和特征维度决定。\n", " self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n", " # 位置信息嵌入层，基于文本长度和特征维度生成位置信息。\n", " self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n", " # Dropout 层，用于随机丢弃一部分嵌入信息以减少过拟合。\n", "\n", " # 使用多个 Transformer 块（占位符）\n", " self.trf_blocks = nn.Sequential(\n", " *[DummyTransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])]\n", " )\n", " # Transformer 模块的堆叠，模型核心部分。\n", "\n", " # 使用归一化层（占位符）\n", " self.final_norm = DummyLayerNorm(cfg[\"emb_dim\"])\n", " # 最终归一化层，用于调整特征分布。\n", "\n", " self.out_head = nn.Linear(\n", " cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n", " )\n", " # 输出层，将特征映射到词表分布，最终预测输出单词。\n", "\n", " def forward(self, in_idx):\n", " batch_size, seq_len = in_idx.shape\n", " # 获取批次大小和序列长度。\n", "\n", " tok_embeds = self.tok_emb(in_idx) \n", " # 根据输入索引生成词嵌入。\n", " pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n", " # 生成对应的位置信息嵌入。\n", "\n", " x = tok_embeds + pos_embeds\n", " # 将词嵌入和位置信息嵌入相加。\n", " x = self.drop_emb(x)\n", " # 应用 Dropout 随机丢弃部分信息。\n", " x = self.trf_blocks(x)\n", " # 通过多个 Transformer 块处理特征。\n", " x = self.final_norm(x)\n", " # 应用最终的归一化层。\n", " logits = self.out_head(x)\n", " # 将隐藏状态映射到词表分布，生成预测结果。\n", " return logits\n", "\n", "\n", "class DummyTransformerBlock(nn.Module):\n", " # Transformer 块的占位类。\n", " def __init__(self, cfg):\n", " super().__init__()\n", " # 占位，实际模型应实现注意力机制和前馈网络。\n", "\n", " def forward(self, x):\n", " # 此块不执行任何操作，仅返回输入。\n", " return x\n", "\n", "\n", "class DummyLayerNorm(nn.Module):\n", " # 归一化层的占位类。\n", " def __init__(self, normalized_shape, eps=1e-5):\n", " super().__init__()\n", " # 参数用于模拟 LayerNorm 的接口。\n", "\n", " def forward(self, x):\n", " # 此层不执行任何操作，仅返回输入。\n", " return x" ] }, { "cell_type": "markdown", "id": "9665e8ab-20ca-4100-b9b9-50d9bdee33be", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": 4, "id": "794b6b6c-d36f-411e-a7db-8ac566a87fee", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[6109, 3626, 6100, 345],\n", " [6109, 1110, 6622, 257]])\n" ] } ], "source": [ "import tiktoken\n", "\n", "tokenizer = tiktoken.get_encoding(\"gpt2\")\n", "#召唤gpt大神\n", "batch = []\n", "\n", "txt1 = \"Every effort moves you\"\n", "txt2 = \"Every day holds a\"\n", "\n", "batch.append(torch.tensor(tokenizer.encode(txt1)))\n", "batch.append(torch.tensor(tokenizer.encode(txt2)))\n", "#编码输入文本\n", "batch = torch.stack(batch, dim=0)\n", "#按照横向来叠加两个向量\n", "print(batch)" ] }, { "cell_type": "code", "execution_count": 5, "id": "009238cd-0160-4834-979c-309710986bb0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Output shape: torch.Size([2, 4, 50257])\n", "tensor([[[-1.2034, 0.3201, -0.7130, ..., -1.5548, -0.2390, -0.4667],\n", " [-0.1192, 0.4539, -0.4432, ..., 0.2392, 1.3469, 1.2430],\n", " [ 0.5307, 1.6720, -0.4695, ..., 1.1966, 0.0111, 0.5835],\n", " [ 0.0139, 1.6754, -0.3388, ..., 1.1586, -0.0435, -1.0400]],\n", "\n", " [[-1.0908, 0.1798, -0.9484, ..., -1.6047, 0.2439, -0.4530],\n", " [-0.7860, 0.5581, -0.0610, ..., 0.4835, -0.0077, 1.6621],\n", " [ 0.3567, 1.2698, -0.6398, ..., -0.0162, -0.1296, 0.3717],\n", " [-0.2407, -0.7349, -0.5102, ..., 2.0057, -0.3694, 0.1814]]],\n", " grad_fn=)\n" ] } ], "source": [ "torch.manual_seed(123)\n", "model = DummyGPTModel(GPT_CONFIG_124M)\n", "\n", "logits = model(batch)\n", "print(\"Output shape:\", logits.shape)\n", "print(logits)" ] }, { "cell_type": "markdown", "id": "f8fad0fe-895d-4493-9e48-962e2d46c66f", "metadata": {}, "source": [ "---\n", "\n", "**Note**\n", "\n", "- 系统为Windows或者Linux, 运行结果如下所示:\n", " \n", "```\n", "Output shape: torch.Size([2, 4, 50257])\n", "tensor([[[-0.9289, 0.2748, -0.7557, ..., -1.6070, 0.2702, -0.5888],\n", " [-0.4476, 0.1726, 0.5354, ..., -0.3932, 1.5285, 0.8557],\n", " [ 0.5680, 1.6053, -0.2155, ..., 1.1624, 0.1380, 0.7425],\n", " [ 0.0447, 2.4787, -0.8843, ..., 1.3219, -0.0864, -0.5856]],\n", "\n", " [[-1.5474, -0.0542, -1.0571, ..., -1.8061, -0.4494, -0.6747],\n", " [-0.8422, 0.8243, -0.1098, ..., -0.1434, 0.2079, 1.2046],\n", " [ 0.1355, 1.1858, -0.1453, ..., 0.0869, -0.1590, 0.1552],\n", " [ 0.1666, -0.8138, 0.2307, ..., 2.5035, -0.3055, -0.3083]]],\n", " grad_fn=)\n", "```\n", "\n", "- Since these are just random numbers, this is not a reason for concern, and you can proceed with the remainder of the chapter without issues\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "f8332a00-98da-4eb4-b882-922776a89917", "metadata": {}, "source": [ "## 4.2 归一化操作" ] }, { "cell_type": "markdown", "id": "066cfb81-d59b-4d95-afe3-e43cf095f292", "metadata": {}, "source": [ "- **层归一化（LayerNorm）**，也称为层标准化（[Ba et al. 2016](https://arxiv.org/abs/1607.06450)），将神经网络层的激活值中心化为均值为 0，并将其方差归一化为 1。\n", "- 这种方法能够稳定训练过程，并加速权重的高效收敛。\n", "- 在 Transformer 块中，层归一化会在多头注意力模块的前后应用（我们将在后续实现），并在最终输出层之前再次应用。" ] }, { "cell_type": "markdown", "id": "314ac47a-69cc-4597-beeb-65bed3b5910f", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": 6, "id": "79e1b463-dc3f-44ac-9cdb-9d5b6f64eb9d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tensor([[0.2260, 0.3470, 0.0000, 0.2216, 0.0000, 0.0000],\n", " [0.2133, 0.2394, 0.0000, 0.5198, 0.3297, 0.0000]],\n", " grad_fn=)\n" ] } ], "source": [ "torch.manual_seed(123)\n", "\n", "# create 2 training examples with 5 dimensions (features) each\n", "batch_example = torch.randn(2, 5) \n", "\n", "layer = nn.Sequential(nn.Linear(5, 6), nn.ReLU())\n", "#一个按照顺序执行的神经网络\n", "#具体: 全链接层跟,激活函数\n", "out = layer(batch_example)\n", "print(out)" ] }, { "cell_type": "markdown", "id": "8fccc29e-71fc-4c16-898c-6137c6ea5d2e", "metadata": {}, "source": [ "- 计算上述信息的均值与方差" ] }, { "cell_type": "code", "execution_count": 7, "id": "9888f79e-8e69-44aa-8a19-cd34292adbf5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mean:\n", " tensor([[0.1324],\n", " [0.2170]], grad_fn=)\n", "Variance:\n", " tensor([[0.0231],\n", " [0.0398]], grad_fn=)\n" ] } ], "source": [ "mean = out.mean(dim=-1, keepdim=True)\n", "var = out.var(dim=-1, keepdim=True)\n", "\n", "print(\"Mean:\\n\", mean)\n", "print(\"Variance:\\n\", var)" ] }, { "cell_type": "markdown", "id": "052eda3e-b395-48c4-acd4-eb8083bab958", "metadata": {}, "source": [ "- 归一化会单独对两个输入（行）进行处理；\n", "- 设置 `dim=-1` 的意思是让计算沿着最后一个维度进行（在这里是特征维度），而不是按行处理。" ] }, { "cell_type": "markdown", "id": "570db83a-205c-4f6f-b219-1f6195dde1a7", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "9f8ecbc7-eb14-4fa1-b5d0-7e1ff9694f99", "metadata": {}, "source": [ "- 通过减去均值并除以方差的平方根（即标准差），可以让输入在列（特征）维度上的均值变为 0，方差变为 1：" ] }, { "cell_type": "code", "execution_count": 8, "id": "9a1d1bb9-3341-4c9a-bc2a-d2489bf89cda", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Normalized layer outputs:\n", " tensor([[ 0.6159, 1.4126, -0.8719, 0.5872, -0.8719, -0.8719],\n", " [-0.0189, 0.1121, -1.0876, 1.5173, 0.5647, -1.0876]],\n", " grad_fn=)\n", "Mean:\n", " tensor([[-5.9605e-08],\n", " [ 1.9868e-08]], grad_fn=)\n", "Variance:\n", " tensor([[1.0000],\n", " [1.0000]], grad_fn=)\n" ] } ], "source": [ "out_norm = (out - mean) / torch.sqrt(var)\n", "#执行归一化操作\n", "print(\"Normalized layer outputs:\\n\", out_norm)\n", "\n", "mean = out_norm.mean(dim=-1, keepdim=True)\n", "var = out_norm.var(dim=-1, keepdim=True)\n", "print(\"Mean:\\n\", mean)\n", "print(\"Variance:\\n\", var)" ] }, { "cell_type": "markdown", "id": "ac62b90c-7156-4979-9a79-ce1fb92969c1", "metadata": {}, "source": [ "- 每个输入的均值都会被调整为 0，方差被归一化为 1。\n", "- 为了结果容易阅读，我们可以禁用 PyTorch 的科学计数法：" ] }, { "cell_type": "code", "execution_count": 9, "id": "3e06c34b-c68a-4b36-afbe-b30eda4eca39", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mean:\n", " tensor([[ -0.0000],\n", " [ 0.0000]], grad_fn=)\n", "Variance:\n", " tensor([[1.0000],\n", " [1.0000]], grad_fn=)\n" ] } ], "source": [ "torch.set_printoptions(sci_mode=False)\n", "print(\"Mean:\\n\", mean)\n", "print(\"Variance:\\n\", var)" ] }, { "cell_type": "markdown", "id": "944fb958-d4ed-43cc-858d-00052bb6b31a", "metadata": {}, "source": [ "- 上面我们对每个输入的特征进行了归一化。\n", "- 现在，基于相同的思路，我们可以实现一个 `LayerNorm` 类：" ] }, { "cell_type": "code", "execution_count": 10, "id": "3333a305-aa3d-460a-bcce-b80662d464d9", "metadata": {}, "outputs": [], "source": [ "class LayerNorm(nn.Module):\n", " #layer归一化的函数,可以避免信息泄露也可以稳定\n", " def __init__(self, emb_dim):\n", " super().__init__()\n", " self.eps = 1e-5 #避免0的产生导致崩溃\n", " self.scale = nn.Parameter(torch.ones(emb_dim)) #动态的缩放参数\n", " self.shift = nn.Parameter(torch.zeros(emb_dim)) #动态的偏移参数\n", "\n", " def forward(self, x):\n", " mean = x.mean(dim=-1, keepdim=True)#算平均值\n", " var = x.var(dim=-1, keepdim=True, unbiased=False)#算方差\n", " norm_x = (x - mean) / torch.sqrt(var + self.eps)#归一化\n", " return self.scale * norm_x + self.shift #通过Ω和 œ 调整归一化后的值范围和位置" ] }, { "cell_type": "markdown", "id": "e56c3908-7544-4808-b8cb-5d0a55bcca72", "metadata": {}, "source": [ "## 缩放与平移\n", "\n", "- 除了通过减去均值并除以方差来执行归一化操作外，我们还引入了两个可训练参数：`scale`（缩放参数）和 `shift`（平移参数）。\n", "- 初始时，`scale` 值为 1，`shift` 值为 0，不会对结果产生影响；但在训练过程中，LLM 会自动调整这两个参数，以提升模型在任务中的表现。\n", "- 这种设计使模型能够学习到最适合其数据的缩放和平移方式。\n", "- 此外，在计算方差的平方根时，我们会添加一个较小的值（`eps`），以避免方差为 0 时出现除以 0 的错误。\n", "\n", "## 偏差方差\n", "\n", "- 在方差计算中，设置 `unbiased=False` 使用公式 $\\frac{\\sum_i (x_i - \\bar{x})^2}{n}$，其中 $n$ 是样本大小（即特征或列的数量）。该公式未使用贝塞尔校正（分母为 `n` 而非 `n-1`），因此提供的是方差的偏差估计。\n", "- 对于嵌入维度 $n$ 较大的 LLM 来说，使用 `n` 和 `n-1` 的差异可以忽略不计。\n", "- 然而，由于 GPT-2 在归一化层的训练中使用了偏差方差，为了与预训练权重兼容，我们也采用了这种设置。\n", "\n", "## 实践 LayerNorm\n", "\n", "- 现在让我们通过实际代码尝试 `LayerNorm` 的应用：" ] }, { "cell_type": "code", "execution_count": 11, "id": "23b1000a-e613-4b43-bd90-e54deed8d292", "metadata": {}, "outputs": [], "source": [ "ln = LayerNorm(emb_dim=5)#归一化一个五维度\n", "out_ln = ln(batch_example)" ] }, { "cell_type": "code", "execution_count": 12, "id": "94c12de2-1cab-46e0-a099-e2e470353bff", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mean:\n", " tensor([[ -0.0000],\n", " [ 0.0000]], grad_fn=)\n", "Variance:\n", " tensor([[1.0000],\n", " [1.0000]], grad_fn=)\n" ] } ], "source": [ "mean = out_ln.mean(dim=-1, keepdim=True)\n", "var = out_ln.var(dim=-1, unbiased=False, keepdim=True)\n", "\n", "print(\"Mean:\\n\", mean)\n", "print(\"Variance:\\n\", var)" ] }, { "cell_type": "markdown", "id": "e136cfc4-7c89-492e-b120-758c272bca8c", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "11190e7d-8c29-4115-824a-e03702f9dd54", "metadata": {}, "source": [ "## 4.3 GELU作为激活函数" ] }, { "cell_type": "markdown", "id": "b0585dfb-f21e-40e5-973f-2f63ad5cb169", "metadata": {}, "source": [ "- 在本节中，我们将实现一个小型神经网络子模块，该模块是 LLM 中 Transformer 块的核心组成部分。\n", "- 首先，我们从激活函数开始。\n", "- 在深度学习中，ReLU（线性整流单元）激活函数因其简单性和在各种神经网络架构中的高效性而被广泛使用。\n", "- 在 LLM 中，除了传统的 ReLU，还使用了其他类型的激活函数。其中两个典型的例子是 GELU（高斯误差线性单元）和 SwiGLU（Swish 门控线性单元）。\n", "- GELU 和 SwiGLU 是更复杂的平滑激活函数，分别结合了高斯函数和 sigmoid 门控线性单元，提供了比 ReLU 这种简单分段线性函数更好的性能，尤其适用于深度学习模型。" ] }, { "cell_type": "markdown", "id": "7d482ce7-e493-4bfc-a820-3ea99f564ebc", "metadata": {}, "source": [ "- **GELU**（[Hendrycks 和 Gimpel, 2016](https://arxiv.org/abs/1606.08415)）可以通过多种方式实现；其精确定义为 $\\text{GELU}(x) = x \\cdot \\Phi(x)$，其中 $\\Phi(x)$ 是标准高斯分布的累积分布函数。\n", "- 在实际应用中，通常会使用一种计算成本更低的近似形式： \n", " $\\text{GELU}(x) \\approx 0.5 \\cdot x \\cdot \\left(1 + \\tanh\\left[\\sqrt{\\frac{2}{\\pi}} \\cdot \\left(x + 0.044715 \\cdot x^3\\right)\\right]\\right)$ \n", " （原始 GPT-2 模型也是使用该近似公式进行训练的）。" ] }, { "cell_type": "code", "execution_count": 13, "id": "f84694b7-95f3-4323-b6d6-0a73df278e82", "metadata": {}, "outputs": [], "source": [ "class GELU(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", "\n", " def forward(self, x):\n", " return 0.5 * x * (1 + torch.tanh(\n", " #这一步把它变得平滑了很多\n", " torch.sqrt(torch.tensor(2.0 / torch.pi)) * \n", " (x + 0.044715 * torch.pow(x, 3))\n", " ))" ] }, { "cell_type": "code", "execution_count": 14, "id": "fc5487d2-2576-4118-80a7-56c4caac2e71", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAEiCAYAAABkykQ1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABoBElEQVR4nO3deVhUZfsH8O8My7AJiiDIIioqigsqpKG5lYpbRSnZ4p6WhpVLlvgrTXuTytxyt1KSNPelzExcSM0dREWDXEBc2JRVlmGYOb8/kEkElGE7Z4bv57rmet85c5b7nsl5uOc5z/PIBEEQQEREREREVAVysQMgIiIiIiL9x8KCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRlbGwICIiIiKiKmNhQUREREREVcbCgqgMn3/+OWQymSjXDgkJgUwmQ3x8fK1fu7CwEB9//DFcXV0hl8vh7+9f6zFUhJjvERHVbWPGjEHTpk1FubaYbdODBw8wfvx4ODo6QiaTYcqUKaLE8TRivkfEwqJOiouLw+TJk9GqVStYWFjAwsICnp6eCAwMxMWLF0vsW/wPtLxHUlISACA+Ph4ymQzffvttuddt2rQphgwZUuZr586dg0wmQ0hISLXl+TS5ubn4/PPPER4eXmvXfNT8+fOxe/duUa5dnnXr1mHBggUYNmwYfvrpJ0ydOlXUeKT4HhEZsuKivfhhbGwMZ2dnjBkzBnfu3KnUOcPDwyGTybB9+/Zy95HJZJg8eXKZr23fvh0ymaxWv6vv3r2Lzz//HFFRUbV2zWJit03lmT9/PkJCQjBp0iSEhoZi5MiRosUi1feIAGOxA6DatXfvXgwfPhzGxsZ466234OXlBblcjpiYGOzcuROrVq1CXFwc3NzcShy3atUqWFlZlTpf/fr1ayny6pebm4u5c+cCAHr37l3itU8//RQzZ86s0evPnz8fw4YNK9UrMHLkSLz++utQKBQ1ev2yHD58GM7Ozli8eHGtX7ssUnyPiOqCefPmoVmzZsjPz8epU6cQEhKC48ePIzo6GmZmZmKHV+Pu3r2LuXPnomnTpujYsWOJ177//ntoNJoau7bYbVN5Dh8+jGeffRZz5swR5fqPkup7RCws6pTr16/j9ddfh5ubGw4dOoTGjRuXeP3rr7/GypUrIZeX7sgaNmwY7OzsaitU0RkbG8PYWJx/HkZGRjAyMhLl2ikpKXpRLIr5HhHVBQMHDoSPjw8AYPz48bCzs8PXX3+NX3/9Fa+99prI0YnLxMREtGuL2TalpKTA09NTlGvrQsz3iHgrVJ3yzTffICcnB+vXry9VVABF/xg/+OADuLq6ihBdxaSlpeGjjz5C+/btYWVlBWtrawwcOBAXLlwotW9+fj4+//xztGrVCmZmZmjcuDFeffVVXL9+HfHx8bC3twcAzJ07V9vt//nnnwMofY9mu3bt0KdPn1LX0Gg0cHZ2xrBhw7Tbvv32W3Tr1g0NGzaEubk5vL29S90CIJPJkJOTg59++kl77TFjxgAof/zAypUr0bZtWygUCjg5OSEwMBAZGRkl9unduzfatWuHK1euoE+fPrCwsICzszO++eabJ76vxbeyHTlyBJcvX9bGFB4err2N4fEu5+JjHr19bcyYMbCyssKdO3fg7+8PKysr2Nvb46OPPoJarS713i1duhTt27eHmZkZ7O3tMWDAAJw7d06S7xFRXdajRw8ART9QPSomJgbDhg2Dra0tzMzM4OPjg19//VWMEHHz5k2899578PDwgLm5ORo2bIiAgIAyx2JlZGRg6tSpaNq0KRQKBVxcXDBq1Cjcu3cP4eHheOaZZwAAY8eO1X7/FH/XPTrGQqVSwdbWFmPHji11jaysLJiZmeGjjz4CABQUFGD27Nnw9vaGjY0NLC0t0aNHDxw5ckR7jK5tE1A0Nu6LL76Au7s7FAoFmjZtilmzZkGpVJbYr/h25OPHj6NLly4wMzND8+bNsWHDhie+r8VtQFxcHH7//XdtTPHx8eV+F5fVbujy3Vud7XdtvEf0HxYWdcjevXvRokULdO3aVedj09LScO/evRKPx/9gqw03btzA7t27MWTIECxatAgzZszApUuX0KtXL9y9e1e7n1qtxpAhQzB37lx4e3tj4cKF+PDDD5GZmYno6GjY29tj1apVAIBXXnkFoaGhCA0NxauvvlrmdYcPH46jR49qx5QUO378OO7evYvXX39du23p0qXo1KkT5s2bh/nz58PY2BgBAQH4/ffftfuEhoZCoVCgR48e2mu/++675eb9+eefIzAwEE5OTli4cCGGDh2KNWvWoH///lCpVCX2TU9Px4ABA+Dl5YWFCxeidevW+OSTT/DHH3+Ue357e3uEhoaidevWcHFx0cbUpk2bco8pj1qthp+fHxo2bIhvv/0WvXr1wsKFC7F27doS+7399tuYMmUKXF1d8fXXX2PmzJkwMzPDqVOnJPkeEdVlxX84NmjQQLvt8uXLePbZZ/HPP/9g5syZWLhwISwtLeHv749du3bVeoxnz57FiRMn8Prrr+O7777DxIkTcejQIfTu3Ru5ubna/R48eIAePXpg2bJl6N+/P5YuXYqJEyciJiYGt2/fRps2bTBv3jwAwDvvvKP9/unZs2epa5qYmOCVV17B7t27UVBQUOK13bt3Q6lUatuHrKws/PDDD+jduze+/vprfP7550hNTYWfn592LIeubRNQ1KM0e/ZsdO7cGYsXL0avXr0QHBxcol0qdu3aNQwbNgz9+vXDwoUL0aBBA4wZMwaXL18u9/xt2rRBaGgo7Ozs0LFjR21MxX/c66Ii373V3X7XxntEjxCoTsjMzBQACP7+/qVeS09PF1JTU7WP3Nxc7Wtz5swRAJT58PDw0O4XFxcnABAWLFhQbgxubm7C4MGDy3zt7NmzAgBh/fr1T8wjPz9fUKvVJbbFxcUJCoVCmDdvnnbbunXrBADCokWLSp1Do9EIgiAIqampAgBhzpw5pfYpzrtYbGysAEBYtmxZif3ee+89wcrKqsR79uj/FwRBKCgoENq1ayc8//zzJbZbWloKo0ePLnXt9evXCwCEuLg4QRAEISUlRTA1NRX69+9fIvfly5cLAIR169Zpt/Xq1UsAIGzYsEG7TalUCo6OjsLQoUNLXetxvXr1Etq2bVti25EjRwQAwpEjR0psL/7MH/3MRo8eLQAo8VkIgiB06tRJ8Pb21j4/fPiwAED44IMPSsVQ/PkIgjTfIyJDVvxv6+DBg0Jqaqpw69YtYfv27YK9vb2gUCiEW7duafd94YUXhPbt2wv5+fnabRqNRujWrZvQsmVL7bbi75Bt27aVe10AQmBgYJmvbdu2rczvoMc9/t0rCIJw8uTJUv/eZ8+eLQAQdu7cWWr/4u+fJ7VJo0ePFtzc3LTP//zzTwGA8Ntvv5XYb9CgQULz5s21zwsLCwWlUllin/T0dMHBwUEYN26cdpsubVNUVJQAQBg/fnyJ/T766CMBgHD48GHtNjc3NwGAcPToUe22lJQUQaFQCNOnTy91rceV1YY//l1crKx2o6LfvdXdftfme0SCwB6LOiIrKwsAyhyA3bt3b9jb22sfK1asKLXPjh07EBYWVuKxfv36Go/7cQqFQjsGRK1W4/79+7CysoKHhwciIyNLxGtnZ4f333+/1DkqMw1dq1at0LFjR2zZskW7Ta1WY/v27XjxxRdhbm6u3f7o/09PT0dmZiZ69OhRIj5dHDx4EAUFBZgyZUqJ8S8TJkyAtbV1iZ4QoOgzHjFihPa5qakpunTpghs3blTq+pUxceLEEs979OhR4vo7duyATCYrcxBgZT4ffXyPiKSsb9++sLe3h6urK4YNGwZLS0v8+uuvcHFxAVDUi3348GG89tpryM7O1vZk379/H35+frh69WqlZ5GqrEe/e1UqFe7fv48WLVqgfv36pdoHLy8vvPLKK6XOUZnvn+effx52dnYl2of09HSEhYVh+PDh2m1GRkYwNTUFUHQraFpaGgoLC+Hj41Pp9mHfvn0AgGnTppXYPn36dAAo9d3n6empva0NKOoh8fDwqLXvvop891Z3+61v75G+4+iWOqJevXoAirqAH7dmzRpkZ2cjOTm5xD/4R/Xs2bNWBm8/7Uuj+L78lStXIi4ursR9+w0bNtT+/+vXr8PDw6NaB3ANHz4cs2bNwp07d+Ds7Izw8HCkpKSUaDiAolvO/ve//yEqKqrE/ZuVnVf75s2bAAAPD48S201NTdG8eXPt68VcXFxKXatBgwalphKuKcXjJR6/fnp6uvb59evX4eTkBFtb22q5pr69R0RSt2LFCrRq1QqZmZlYt24djh49WmIWtmvXrkEQBHz22Wf47LPPyjxHSkoKnJ2dqy2mp32H5uXlITg4GOvXr8edO3cgCIL2tczMTO3/v379OoYOHVptcRkbG2Po0KHYtGkTlEolFAoFdu7cCZVKVap9+Omnn7Bw4ULExMSUuEWzWbNmlbr2zZs3IZfL0aJFixLbHR0dUb9+/VLffU2aNCl1jse/n2tSRb57q7v91rf3SN+xsKgjbGxs0LhxY0RHR5d6rXjMRU0vNmZmZoa8vLwyXyu+//Vp0xjOnz8fn332GcaNG4cvvvgCtra2kMvlmDJlSo1O/wcUFRZBQUHYtm0bpkyZgq1bt8LGxgYDBgzQ7nPs2DG89NJL6NmzJ1auXInGjRvDxMQE69evx6ZNm2o0vmLlzZb0aCOri/Ia88cHYz/t+lJS3e8RkaHp0qWLdlYof39/PPfcc3jzzTcRGxsLKysr7fftRx99BD8/vzLP8fgfck+iUCiq3D68//77WL9+PaZMmQJfX1/Y2NhAJpPh9ddfr/H24fXXX8eaNWvwxx9/wN/fH1u3bkXr1q3h5eWl3efnn3/GmDFj4O/vjxkzZqBRo0YwMjJCcHBwqUHxuqroD1dSbR9q47tXrPeormFhUYcMHjwYP/zwA86cOYMuXbrU+vXd3Nxw5cqVMl+LjY3V7vMk27dvR58+ffDjjz+W2J6RkVGiR8Xd3R2nT5+GSqUqd2pAXXsQmjVrhi5dumDLli2YPHkydu7cCX9//xK/4u3YsQNmZmb4888/S2wv67axil6/+D2JjY1F8+bNtdsLCgoQFxeHvn376pSHrooHaz4+WP/xX3l04e7ujj///BNpaWlP7LXQl/eIyJAV//Hbp08fLF++HDNnztT+OzMxMamWf19ubm7aduBxurQPo0ePxsKFC7Xb8vPzS313ubu7l/kj26N0bR969uyJxo0bY8uWLXjuuedw+PBh/N///V+p+Jo3b46dO3eWOP/jt4Tqcm03NzdoNBpcvXq1xGQbycnJyMjIeOp7VlU11T5UZ/st9ntU13CMRR3y8ccfw8LCAuPGjUNycnKp12u6Gh80aBBu375daiVlpVKJH374AY0aNULnzp2feA4jI6NScW7btq3UvbxDhw7FvXv3sHz58lLnKD7ewsICQOkvxCcZPnw4Tp06hXXr1uHevXulurmNjIwgk8lK/FoTHx9f5urRlpaWFbp23759YWpqiu+++65E7j/++CMyMzMxePDgCsdfGW5ubjAyMsLRo0dLbF+5cmWlzzl06FAIgqBd4OhRj+aoL+8RkaHr3bs3unTpgiVLliA/Px+NGjVC7969sWbNGiQmJpbaPzU1VafzDxo0CKdOnUJERESJ7RkZGdi4cSM6duwIR0fHJ56jrPZh2bJlpX49Hzp0KC5cuFDmzFXFx1taWmqvXxFyuRzDhg3Db7/9htDQUBQWFpbZPjx6DQA4ffo0Tp48WWI/XdqmQYMGAQCWLFlSYvuiRYsAoMa/+9zd3QGgRPugVqtLzQKoi+puv8V+j+oa9ljUIS1btsSmTZvwxhtvwMPDQ7vytiAIiIuLw6ZNmyCXy7WD8x61ffv2Mgd+9+vXDw4ODtrnhw4dQn5+fqn9/P398c4772DdunUICAjAuHHj0KlTJ9y/fx9btmxBdHQ0NmzYoB3YVp4hQ4Zg3rx5GDt2LLp164ZLly5h48aNJX6lBoBRo0Zhw4YNmDZtGs6cOYMePXogJycHBw8exHvvvYeXX34Z5ubm8PT0xJYtW9CqVSvY2tqiXbt2aNeuXbnXf+211/DRRx/ho48+gq2tbalf6gYPHoxFixZhwIABePPNN5GSkoIVK1agRYsWpe7f9/b2xsGDB7Fo0SI4OTmhWbNmZU4FbG9vj6CgIMydOxcDBgzASy+9hNjYWKxcuRLPPPNMueNiqouNjQ0CAgKwbNkyyGQyuLu7Y+/evUhJSan0Ofv06YORI0fiu+++w9WrVzFgwABoNBocO3YMffr0weTJkwHoz3tEVBfMmDEDAQEBCAkJwcSJE7FixQo899xzaN++PSZMmIDmzZsjOTkZJ0+exO3bt0utL7Rjxw7ExMSUOu/o0aMxc+ZMbNu2DT179sS7776L1q1b4+7duwgJCUFiYmKFJgsZMmQIQkNDYWNjA09PT5w8eRIHDx4sMf6uOI/t27dr2yJvb2+kpaXh119/xerVq+Hl5QV3d3fUr18fq1evRr169WBpaYmuXbs+cSzE8OHDsWzZMsyZMwft27cvNV33kCFDsHPnTrzyyisYPHgw4uLisHr1anh6epYY/6hL2+Tl5YXRo0dj7dq1yMjIQK9evXDmzBn89NNP8Pf3L3P9perUtm1bPPvsswgKCtL2QG/evBmFhYWVPmd1t99iv0d1Ti3PQkUScO3aNWHSpElCixYtBDMzM8Hc3Fxo3bq1MHHiRCEqKqrEvk+abhaPTCVXPPVoeY/Q0FBBEIqm1ps6darQrFkzwcTERLC2thb69Okj/PHHHxWKPT8/X5g+fbrQuHFjwdzcXOjevbtw8uRJoVevXkKvXr1K7Jubmyv83//9n/Zajo6OwrBhw4Tr169r9zlx4oTg7e0tmJqalpi67vHp6h7VvXv3MqeuK/bjjz8KLVu2FBQKhdC6dWth/fr1ZZ4vJiZG6Nmzp2Bubi4A0E6rWt70fcuXLxdat24tmJiYCA4ODsKkSZOE9PT0EvuUNV2sIJSeHrE85R2fmpoqDB06VLCwsBAaNGggvPvuu0J0dHSZ081aWlqWOr6s/AsLC4UFCxYIrVu3FkxNTQV7e3th4MCBQkREhHYfKb5HRIas+N/W2bNnS72mVqsFd3d3wd3dXSgsLBQEQRCuX78ujBo1SnB0dBRMTEwEZ2dnYciQIcL27du1xxVPPVre49ixY4IgCMLt27eF8ePHC87OzoKxsbFga2srDBkyRDh16lSFYk9PTxfGjh0r2NnZCVZWVoKfn58QExMjuLm5lZq2+v79+8LkyZMFZ2dnwdTUVHBxcRFGjx4t3Lt3T7vPnj17BE9PT8HY2LjEd1153xUajUZwdXUVAAj/+9//ynx9/vz5gpubm6BQKIROnToJe/fuLfN8urRNKpVKmDt3rratc3V1FYKCgkpMAywI5U/5Xlb7WZbyjr9+/brQt29fQaFQCA4ODsKsWbOEsLCwMqebreh3b3W337X1HpEgyASBo1GIiIiIiKhqOMaCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRldW5BfI0Gg3u3r2LevXq6bQkPBGRIRMEAdnZ2XBycoJcXnd/c2IbQURUki7tQ50rLO7evQtXV1exwyAikqRbt27BxcVF7DBEwzaCiKhsFWkf6lxhUa9ePQBFb461tbVOx6pUKhw4cAD9+/eHiYlJTYRXKwwhD+YgHYaQhyHkAFQtj6ysLLi6umq/I+uqut5GMAfpMIQ8DCEHwDDyqK32oc4VFsVd29bW1pVqNCwsLGBtba23/2EBhpEHc5AOQ8jDEHIAqiePun77T11vI5iDdBhCHoaQA2AYedRW+1B3b6QlIiIiIqJqw8KCiIiIiIiqTNTCYtWqVejQoYO2y9nX1xd//PHHE4/Ztm0bWrduDTMzM7Rv3x779u2rpWiJiKi2sH0gItI/ohYWLi4u+OqrrxAREYFz587h+eefx8svv4zLly+Xuf+JEyfwxhtv4O2338b58+fh7+8Pf39/REdH13LkRERUk9g+EBHpH1ELixdffBGDBg1Cy5Yt0apVK3z55ZewsrLCqVOnytx/6dKlGDBgAGbMmIE2bdrgiy++QOfOnbF8+fJajpyIiGoS2wciIv0jmVmh1Go1tm3bhpycHPj6+pa5z8mTJzFt2rQS2/z8/LB79+5yz6tUKqFUKrXPs7KyABSNjlepVDrFWLy/rsdJjSHkwRykwxDyMIgc1BrM23sFrdSVy0PKuddU+0BEVFccu3oPh+/KMFAQavQ6ohcWly5dgq+vL/Lz82FlZYVdu3bB09OzzH2TkpLg4OBQYpuDgwOSkpLKPX9wcDDmzp1bavuBAwdgYWFRqZjDwsIqdZzUGEIezEE6DCEPfc5h6w05/k6Wo6HCCDamYTDWsT86Nze3ZgKrgppuHwD++PQ45iAdhpCHIeQA6H8eN9NyMWXrRWTlG8HnbAJe7+Km0/G65C16YeHh4YGoqChkZmZi+/btGD16NP76669yGw9dBQUFlfgVq3iRj/79+1dqjvKwsDD069dPb+cxBgwjD+YgHYaQh77n8PPpBPx9MgYyAK801WCgn+55FP9BLSU13T4A/PGpPMxBOgwhD0PIAdDPPJRqYPElI2Tly+BmJcAi5TL27St7rFp5dPnhSfTCwtTUFC1atAAAeHt74+zZs1i6dCnWrFlTal9HR0ckJyeX2JacnAxHR8dyz69QKKBQKEptNzExqfQfEFU5VkoMIQ/mIB2GkIc+5nDsair+ty8WADC9X0u4PvinUnlIMe+abh8A/vj0OOYgHYaQhyHkAOhvHoIgYMrWi0jMS0ZDS1OMa5Vb4z88iV5YPE6j0ZToln6Ur68vDh06hClTpmi3hYWFlXvPLRGRIbuR+gCBGyOh1gh4tbMz3unRFH/88Y/YYdWYmmgf+ONT2ZiDdBhCHoaQA6B/eaz+6zr2RSfDWC7D8je8kHL5ZI3/8CRqYREUFISBAweiSZMmyM7OxqZNmxAeHo4///wTADBq1Cg4OzsjODgYAPDhhx+iV69eWLhwIQYPHozNmzfj3LlzWLt2rZhpEBHVusxcFcb/dA5Z+YXo3KQ+5r/SHjJoxA6r2rB9ICKqvKP/puKb/TEAgDkvtYWPWwPoeAdUpYhaWKSkpGDUqFFITEyEjY0NOnTogD///BP9+vUDACQkJEAu/28EYrdu3bBp0yZ8+umnmDVrFlq2bIndu3ejXbt2YqVARFTrCtUaTP4lEjfu5cDJxgxrRvrAzMQIKpXhFBZsH4iIKifhfi7e/+U8NAIQ4O2CEV2boLCwsFauLWph8eOPPz7x9fDw8FLbAgICEBAQUEMRERFJ3/9+/wfHrt6DuYkRvh/tA/t6pW/l0XdsH4iIdJdbUIh3Qs8hM08FL9f6+MK/HWQyWa1dX9QF8oiISDebTicg5EQ8AGDxcC+0dbIRNyAiIpIEQRDwyY5LiEnKhp2VKVaP6AwzE6NajYGFBRGRnjh5/T5m74kGAEzv1woD2jUWOSIiIpKKH47F4bcLd2Esl2HlW95obGNe6zGwsCAi0gMJ93MxaWMECjUCXvRywuTnW4gdEhERScTxq/cQ/HBWwM+GeKJLM1tR4mBhQUQkcdn5KozfcBYZuSp0cLHBgmEdavWeWSIikq5babl4/5dIaARgmLcLRvnqtrJ2dWJhQUQkYWqNgCmbo/Bv8gM4WCvw/SifWr9nloiIpCmvQI13QyOQ/vCHp//V8mDtx7GwICKSsAV/xuJQTAoUxnKsHekDB2szsUMiIiIJEAQBQTsv4kpiFhpammL1CG/Rf3hiYUFEJFE7I29j9V/XAQDfDOsAL9f64gZERESSse7veOyOugsjuQwr3uoMp/q1P1j7cSwsiIgk6HxCOmbuvAQACOzjjpc7OoscERERScWJ6/cwf1/RYO1PB7fBs80bihxRERYWREQSk5iZh3dCI1BQqEE/TwdM7+chdkhERCQRt9NzMXnTeag1Al7t7Iwx3ZqKHZIWCwsiIgnJV6nxzoYIpGYr0dqxHpYM7wi5nDNAERFRURsx8ecIpOUUoJ2zNea/0l5SswSysCAikghBEDBj+0VcupMJW0tTfD/KB5YKY7HDIiIiCRAEAbN2XUL0nSzYWppizUjpzRLIwoKISCJWhl9/ZNXUznC1tRA7JCIikoiQE/HYGXkHRnIZlr/ZCc4SGKz9OBYWREQSEHYlGd8eiAUAzH25rWQG4hERkfhO3biP//1eNFh71qA26OZuJ3JEZWNhQUQkstikbEzZfB6CAIzydcNbXcVbNZWIiKTlTkYeAjdGQq0R4N/RCeO6NxU7pHKxsCAiElF6TgHGbziLnAI1fJs3xGdDPMUOiYiIJCJfpcaknyNwP6cAbZ2sEfxqB0kN1n4cCwsiIpGo1Bq8tzESt9Ly4GprjpVvdYaJEb+WiYioaLD2/+2KxsXbmWhgYYLVI7xhbiqtwdqPYwtGRCSS/+29gpM37sPS1Ag/jHoGDSxNxQ6JiIgkIvTUTeyIvA25DFj+pn5M6MHCgohIBL+cScBPJ28CABYP7wgPx3oiR0RERFJx+sZ9zPvtCgAgaGAbdG8hzcHajxO1sAgODsYzzzyDevXqoVGjRvD390dsbOwTjwkJCYFMJivxMDMzq6WIiYiq7mx8GmbviQYAfNS/Ffq3dRQ5IiIikorEzDwEbopEoUbAS15OGN+jmdghVZiohcVff/2FwMBAnDp1CmFhYVCpVOjfvz9ycnKeeJy1tTUSExO1j5s3b9ZSxEREVXMnIw8TQyOgUgsY3KExAvu0EDskIiKSiKKVtSNx70EB2jS2xtdDpT1Y+3GiFhb79+/HmDFj0LZtW3h5eSEkJAQJCQmIiIh44nEymQyOjo7ah4ODQy1FTERUeXkFarwbeg73cwrg2dgaC4bpV4NRm9ijTUR1jSAImL0nGhduZaC+hQnWjpT+YO3HSWqMRWZmJgDA1tb2ifs9ePAAbm5ucHV1xcsvv4zLly/XRnhERJUmCAI+2XER0XeyYGtpirWjvGFhaix2WJLFHm0iqmt+Pp2AreeKBmsve6OTXgzWfpxkWjWNRoMpU6age/fuaNeuXbn7eXh4YN26dejQoQMyMzPx7bffolu3brh8+TJcXFxK7a9UKqFUKrXPs7KyAAAqlQoqlUqnGIv31/U4qTGEPJiDdBhCHrWRw9pjcfj1wl0Yy2X4bngHOFiZVPv1qpKH1D6//fv3l3geEhKCRo0aISIiAj179iz3uOIebSIifXI2Pg1zfy36ofyTAa3Ro6W9yBFVjmQKi8DAQERHR+P48eNP3M/X1xe+vr7a5926dUObNm2wZs0afPHFF6X2Dw4Oxty5c0ttP3DgACwsKlcJhoWFVeo4qTGEPJiDdBhCHjWVw5V0GdbGyAHI4O9WiPv/nMK+f2rkUgAql0dubm4NRFJ9dO3R1mg06Ny5M+bPn4+2bdvWRohERJWSlJmPST8XDdYe3KEx3unZXOyQKk0ShcXkyZOxd+9eHD16tMxehycxMTFBp06dcO3atTJfDwoKwrRp07TPs7Ky4Orqiv79+8Pa2lqna6lUKoSFhaFfv34wMTHR6VgpMYQ8mIN0GEIeNZlD3L0cfLrmNAQUYriPC754qU2NjauoSh7FvblSVFM92gB7tR/HHKTDEPIwhByAms1DWajBxJ/P4d4DJTwcrDD/5TYoLCys9uvUVo+2qIWFIAh4//33sWvXLoSHh6NZM92n01Kr1bh06RIGDRpU5usKhQIKhaLUdhMTk0r/AVGVY6XEEPJgDtJhCHlUdw7Z+SpM2hSF7PxC+Lg1wBf+7WFqXPND2yqTh5Q/u5rq0QbYq10e5iAdhpCHIeQA1Ewem6/LEZUih4WRgNecMhB+8EC1X+NRNd2jLWphERgYiE2bNmHPnj2oV68ekpKSAAA2NjYwNzcHAIwaNQrOzs4IDg4GAMybNw/PPvssWrRogYyMDCxYsAA3b97E+PHjRcuDiOhxGo2AqVuicD01B41tzLBqhHetFBWGpiZ7tAH2aj+OOUiHIeRhCDkANZfH5rO3cfLkFchkwPK3vNGjZc0tgldbPdqiFharVq0CAPTu3bvE9vXr12PMmDEAgISEBMjl/zXG6enpmDBhApKSktCgQQN4e3vjxIkT8PT0rK2wiYieavHBf3HwnxQojOVYM9Ib9vVK95xS+WqjRxtgr3Z5mIN0GEIehpADUL15RNxMw7zfiwbbzfDzwPOejavlvE9T0z3aot8K9TTh4eElni9evBiLFy+uoYiIiKruj0uJWHa46Ffy4Ffbo4NLfXED0kPs0SYiQ5WclY+JP0dCpRYwqL0jJvVyFzukaiOJwdtERIYiJikL07ddAAC8/VwzvNpZt9t3qAh7tInIECkL1Zj0cwRSs5Vo5WCFBcO8DGqhVBYWRETVJCO3AO9siEBugRrd3BsiaGBrsUPSW+zRJiJDNPe3K4hMyIC1mTHWjvSBpcKw/hTnSEIiomqg1gh4/5fzSEjLhUsDcyx/szOMjfgVS0RERX45k4BNpxMgkwFLX++EpnaWYodU7djqERFVgwV/xuLY1XswM5Fj7Ugf2Fqaih0SERFJRGRCOubsKVpZe3q/VujTupHIEdUMFhZERFW09+JdrP7rOgBgwTAveDrpNk0pEREZrpTsfEz6OQIFag0GtHVEYJ8WYodUY1hYEBFVwT+JWZix7SIA4N1ezfGil5PIERERkVQUFGrw3s+RSM5SokUjK3z7mmEN1n4cCwsiokrKyC3Au6ERyFOp0aOlHT7242BtIiL6zxd7r+DczXTUUxhj7UhvWBnYYO3HsbAgIqoEtUbAB5ujkJCWC1dbcyx7oxOM5Ib7KxQREelm69lbCD11s2iw9hsd0dzeSuyQahwLCyKiSlh4IBZH/02FmYkca0b4oL4FB2sTEVGRqFsZ+HR3NABgat9WeL61g8gR1Q4WFkREOvrjUiJWhhcN1v56aAcO1iYiIq3UbCUmhhYN1u7v6YDJBjxY+3EsLIiIdHA1ORsfPVxZe/xzzfByR2eRIyIiIqlQqTUI3BiJpKx8uNtbYuFrXpDXodtkWVgQEVVQVr4K74ZGIOfhytozubI2ERE94svf/8GZ+LSiwdqjfFDPzETskGoVCwsiogrQaARM23IBN+7lwLl+0WBtrqxNRETFdkTcRsiJeADA4uEd4V4HBms/jq0iEVEFLD9yDQf/SYapsRyrRnRGQyuF2CEREZFEXLydgaBdlwAAU/q2RF/PujFY+3EsLIiInuJITAoWH/wXAPA//3bo4FJf3ICIiEgy7j14OFi7UIO+bRzwwfMtxQ5JNCwsiIie4Ob9HHy4+TwEAXiraxO85uMqdkhERCQRxYO172bmo7m9JRYNr1uDtR/HwoKIqBx5BWpM/DkSWfmF6NSkPma/6Cl2SEREJCHz9/2D03FpsFIYY+1IH1jXscHaj2NhQURUBkEQMGvXJfyTmAU7K1OsessbCmMjscMiIiKJ2Bl5G+v/jgcALHzNCy0a1b3B2o9jYUFEVIYNJ29i1/k7MJLLsPzNznC0MRM7JCIikojoO5kI2lk0WPuD51vAr62jyBFJg6iFRXBwMJ555hnUq1cPjRo1gr+/P2JjY5963LZt29C6dWuYmZmhffv22LdvXy1ES0R1RcTNNHyx9woAIGhgazzbvKHIERERkVTcf6DEu6ERUBZq8HzrRpjSt5XYIUmGqIXFX3/9hcDAQJw6dQphYWFQqVTo378/cnJyyj3mxIkTeOONN/D222/j/Pnz8Pf3h7+/P6Kjo2sxciIyVCnZ+XhvYyQKNQIGd2iMt59rJnZIREQkEYVqDSZvOo87GXloZmeJxcM71unB2o8zFvPi+/fvL/E8JCQEjRo1QkREBHr27FnmMUuXLsWAAQMwY8YMAMAXX3yBsLAwLF++HKtXr67xmInIcKkeNhjJWUq0bGSFb4Z2gEzGBoOIiIp89UcMTt64D0tTI6wd6Q0b87o9WPtxohYWj8vMzAQA2NralrvPyZMnMW3atBLb/Pz8sHv37jL3VyqVUCqV2udZWVkAAJVKBZVKpVN8xfvrepzUGEIezEE6DCGP4ti/2R+LM3FpsFQYYdnrXjCVC3qVV1U+C6nlGRwcjJ07dyImJgbm5ubo1q0bvv76a3h4eDzxuG3btuGzzz5DfHw8WrZsia+//hqDBg2qpaiJyJD9eiERPxyPA1A0WLulQz2RI5IeyRQWGo0GU6ZMQffu3dGuXbty90tKSoKDQ8nVDB0cHJCUlFTm/sHBwZg7d26p7QcOHICFhUWlYg0LC6vUcVJjCHkwB+nQ9zzO35ch5N9bAIDhbgWIPfsXnj7iS5oq81nk5ubWQCSVV3yr7DPPPIPCwkLMmjUL/fv3x5UrV2BpaVnmMcW3ygYHB2PIkCHYtGkT/P39ERkZ+cR2hYjoaW7nAMv2XAYATO7TAgPaNRY5ImmSTGERGBiI6OhoHD9+vFrPGxQUVKKHIysrC66urujfvz+sra11OpdKpUJYWBj69esHExP97foyhDyYg3QYQh6xiRn4ePVpAMD455riEz/9HIhXlc+iuDdXKnirLBFJRVpOAX6MNUK+SoPeHvaY2k8/24jaIInCYvLkydi7dy+OHj0KFxeXJ+7r6OiI5OTkEtuSk5Ph6Fj2NF8KhQIKhaLUdhMTk0r/EVSVY6XEEPJgDtKhr3nkKAsxZdtlKDUydGnaADMHtoGxkX7PxF2Zz0Lqn11N3CpLRPQ0hWoNpm69iDSlDE1szbF0eCcYcbB2uUQtLARBwPvvv49du3YhPDwczZo9ffYVX19fHDp0CFOmTNFuCwsLg6+vbw1GSkSGSBAEzNx5CddSc2BtImDJax30vqgwRDV1qyzAcXiPYw7SYQh5GEIOX+2PxYkbaTCVC1j2WjtYmOhnPrU1Bk/UwiIwMBCbNm3Cnj17UK9ePe2Xv42NDczNzQEAo0aNgrOzM4KDgwEAH374IXr16oWFCxdi8ODB2Lx5M86dO4e1a9eKlgcR6aefTsTjtwt3YSyXYWyrQtjXK927SeKrqVtlAY7DKw9zkA5DyENfc4i8J8NPV40AAG+10CD+wknEXxA5qCqq6TF4ohYWq1atAgD07t27xPb169djzJgxAICEhATI5f/9gtitWzds2rQJn376KWbNmoWWLVti9+7dHJhHRDqJTEjHl/v+AQB87NcKDhmXRY6IylKTt8oCHIf3OOYgHYaQhz7n8E9iNj75/jQADcZ3b4L2mht6mUex2hqDJ/qtUE8THh5ealtAQAACAgJqICIiqgvuP1AicGMkVGoBg9s3xhjfJvjjDxYWUlJbt8pyHF7ZmIN0GEIe+pZDek4BAjdHIV+lQc9W9viovwf+3H9D7/IoS02PwZPE4G0iotqi1giYsiUKiZn5aG5via+GtgfXwJMe3ipLRGIoVGvwwebzuJWWhya2Fvju9Y4crK0DjlIkojpl6aGrOHb1HsxNjLB6hDfqmen3r0+GatWqVcjMzETv3r3RuHFj7WPLli3afRISEpCYmKh9Xnyr7Nq1a+Hl5YXt27fzVlki0smCA7HaNmLNSG/UtzAVOyS9Uqkei7i4OBw7dgw3b95Ebm4u7O3t0alTJ/j6+sLMzKy6YyQiqhbhsSlYdvgqAGD+q+3QiqumShZvlSWi2rb34l2s+esGAOCbYR3QprFu46xIx8Ji48aNWLp0Kc6dOwcHBwc4OTnB3NwcaWlpuH79OszMzPDWW2/hk08+gZubW03FTESkszsZeZiyJQqCALzVtQle6fTkgcBERFR3xCRlYca2iwCAd3s2x4teTiJHpJ8qXFh06tQJpqamGDNmDHbs2AFXV9cSryuVSpw8eRKbN2+Gj48PVq5cyV+NiEgSCgo1eG9jJDJyVejgYoPZL3qKHZJBY682EemTjNwCvLMhAnkqNXq0tMPHA1qLHZLeqnBh8dVXX8HPz6/c1xUKBXr37o3evXvjyy+/RHx8fHXER0RUZfP3/YMLtzJgY26CFW92hsLYSOyQDBJ7tYlI36g1Aj7YHIWEtFy42prju9e5snZVVLiweFJR8biGDRuiYcOGlQqIiKg6/X4xESEn4gEAi17zgqtt5RY9oydjrzYR6aOFB2Jx9N9UmJnIsWaEDxpYcrB2VVRqVqiQkJAytxcWFiIoKKgq8RARVZsbqQ/wyY6ie2Yn9XbHC20cRI7IcH311Vc4ffo03nvvvVJFBfBfr/bq1asRExOD5s2bixAlEdF/9l1KxMrw6wCAb4Z5wdOJg7WrqlKFxQcffICAgACkp6drt8XGxqJr16745Zdfqi04IqLKyitQ472NkXigLESXZraY3q+V2CEZNF17tb29vWswGiKiJ4tNysZH2y4AAN7p2RwvcbB2tahUYXH+/Hncvn0b7du3R1hYGFasWIHOnTujdevWuHDhQnXHSESkszm/RiMmKRt2VqZY/kYnGBtx2Z7awl5tIpKyzFwV3gk9h9wCNbq3aIiP/TzEDslgVKqldXd3x99//41XX30VAwYMwNSpU/HDDz9g48aNsLGxqe4YiYh0su3cLWw9dxtyGfDd653QyJozEdUm9moTkVSpNQI+3HIeN+/nwrm+OZa90Zk/PFWjSr+Tv//+OzZv3gxfX1/Ur18fP/74I+7evVudsRER6Sw2KRuf7YkGAEzt2wrdWtiJHFHdw15tIpKqxWH/Ijz24WDtkd6w5WDtalWpwuLdd99FQEAAPvnkExw7dgwXL16Eqakp2rdvj61bt1Z3jEREFZKjLMSkjRHIV2nQs5U9Avu0EDukOom92kQkRfujE7H8yDUAwFevdkA7Z34fVbdKFRZ///03Tp8+jenTp0Mmk8HR0RH79u3DvHnzMG7cuOqOkYjoqQRBwKxdl3AjNQeO1mZYMrwj5JyLXDTs1SYiKbmanI3pW4t6TMd1bwb/Ts4iR2SYKlVYREREwMvLq9T2wMBAREREVDkoIiJd/XLmFvZE3YWRXIblb3Zi97aI2KtNRFKSmafCO6ERyClQ49nmtpg1iCtr15QKL5D3KIVCUe5rHh4cWU9EtSv6TiY+/+0yAOBjPw/4NLUVOaK6rbhXu/gHqOJe7RUrVmDcuHF47bXXRI6QiOoKjUbA1C1RiLuXAycbM6x4k4O1a1KF39kBAwbg1KlTT90vOzsbX3/9NVasWFGlwIiIKiI7X4XJmyJRUKjBC60bYUIPLrwmNvZqE5FULDl0FYdjUmBqLMeakT5oaFX+j+NUdRXusQgICMDQoUNhY2ODF198ET4+PnBycoKZmRnS09Nx5coVHD9+HPv27cPgwYOxYMGCmoybiAiCIGDmzkuIfzht4MLXvDiuQgLYq01EUvDn5SR8d+gqACD4lfZo78LB2jWtwj0Wb7/9Nm7cuIFZs2bhypUreOedd9CjRw8888wz8PPzw/fff48mTZrg7Nmz2LJlC5o0afLUcx49ehQvvvginJycIJPJsHv37ifuHx4eDplMVuqRlJRU0TSIyID8fOomfr+YCGO5DMve7IT6FhxXIRb2ahORlFxL+W+w9phuTTHU20XkiOoGncZYKBQKjBgxAiNGjAAAZGZmIi8vDw0bNoSJiYnOF8/JyYGXlxfGjRuHV199tcLHxcbGwtraWvu8UaNGOl+biPTbpduZ+GLvPwCAmQNbo3OTBiJHVLexV5uIpCIrv2iw9gNlIbo2s8X/DW4jdkh1RqUGbxezsbGp0pzkAwcOxMCBA3U+rlGjRqhfv36lr0tE+i0rX4XATZEoUGvQz9MBbz/XTOyQ6ry3334bI0aMwLZt27BlyxasXbsWmZmZAACZTAZPT0/4+fnh7NmzaNOGjTwR1QyNRsC0LVG4kZqDxjZmWPFWZ5hwsHat0amw+O6778rcbmNjg1atWsHX17dagnqajh07QqlUol27dvj888/RvXv3cvdVKpVQKpXa51lZWQAAlUoFlUql03WL99f1OKkxhDyYg3TUdh6CIODjbReRkJYL5/pmCPb3RGFhYZXOyc+ienKv7l5tIiJdfXf4Kg7+UzRYe/UIb9hxsHat0qmwWLx4cZnbMzIykJmZiW7duuHXX3+FrW3NTPXYuHFjrF69Gj4+PlAqlfjhhx/Qu3dvnD59Gp07dy7zmODgYMydO7fU9gMHDsDCwqJScYSFhVXqOKkxhDyYg3TUVh7HkmTYH2cEI5mA4S4P8PeR6rtuXf4scnNzqz2OqvZqExHp4uCVZCw5WDRY+0v/dvByrS9uQHWQToVFXFxcua/duHEDI0aMwKeffoqVK1dWObCyeHh4lJhRpFu3brh+/ToWL16M0NDQMo8JCgrCtGnTtM+zsrLg6uqK/v37lxinUREqlQphYWHo16+fXv/6Zgh5MAfpqM08Lt/NwkdrTwMQ8MmA1hjbza1azsvP4r/e3Kqo7l7to0ePYsGCBYiIiEBiYiJ27doFf3//cvcPDw9Hnz59Sm1PTEyEo6OjTtcmIv1yPfUBpm6JAgCM8nVDgI+ruAHVUVUaY/Go5s2b46uvvsK4ceOq65QV0qVLFxw/frzc1xUKRZlTH5qYmFT6D4iqHCslhpAHc5COms4jK1+FD7dehEotoG8bB0zo6Q6ZrHqnlq3Ln0V15F3dvdqc4IOIKiI7X4V3NpxDtrIQXZra4rMhnmKHVGdVW2EBAE2aNKn1qV+joqLQuHHjWr0mEdUuQRAQtOMSbj5cr+LbgA7VXlRQ1VV3rzYn+CCip9FoBEzfegHXU3PgaM3B2mKr1sLi0qVLcHOr+K0JDx48wLVr17TP4+LiEBUVBVtbWzRp0gRBQUG4c+cONmzYAABYsmQJmjVrhrZt2yI/Px8//PADDh8+jAMHDlRnGkQkMT+fTsDvl4rWq1jO9Sr0Um32ausywQcR6bcVR67hwJVkmBrJsWpEZ9jX42BtMelUWJR3D25mZiYiIiIwffp0jB49usLnO3fuXIn7YYvHQowePRohISFITExEQkKC9vWCggJMnz4dd+7cgYWFBTp06ICDBw+WeU8tERmG6DuZ+OK3KwCATwa0RieuV6G3arpXuzITfHDmwJKYg3QYQh41ncOR2FQsOvgvAODzF9ugXWOrGrlWXf8sdDlGp8Kifv365d5+IJPJMH78eMycObPC5+vduzcEQSj39ZCQkBLPP/74Y3z88ccVPj8R6bfsfBUmP1yv4oXWjTC+B9er0Ge69mrrqjITfHDmwLIxB+kwhDxqIoeUPGDRJSMIggzdHTSwTL6AffsuVPt1HlVXPwtdZg3UqbA4cuRImdutra3RsmVLmJmZISUlBU5OTrqcloioFEEQMGtXNOLv58LJxgzfBnhxXIXEVXevdnV42gQfnDmwJOYgHYaQR03l8EBZiIA1p5GnzoF3k/pYO9YHpsY1N66irn8WuswaqFNh0atXrye+fuHCBXTu3BlqtVqX0xIRlfLLmVv47cJdGMllWPZmJzSw5LgKqavuXu3q8LQJPjhzYNmYg3QYQh7VmYMgCAjafBHXUnPgYK3AqpHesDSvnXEVdfWz0GX/ah28TURUHf5JzMLc3y4DAGb4ecDbrWYW3aTqVd292pzgg4getzL8OvZfToKJkQyrRnijUT0zsUOiR7CwICJJyVEWInBTJJSFGvT2sMc7PZqLHRJVUHX3anOCDyJ61JHYFHx7IBYAMO/ldujMyTwkh4UFEUmGIAj4dHc0bjycj3zRax0hl3NcRV3FCT6IqFj8vRx8+Mt5CALwZtcmeKNLE7FDojLoVFhcvHjxia/HxsZWKRgiqtu2nbuNXefvwEguw3dvdIItx1UQEdV5OcpCvBsagaz8QnRuUh9zXuTK2lKlU2HRsWNHyGSyMn9BKt7OWVuIqDL+Tc7G7F+jAQDT+rVCl2YcV0FEVNcJgoAZ2y8gNjkb9vUUWDXCGwpjI7HDonLoVFjExcXVVBxEVIflFhQicGMk8lUa9Ghph0m93MUOiSqBvdpEVN1W/3UD+y49HKz9Vmc4WHOwtpTpVFjU5MJGRFR3zdlzGVdTHqBRPQUWD+e4Cn3FXm0iqk5//ZuKb/6MAQB8/lJb+DRlT7bU6VRYfPPNN3j//fdhbm4OAPj777/h4+OjnQM8Ozsbn3zyCVauXFn9kRKRQdoRcRvbIm5DLgOWvt4Jdla1Mx85VT/2ahNRdbl5PwcfPBys/fozrniTg7X1gk6FRVBQEMaMGaMtLAYOHIioqCg0b140HWRubi7WrFnDwoKIKuRaSjY+3V00rmJK31bwdW8ockRUFezVJqLqkFtQNFg7M0+Fjq71Mffltuzt1BM6rX/+ePf2k6YBJCJ6krwCNQI3nkeeSo3uLRoisE8LsUOianTs2DGMGDECvr6+uHPnDgAgNDQUx48fFzkyIpIyQRDw8faLiEnKhp2VKVaN6MzB2npEp8KCiKi6fP7rZcQmZ8POSoElwzvBiOMqDMaOHTvg5+cHc3NznD9/HkqlEgCQmZmJ+fPnixwdEUnZ98duYO/FRBjLZVj5ljca25iLHRLpgIUFEdW6nZG3seXcLchkwHevd4R9PY6rMCT/+9//sHr1anz//fcwMTHRbu/evTsiIyNFjIyIpOz41Xv46o+iwdpzXvTktON6SOeVt3/44QdYWVkBAAoLCxESEgI7OzsARYO3iYie5FpKNv5vV9G4ig9faIluLexEjoiqW2xsLHr27Flqu42NDTIyMmo/ICKSvFtpuZj8SyQ0AhDg7YIRz3LMlj7SqbBo0qQJvv/+e+1zR0dHhIaGltqHiKgsj46r6ObeEO8/31LskKgGODo64tq1a2jatGmJ7cePH9dO9kFEVCyvQI13QiOQkauCl4sNvvBvx8HaekqnwiI+Pr6GwiCiumDOr9H/jat4vSPHVRioCRMm4MMPP8S6desgk8lw9+5dnDx5EtOnT8fs2bPFDo+IJEQQBMzceRH/JGY9HKztDTMTDtbWVzoVFvn5+Th48CCGDBkCoGj62eJBeQBgbGyMefPmwcyMqyISUUk7Im5j67mi9Sq+e70jGtXj94ShmjlzJjQaDV544QXk5uaiZ8+eUCgUmDFjBsaPHy92eEQkIT8ej8OeqLswlsuw4s3OcKrPwdr6TKfB2yEhIVizZo32+fLly3HixAmcP38e58+fR2hoqE5rWBw9ehQvvvginJycIJPJsHv37qceEx4ejs6dO0OhUKBFixYICQnRJQUiEsHV5P/Wq/jwhVYcV2HgZDIZ/u///g9paWmIjo7GqVOnkJqaChsbGzRr1kzs8IhIIk5cu4f5+/4BAHw6uA26NudaRvpOp8Ji48aNeOedd0ps27RpE44cOYIjR45gwYIF2LZtW4XPl5OTAy8vL6xYsaJC+8fFxWHw4MHo06cPoqKiMGXKFIwfPx5//vmnLmkQUS3KLSjEexsjkadS47kWdpj8PNerMFRKpRJBQUHw8fFB9+7dsW/fPnh6euLy5cvw8PDA0qVLMXXqVLHDJCIJuJWWi8BNRYO1h3Z2wehuTcUOiaqBTrdCXbt2De3bt9c+NzMzg1z+X23SpUsXBAYGVvh8AwcOxMCBAyu8/+rVq9GsWTMsXLgQANCmTRscP34cixcvhp+fX4XPQ0S1QxAEfLo7GldTHsC+ngKLh3NchSGbPXs21qxZg759++LEiRMICAjA2LFjcerUKSxcuBABAQEwMuK900R1XV6BGu+GRiA9V4X2zjb48hUO1jYUOhUWGRkZJcZUpKamlnhdo9GUeL26nTx5En379i2xzc/PD1OmTKmxaxJR5W07dxs7I+9ALgOWvdGJ61UYuG3btmHDhg146aWXEB0djQ4dOqCwsBAXLlzgHw1EBKDoB6egnRdxJTELDS1NsXokB2sbEp0KCxcXF0RHR8PDw6PM1y9evAgXF5dqCawsSUlJcHBwKLHNwcEBWVlZyMvLg7l56QE/SqWyRLGTlZUFAFCpVFCpVDpdv3h/XY+TGkPIgzlIR3l5xCRl47M9ReMqpr7QAt6u1pLN1dA/C12OrYrbt2/D29sbANCuXTsoFApMnTqVRQURaa37Ox67o+7CSC7D8jc7w5mDtQ2KToXFoEGDMHv2bAwePLjUzE95eXmYO3cuBg8eXK0BVlVwcDDmzp1bavuBAwdgYWFRqXOGhYVVNSxJMIQ8mIN0PJpHvhpYeNEIykIZ2tTXwOVBDPbtixExuooxxM+ionJzc6t8XbVaDVNTU+1zY2Nj7YKqREQnrv83WPv/BrWBrzsHaxsanQqLWbNmYevWrfDw8MDkyZPRqlUrAEWrrC5fvhyFhYWYNWtWjQQKFC26lJycXGJbcnIyrK2ty+ytAIqmxJ02bZr2eVZWFlxdXdG/f39YW1vrdH2VSoWwsDD069cPJiYmuicgEYaQB3OQjsfzEAQBU7ZeREp+MhytFfhpki8aWJg+/UQiMtTPQhfFvblVIQgCxowZA4Wi6Ja3/Px8TJw4EZaWliX227lzZ5WvRUT65U5GHiZvOg+1RsArnZwxtntTsUOiGqBTYeHg4IATJ05g0qRJmDlzJgRBAFA0tWC/fv2wcuXKUrcqVSdfX1/s27evxLawsDD4+vqWe4xCodA2co8yMTGp9B8QVTlWSgwhD+YgHcV5hPwdh33RyTCWy7ByhDca2Vg+/WCJMLTPQtdjqmr06NElno8YMaJK5zt69CgWLFiAiIgIJCYmYteuXfD393/iMeHh4Zg2bRouX74MV1dXfPrppxgzZkyV4iCiqslXqfFu6Dmk5RSgrZM1gl9tz1skDZROhQUANGvWDPv370daWhquXbsGAGjRogVsbW11vviDBw+05wCKppONioqCra0tmjRpgqCgINy5cwcbNmwAAEycOBHLly/Hxx9/jHHjxuHw4cPYunUrfv/9d52vTUTVLzIhHV8+7OaeNagNOjdpIHJEVJvWr19frecrnpJ83LhxePXVV5+6f/GU5BMnTsTGjRtx6NAhjB8/Ho0bN+bMgUQiEQRg9q9XEH0nCw0sTLCGg7UNms6FRTFbW1t06dKlShc/d+4c+vTpo31efMvS6NGjERISgsTERCQkJGhfb9asGX7//XdMnToVS5cuhYuLC3744Qc2GEQSkJZTgMkbI6FSCxjU3pHd3FRlnJKcSP8dS5JhV3wi5DJgxZud4dKgcuNbST9UurCoDr1799beTlWWslbV7t27N86fP1+DURGRrjQCMH37JdzNzEczO0t8PbQDu7mp1lVmSnLOHFgSc5AOQ8jj5LVU7IovWu/sE79WeMbNRi/zMYTPorZmDRS1sCAiw/DnbTmO374PMxM5Vo3ojHpm+j9OgfRPZaYk58yBZWMO0qGveaQrgW8vGkEDGbztNHDIuIJ9+66IHVaV6Otn8aianjWQhQURVcnRq/fw5+2i3ongV9ujtaNus60RiYkzB5bEHKRDn/NQqtR448ezeFCYBWcLAWsn9Ia1hdnTD5Qoff4sitXWrIEsLIio0m6n52L6tksQIMObXVzwSqeaWyCT6GkqMyU5Zw4sG3OQDn3LQxAEBO2+gkt3slDf3ARve+TB2sJMr3Ioj759FmWp6VkD5boGREQEFE0fOOnnSGTkqdDEUsCsga3FDonqOF9fXxw6dKjEtqdNSU5E1Sv01E1sj7gNuQxYMrwDGupvRwVVAgsLItKZIAiYvScal+5kooGFCcZ6qKEw5tcJVa8HDx4gKioKUVFRAP6bkrx4tsCgoCCMGjVKu//EiRNx48YNfPzxx4iJicHKlSuxdetWTJ06VYzwieqcM3FpmPdb0TiKmQNboztX1q5z+JcAEels89lb2Hru4S9Sr3WAbek7SYiq7Ny5c+jUqRM6deoEoGhK8k6dOmH27NkAUO6U5GFhYfDy8sLChQs5JTlRLUnMzMN7GyNQqBHwopcTJvRoLnZIJAKOsSAinZxPSMecPZcBAB/5eaCbe0PsixU5KDJInJKcSD8oC9WY+HMk7j0oQGvHevh6KFfWrqvYY0FEFZaSnY9JP0eiQK2BX1sHTOrlLnZIREQkIkEQMGfPZVy4lQEbcxOsHekDC1P+bl1XsbAgogopKNQgcGMkkrLy4W5viW8DvPiLFBFRHbfxdAI2n70FuQxY9kYnNGnIlbXrMhYWRFQhX/5+BWfj02GlMMbaUT5cBI+IqI47F5+Gub8V3Rr78YDW6NnKXuSISGwsLIjoqbaeu4WfTt4EACwe3hHu9lYiR0RERGJKzsrHpI2RUKkFDG7fGO/25GBtYmFBRE8RmZCOT3dFAwA+fKEl+nk6iBwRERGJqWiwdgRSs5XwcKiHb4Z14K2xBICFBRE9QXJWPiaGRqBArUF/Twd8+EJLsUMiIiKRff7rFZxPyIC1mTHWjPSGpYKDtakICwsiKlO+So13QiOQkq1EKwcrLBreEXI5f5EiIqrLNp1OwC9nEiCTAd+90QlN7SzFDokkhIUFEZUiCAKCdl7STh/4/SgfWPEXKSKiOi3iZjrm/Fp0a+xH/T3Q26ORyBGR1LCwIKJSVv11HbvO34GRXIaVb3WGW0P+IkVEVJelZOVj0s8RUKkFDGrviPd6cx0jKo2FBRGVcOByEhb8WbSU9ucveqJ7CzuRIyIiIjEVFGowaWOk9tbYBcO4jhGVjYUFEWlduZuFKVuiIAjAiGebYKRvU7FDIiIikc3bexkRN9NhbWaMtSN9OFibysXCgogAFM0A9fZPZ5FboEY394aY82JbsUMiIiKRbT17Cz+fKhqsvfR1DtamJ5NEYbFixQo0bdoUZmZm6Nq1K86cOVPuviEhIZDJZCUeZmZmtRgtkeHJLSjE+J/OITEzH+72llj1ljdMjCTx9UBERCI5n5COT3cXDdae3q8V+rTmYG16MtH/ctiyZQumTZuGOXPmIDIyEl5eXvDz80NKSkq5x1hbWyMxMVH7uHnzZi1GTGRYNBoBU7dE4dKdTNhammLdmGdgY2EidlhERCSilOx8TPo5EgVqDQa0dURgnxZih0R6QPTCYtGiRZgwYQLGjh0LT09PrF69GhYWFli3bl25x8hkMjg6OmofDg5cCZiosr7c9w/+vJwMUyM51o705gxQRER1XEGhBoEbI5GUlY8Wjazw7WscrE0VI+rom4KCAkRERCAoKEi7TS6Xo2/fvjh58mS5xz148ABubm7QaDTo3Lkz5s+fj7Zty74fXKlUQqlUap9nZWUBAFQqFVQqlU7xFu+v63FSYwh5MIfqEXLyJn48HgcA+OrVtvByrlcn/10YQg5A1fLQ99yJqPr87/crOBufjnoKY6wd6c11jKjCRP0v5d69e1Cr1aV6HBwcHBATE1PmMR4eHli3bh06dOiAzMxMfPvtt+jWrRsuX74MFxeXUvsHBwdj7ty5pbYfOHAAFhYWlYo7LCysUsdJjSHkwRwq78J9Gdb/Kwcgw0tN1DC6fR77bp+v9Pn4WUhHZfLIzc2tgUiISN9sPXcLG04W3WK+eHhHNLe3Ejki0id6V4L6+vrC19dX+7xbt25o06YN1qxZgy+++KLU/kFBQZg2bZr2eVZWFlxdXdG/f39YW1vrdG2VSoWwsDD069cPJib6ew+6IeTBHKrm3M10bAyJgAAN3uzigs+HtKl0Nzc/C+moSh7FvblEVHdF3crAp7uKBmtP7dsKfT15qznpRtTCws7ODkZGRkhOTi6xPTk5GY6OjhU6h4mJCTp16oRr166V+bpCoYBCoSjzuMr+AVGVY6XEEPJgDrqLTcrGuz+fh7JQg75tGmHey+1hXA0zQPGzkI7K5GEIeRNR5aVmKzExNAIFag36eTrg/ec5WJt0J+rgbVNTU3h7e+PQoUPabRqNBocOHSrRK/EkarUaly5dQuPGjWsqTCKDcTs9F6PWnUZWfiG83Rpg2Rudq6WoICIi/aVSaxC4qWiwdnN7Syx6zQtyOQdrk+5E/4ti2rRp+P777/HTTz/hn3/+waRJk5CTk4OxY8cCAEaNGlVicPe8efNw4MAB3LhxA5GRkRgxYgRu3ryJ8ePHi5UCkV64/0CJUevOIDlLiZaNrPDjaB+YmxqJHRbRE3GdI6Ka9+Xv/+BMXBqsFEUra9czYw8mVY7oYyyGDx+O1NRUzJ49G0lJSejYsSP279+vHdCdkJAAufy/+ic9PR0TJkxAUlISGjRoAG9vb5w4cQKenp5ipUAkeVn5KoxadwY3UnPgZGOGDW93QX0LU7HDInqi4nWOVq9eja5du2LJkiXw8/NDbGwsGjUqe6Eua2trxMbGap9zikyiJ9sRcRshJ+IBFA3WbtGIg7Wp8kQvLABg8uTJmDx5cpmvhYeHl3i+ePFiLF68uBaiIjIMeQVqvB1yFpfvZqGhpSlCx3dFYxtzscMieqpH1zkCgNWrV+P333/HunXrMHPmzDKPKV7niIie7tLtTATtugQA+PCFlujHwdpURZIoLIioZigL1Xj354ii+cjNjLHh7S5w59SBpAdqY50jgGsdPY45SEdN53E/pwDvhJ5DQaEGz3vY472eTav9WvwspKO21jliYUFkoAoKNXjv50gc/TcV5iZGCBn7DNo62YgdFlGF1MY6RwDXOioPc5COmshDrQFW/iNHYpYcjcwE9LdOxP79idV+nWL8LKSjptc5YmFBZIBUag0mb4rEoZgUKIzl+HG0D7zdbMUOi6hG6brOEcC1jh7HHKSjJvP4cl8MrmUlwNLUCD9N6Fpj4yr4WUhHba1zxMKCyMCo1Bp8uPk8DlxJhqmxHN+P8kG3FnZih0Wkk9pY5wjgWkflYQ7SUd157Dp/GyEnEwAAC1/riDbODart3OXhZyEdNb3OkejTzRJR9SkoLOqp2HcpCaZGcqwZ6Y2erezFDotIZ1zniKj6Rd/JxMwdRYO1J/dpgQHtONEBVS/2WBAZiHyVGu9tjMThmBSYGsuxekRn9PEoe0pOIn0wbdo0jB49Gj4+PujSpQuWLFlSap0jZ2dnBAcHAyha5+jZZ59FixYtkJGRgQULFnCdI6KH0nIK8G5oBJSFGvTxsMfUfq3EDokMEAsLIgOQW1CId0MjcOzqPZiZyLF2pA97KkjvcZ0joupR+HDc3Z2MPDRtaIElr3eCEVfWphrAwoJIz2XkFmBcyFlEJmTAwtQIP45+Br7uDcUOi6hacJ0joqr7en8MTly/D0tTI6wd5QMbc/0eJ0DSxcKCSI8lZ+Vj1I9nEJucDWszY6wf+wxnfyIiIq09UXfw/bE4AMC3AV5o5VBP5IjIkLGwINJT11MfYMz6M7iVlodG9RQIfbsrPBzZYBARUZHLdzPxyY6LAIDAPu4Y2J4TGVDNYmFBpIfOxqdhwoZzyMhVwa2hBX5+uytcbSu3mBcRERme9IeDtfNVGvT2sMe0fh5ih0R1AAsLIj2z9+JdTNt6AQWFGnR0rY8fRvvAzqr0PPxERFQ3Fao1eP+X87idnge3hhZYOpyDtal2sLAg0hMajYClh65i6aGrAAC/tg5YMrwTzE2NRI6MiIik5Js/Y3H82j1YmBphzUhv2FhwsDbVDhYWRHogR1mI6VsvYP/lJADAuO7N8H+D2/AXKCIiKuHXC3ex9ugNAMCCYV5o7WgtckRUl7CwIJK4+Hs5mPhzBGKSsmFiJMOX/u3x2jOuYodFREQS809iFj7efgEAMLGXOwZ34GBtql0sLIgkbH90ImZsu4hsZSHsrBRYM7Izp5MlIqJS0nMK8E7oOeSrNOjR0g4z/DhYm2ofCwsiCVIWqvHN/lj8eLxo7vFnmjbAsjc6w9HGTOTIiIhIatQaAR9sPo9baXlwtTXHsjc4WJvEwcKCSGKupWTjg1+icCUxCwDwTs/mmOHnARMjuciRERGRFC34MxbHrt6DuYkR1o70QX0LU7FDojpKEn+prFixAk2bNoWZmRm6du2KM2fOPHH/bdu2oXXr1jAzM0P79u2xb9++WoqUqOZoNAI2nIzH4O+O40piFhpYmGDtSG/MGtSGRQUREZVp78W7WP3XdQDA18M6oE1jDtYm8Yj+18qWLVswbdo0zJkzB5GRkfDy8oKfnx9SUlLK3P/EiRN444038Pbbb+P8+fPw9/eHv78/oqOjazlyouoTfy8Hb3x/CrP3XIaysOj+2D+n9ET/to5ih0ZERBIVk5SFGduKVtZ+p2dzvOTlJHJEVNeJXlgsWrQIEyZMwNixY+Hp6YnVq1fDwsIC69atK3P/pUuXYsCAAZgxYwbatGmDL774Ap07d8by5ctrOXKiqlNrgB+Ox2PA0qM4HZcGcxMjzHnREz+N7YJG1hxPQUREZcvILcA7GyKQp1LjuRZ2+JiDtUkCRB1jUVBQgIiICAQFBWm3yeVy9O3bFydPnizzmJMnT2LatGkltvn5+WH37t1l7q9UKqFUKrXPs7KK7ltXqVRQqVQ6xbsj4hYupciQH3kLChMTGMllMJbLYGwkg5FcBlMjOYzlMpgYyR8+ZDAxlsPUSA5TYzkUDx/GchlkMvEGVRXnrWv+UmIIORz7NwXfXDRCUt6/AIBuzW3xxcueaGJrAbW6EGq1yAFWkCF8FoaQA1C1PPQ9d6K6RK0R8OHmKCSk5cKlQdFgbWPeMksSIGphce/ePajVajg4OJTY7uDggJiYmDKPSUpKKnP/pKSkMvcPDg7G3LlzS20/cOAALCwsdIp37hkj5KmNsPH6Pzod9zgZBJjIoX2YygFTo4f/KxegMELRQw4ojAEzIwFmRoCZEWBuBJgbCzA3AiyMAXPjouMqU6eEhYVVKQ8p0MccUvOAvbfkiLovByCDpbGAl9w06GqfguhTKdDXm/r08bN4nCHkAFQuj9zc3BqIhIhqwsIDsfjr31SYmcixZqQ3GlhysDZJg8HPChUUFFSihyMrKwuurq7o378/rK11G+C0L/M8Eu4mo36DhhAAFGoEFGoEqDUCVGoBhWoNVGoBKrUGhZqi/y0o1KDg4fZiAmQo0AAFmrKuonuFYGosR31zE9Q3N0EDSxPYWpjC1tIUDS1NYWtlCjtLU9jXU8DOyhSN6ilgBA3CwsLQr18/mJiY6Hw9KVCpVHqXw70HSiw/cgNbLt5GoUaAXAZ0d9Dgm5E9YWetW5ErJfr4WTzOEHIAqpZHcW8uEUnbvkuJWBn+cLD20A5o62QjckRE/xG1sLCzs4ORkRGSk5NLbE9OToajY9mDVh0dHXXaX6FQQKFQlNpuYmKic8O7/I1O2LdvHwYNekbnYzUaAQVqDZQqDZSFauSrNMgvVCNfpUZegRq5KjXyC9TIKVAjr6AQOQVq5CgL8UBZiBxlIbLzix8qZOcXIjNPhcw8FQo1AgoKNUjJViIlW/n0QABYmxnDQmaEbakX4VTfHI425nCyMYNTfXM41TeHc31zmJsa6ZSfWCrzOda2xMw8fH80Dr+cSUCequj+pl6t7DG9bwvEnT8GO2sLyedQEfrwWTyNIeQAVC4PQ8ibyND9m5yNj7YVraw9/rlmeLmjs8gREZUkamFhamoKb29vHDp0CP7+/gAAjUaDQ4cOYfLkyWUe4+vri0OHDmHKlCnabWFhYfD19a2FiCtPLpfBTG4EMxMjANXTgAuCgNwCNdJzC5CRq0J6bgHScv573HtQgHsPlLj/QInUB0qkZCmhLNQgK78QWZAh6dr9cs9tZ2UK5wYWcGlgjia2FnBtYIEmthZwa2gBp/rmXHinAv5JzELI3/HYef62tseqo2t9fDKgNXzdG0KlUiHuvMhBEhGRXsjMU+GdDeeQW6BGN/eGmDmwtdghEZUi+q1Q06ZNw+jRo+Hj44MuXbpgyZIlyMnJwdixYwEAo0aNgrOzM4KDgwEAH374IXr16oWFCxdi8ODB2Lx5M86dO4e1a9eKmYYoZDIZLBXGsFQYw6XB0/cXBAHZykLcuf8Avx08Brc2HZD6QIW7mflIyszH3Yw83EnPQ7ay8GFRUoALtzJKncfESAbXBkVFRlM7SzR75OFkYw55HS468lVqhF1JRuipmzgTl6bd3rWZLSY/3wLPtbATdeA+ERHpH41GwNQtUYi/nwvn+uZY/mZnDtYmSRK9sBg+fDhSU1Mxe/ZsJCUloWPHjti/f792gHZCQgLk8v/+8XTr1g2bNm3Cp59+ilmzZqFly5bYvXs32rVrJ1YKekMmk8HazATmjazgUV/AoE7OZd7+kJmnwu30XNxKy3v4v7m4mZaLhLRc3E7LQ4Fagxv3cnDjXg4Qm1riWFNjOZo1tERz+4cPOyu4N7JCc3tLWJsZ5q0Wao2AyIR07Dp/B3sv3EVWfiEAwEguw4C2jhj3XFN4u9mKHCUREemrxQf/xeGYFCiMiwZr23KwNkmU6IUFAEyePLncW5/Cw8NLbQsICEBAQEANR1V32ZibwMbcpswBYWqNgKSsfNy8l4O4+zmIv5eDuHu5iLv3AAlpuSgo1CA2ORuxydmljrWzUqC5vSXcHxYczeyKig9XWwu9W1k6R1mI03H3EXYlGWFXUnDvwX/jWxrbmGGYtwve6uoGRxuuRUFUFStWrMCCBQuQlJQELy8vLFu2DF26dCl3/23btuGzzz5DfHw8WrZsia+//hqDBg2qxYiJqteBK8lYdvgaAOCroe3RzpmDtUm6JFFYkP4wksvg/HCAd7cWdiVeK1RrcCcjDzdSc3A99UFRr0bqA9xIzUFKthL3HhQ9Hr1FqPicrg3M0dTOEk0bWsKtYdFtVk1sLeHSwPzhuBRxpeUUIOpWOs4nZODUjfs4n5CBQs1/M33VMzNGP08HDOvsgmebN6zTt4MRVZctW7Zg2rRpWL16Nbp27YolS5bAz88PsbGxaNSoUan9T5w4gTfeeAPBwcEYMmQINm3aBH9/f0RGRrJXm/TSnRxgxY6iScjHdW+GVzq5iBwR0ZOxsKBqY2wkh1tDS7g1tESf1iUb/ex8FeLu5eBG6sNi4+H/j7uXgzyVGvH3cxF/PxdAaqnzNqqngHODomLGqb45HK3NYGdpjBtZwM37uXBsYAlLU6Mqj11QqTVIyszHrfRc3E7Pw/XUB7ia/AD/Jmfjdnpeqf2b2FqgZys7+LV1RNdmDWFqrF+9LkRSt2jRIkyYMEE75m716tX4/fffsW7dOsycObPU/kuXLsWAAQMwY8YMAMAXX3yBsLAwLF++HKtXr67V2ImqQlmoxorD17HikhHUghrPNrfFrEEcrE3Sx8KCakU9MxN0cKmPDi71S2wXBAHJWUrcuPcAN+/nIv7h7VUJaXlIuJ+DnAK1dird8wkZj53VGEsvHwdQNLbD5uFaHvXMjGFhagwLUyMoTIxgLJdpZ7HSaASoBQH5KjVyH07pm5Gnwv0HBcjMe/LKw+72lujo2gA+TRugu7sdmjTU37UniKSuoKAAERERCAoK0m6Ty+Xo27cvTp48WeYxJ0+eLLFuEQD4+flh9+7d5V5HqVRCqfzvVsbi9TxUKpVOq5Efv3Yfey/exZ07chzdeanE2EB9otFomIMERNxMx417uQBkeM7dFgsDOkDQqKHSqMUOTSfF/4Z0+bckRYaQR1Vy0OUYFhYkKplMBkcbMzjamKGbe8nXBEFAWk4B7jycrepORh7uZuQjOSsfiZl5uJmcjlyNEfJURQsRpmYrkVrBtTzKY2osh0t9czg3MEfThpZo5WCFlg710MbRGjYWhjn4nEiK7t27B7VarZ3Io5iDgwNiYmLKPCYpKanM/ZOSksq9TnBwMObOnVtq+4EDB2BhUfEfD8ITZdgVbwRADqQkVvg4aWIOUlDPRMCrTTXo1DAFp/46KHY4VRIWFiZ2CNXCEPKoTA65ubkV3peFBUmWTCZDQysFGlopSvV0qFSqh4sV+qFAI0N6blGPQ2auCtnKQuQVqJFTUIiCQo12ZXQAMJIDcpkMZiZGsFQYwcLUGNZmJrCvZ4qGlgrYmJtwfARRHRIUFFSilyMrKwuurq7o378/rK2tK3wel9uZcLuaimvXrqJFi5Yw0tNfytUaDXOQAEuFMQZ62uHM8XD069dPbxewVKlUCAsL0+scAMPIoyo5FPfkVgQLC9J7uqzlQUT6wc7ODkZGRkhOTi6xPTk5GY6OjmUe4+joqNP+AKBQKKBQKEpt13X1cu9mdujgYoN9ef9iUJ8Wev3HB3OQhuLbT3T9b1GKDCEHwDDyqEwOuuyvn6U8EREZNFNTU3h7e+PQoUPabRqNBocOHYKvr2+Zx/j6+pbYHyjq9i9vfyIiql7ssSAiIkmaNm0aRo8eDR8fH3Tp0gVLlixBTk6OdpaoUaNGwdnZGcHBwQCADz/8EL169cLChQsxePBgbN68GefOncPatWvFTIOIqM5gYUFERJI0fPhwpKamYvbs2UhKSkLHjh2xf/9+7QDthISEErP+dOvWDZs2bcKnn36KWbNmoWXLlti9ezfXsCAiqiUsLIiISLImT56MyZMnl/laeHh4qW0BAQEICAio4aiIiKgsHGNBRERERERVxsKCiIiIiIiqrM7dCiUIResZ6DInbzGVSoXc3FxkZWXp9XRjhpAHc5AOQ8jDEHIAqpZH8Xdi8XdkXVXX2wjmIB2GkIch5AAYRh611T7UucIiOzsbAODq6ipyJERE0pOdnQ0bGxuxwxAN2wgiorJVpH2QCXXs5ymNRoO7d++iXr16kMl0W2G5eEXWW7du6bQiq9QYQh7MQToMIQ9DyAGoWh6CICA7OxtOTk4lZlqqa+p6G8EcpMMQ8jCEHADDyKO22oc612Mhl8vh4uJSpXNYW1vr7X9YjzKEPJiDdBhCHoaQA1D5POpyT0UxthFFmIN0GEIehpADYBh51HT7UHd/liIiIiIiomrDwoKIiIiIiKqMhYUOFAoF5syZA4VCIXYoVWIIeTAH6TCEPAwhB8Bw8tBXhvD+MwfpMIQ8DCEHwDDyqK0c6tzgbSIiIiIiqn7ssSAiIiIioipjYUFERERERFXGwoKIiIiIiKqMhUUlvfTSS2jSpAnMzMzQuHFjjBw5Enfv3hU7LJ3Ex8fj7bffRrNmzWBubg53d3fMmTMHBQUFYoemky+//BLdunWDhYUF6tevL3Y4FbZixQo0bdoUZmZm6Nq1K86cOSN2SDo5evQoXnzxRTg5OUEmk2H37t1ih6Sz4OBgPPPMM6hXrx4aNWoEf39/xMbGih2WTlatWoUOHTpo5yb39fXFH3/8IXZYdZ6+txGG0j4A+tlGsH0QnyG0D0DttxEsLCqpT58+2Lp1K2JjY7Fjxw5cv34dw4YNEzssncTExECj0WDNmjW4fPkyFi9ejNWrV2PWrFlih6aTgoICBAQEYNKkSWKHUmFbtmzBtGnTMGfOHERGRsLLywt+fn5ISUkRO7QKy8nJgZeXF1asWCF2KJX2119/ITAwEKdOnUJYWBhUKhX69++PnJwcsUOrMBcXF3z11VeIiIjAuXPn8Pzzz+Pll1/G5cuXxQ6tTtP3NsJQ2gdA/9oItg/SYAjtAyBCGyFQtdizZ48gk8mEgoICsUOpkm+++UZo1qyZ2GFUyvr16wUbGxuxw6iQLl26CIGBgdrnarVacHJyEoKDg0WMqvIACLt27RI7jCpLSUkRAAh//fWX2KFUSYMGDYQffvhB7DDoEYbQRuhz+yAI+tNGsH2QJkNpHwShZtsI9lhUg7S0NGzcuBHdunWDiYmJ2OFUSWZmJmxtbcUOw6AVFBQgIiICffv21W6Ty+Xo27cvTp48KWJklJmZCQB6+29ArVZj8+bNyMnJga+vr9jh0EOG0kawfah5bB+kS9/bB6B22ggWFlXwySefwNLSEg0bNkRCQgL27NkjdkhVcu3aNSxbtgzvvvuu2KEYtHv37kGtVsPBwaHEdgcHByQlJYkUFWk0GkyZMgXdu3dHu3btxA5HJ5cuXYKVlRUUCgUmTpyIXbt2wdPTU+yw6jxDaiPYPtQOtg/SpM/tA1C7bQQLi0fMnDkTMpnsiY+YmBjt/jNmzMD58+dx4MABGBkZYdSoURAksN6grnkAwJ07dzBgwAAEBARgwoQJIkX+n8rkQFQVgYGBiI6OxubNm8UORWceHh6IiorC6dOnMWnSJIwePRpXrlwROyyDYwhthCG0DwDbCKpd+tw+ALXbRnDl7Uekpqbi/v37T9ynefPmMDU1LbX99u3bcHV1xYkTJ0S/BUHXPO7evYvevXvj2WefRUhICORy8evNynwWISEhmDJlCjIyMmo4uqopKCiAhYUFtm/fDn9/f+320aNHIyMjQy9/1ZTJZNi1a1eJfPTJ5MmTsWfPHhw9ehTNmjUTO5wq69u3L9zd3bFmzRqxQzEohtBGGEL7ABhuG8H2QXoMrX0AaraNMK72M+oxe3t72NvbV+pYjUYDAFAqldUZUqXoksedO3fQp08feHt7Y/369ZJpNKryWUidqakpvL29cejQIe0XrUajwaFDhzB58mRxg6tjBEHA+++/j127diE8PNxgGg2NRiOJ7yJDYwhthCG0D4DhthFsH6TDUNsHoGbbCBYWlXD69GmcPXsWzz33HBo0aIDr16/js88+g7u7u+i9Fbq4c+cOevfuDTc3N3z77bdITU3Vvubo6ChiZLpJSEhAWloaEhISoFarERUVBQBo0aIFrKysxA2uHNOmTcPo0aPh4+ODLl26YMmSJcjJycHYsWPFDq3CHjx4gGvXrmmfx8XFISoqCra2tmjSpImIkVVcYGAgNm3ahD179qBevXrae5htbGxgbm4ucnQVExQUhIEDB6JJkybIzs7Gpk2bEB4ejj///FPs0OosQ2gjDKV9APSvjWD7IA2G0D4AIrQRNTLXlIG7ePGi0KdPH8HW1lZQKBRC06ZNhYkTJwq3b98WOzSdrF+/XgBQ5kOfjB49uswcjhw5InZoT7Rs2TKhSZMmgqmpqdClSxfh1KlTYoekkyNHjpT5vo8ePVrs0CqsvP/+169fL3ZoFTZu3DjBzc1NMDU1Fezt7YUXXnhBOHDggNhh1WmG0EYYSvsgCPrZRrB9EJ8htA+CUPttBMdYEBERERFRlUnnhkkiIiIiItJbLCyIiIiIiKjKWFgQEREREVGVsbAgIiIiIqIqY2FBRERERERVxsKCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRlbGwIKplqampcHR0xPz587XbTpw4AVNTUxw6dEjEyIiISExsH0jfyQRBEMQOgqiu2bdvH/z9/XHixAl4eHigY8eOePnll7Fo0SKxQyMiIhGxfSB9xsKCSCSBgYE4ePAgfHx8cOnSJZw9exYKhULssIiISGRsH0hfsbAgEkleXh7atWuHW7duISIiAu3btxc7JCIikgC2D6SvOMaCSCTXr1/H3bt3odFoEB8fL3Y4REQkEWwfSF+xx4JIBAUFBejSpQs6duwIDw8PLFmyBJcuXUKjRo3EDo2IiETE9oH0GQsLIhHMmDED27dvx4ULF2BlZYVevXrBxsYGe/fuFTs0IiISEdsH0me8FYqoloWHh2PJkiUIDQ2FtbU15HI5QkNDcezYMaxatUrs8IiISCRsH0jfsceCiIiIiIiqjD0WRERERERUZSwsiIiIiIioylhYEBERERFRlbGwICIiIiKiKmNhQUREREREVcbCgoiIiIiIqoyFBRERERERVRkLCyIiIiIiqjIWFkREREREVGUsLIiIiIiIqMpYWBARERERUZWxsCAiIiIioir7f//TFqXkmtRtAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "gelu, relu = GELU(), nn.ReLU()#先把函数给个小名\n", "\n", "# Some sample data\n", "x = torch.linspace(-3, 3, 100) #初定义一个张量\n", "y_gelu, y_relu = gelu(x), relu(x) #两种激活函数\n", "\n", "plt.figure(figsize=(8, 3))\n", "for i, (y, label) in enumerate(zip([y_gelu, y_relu], [\"GELU\", \"ReLU\"]), 1):\n", " plt.subplot(1, 2, i)\n", " plt.plot(x, y)\n", " plt.title(f\"{label} activation function\")\n", " plt.xlabel(\"x\")\n", " plt.ylabel(f\"{label}(x)\")\n", " plt.grid(True)\n", "\n", "plt.tight_layout()\n", "plt.show()\n", "#一个经典的作图" ] }, { "cell_type": "markdown", "id": "1cd01662-14cb-43fd-bffd-2d702813de2d", "metadata": {}, "source": [ "- 正如我们所见，ReLU 是一种分段线性函数：对于正值直接输出输入值；对于负值则输出零。\n", "- GELU 是一种平滑的非线性函数，它近似 ReLU，但在负值时具有非零梯度（除了大约在 -0.75 处）。\n", "\n", "- 接下来，我们将实现一个小型神经网络模块 `FeedForward`，该模块将用于 LLM 的 Transformer 块中：" ] }, { "cell_type": "code", "execution_count": 15, "id": "9275c879-b148-4579-a107-86827ca14d4d", "metadata": {}, "outputs": [], "source": [ "class FeedForward(nn.Module):\n", " def __init__(self, cfg):\n", " super().__init__()\n", " self.layers = nn.Sequential(\n", " nn.Linear(cfg[\"emb_dim\"], 4 * cfg[\"emb_dim\"]),\n", " GELU(),\n", " nn.Linear(4 * cfg[\"emb_dim\"], cfg[\"emb_dim\"]),\n", " )\n", " #运行一次就线性两次激活一次\n", " def forward(self, x):\n", " return self.layers(x)" ] }, { "cell_type": "code", "execution_count": 16, "id": "7c4976e2-0261-418e-b042-c5be98c2ccaf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "768\n" ] } ], "source": [ "print(GPT_CONFIG_124M[\"emb_dim\"])" ] }, { "cell_type": "markdown", "id": "fdcaacfa-3cfc-4c9e-b668-b71a2753145a", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": 17, "id": "928e7f7c-d0b1-499f-8d07-4cadb428a6f9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "torch.Size([2, 3, 768])\n" ] } ], "source": [ "ffn = FeedForward(GPT_CONFIG_124M)\n", "\n", "# input shape: [batch_size, num_token, emb_size]\n", "x = torch.rand(2, 3, 768) \n", "out = ffn(x)\n", "print(out.shape)" ] }, { "cell_type": "markdown", "id": "8f8756c5-6b04-443b-93d0-e555a316c377", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "e5da2a50-04f4-4388-af23-ad32e405a972", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "4ffcb905-53c7-4886-87d2-4464c5fecf89", "metadata": {}, "source": [ "## 4.4 类似于ResNet的shortcut传递" ] }, { "cell_type": "markdown", "id": "ffae416c-821e-4bfa-a741-8af4ba5db00e", "metadata": {}, "source": [ "- 接下来，我们来讨论**快捷连接**（shortcut connections）的概念，也称为**跳跃连接**（skip connections）或**残差连接**（residual connections）。\n", "- 残差连接最初是在深度网络中提出的，主要应用于计算机视觉中的残差网络（ResNet），以缓解梯度消失问题。\n", "- 残差连接通过为梯度提供一条更短的替代路径，使其能够更顺畅地通过网络流动。\n", "- 具体实现是将某一层的输出与后续某一层的输出相加，通常会跳过中间的一层或多层。\n", "- 以下是一个小型网络的示例来说明这一概念：\n", "\n", "![残差连接示例](https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/12.webp?123)" ] }, { "cell_type": "markdown", "id": "14cfd241-a32e-4601-8790-784b82f2f23e", "metadata": {}, "source": [ "- 代码形式长成这样" ] }, { "cell_type": "code", "execution_count": 18, "id": "05473938-799c-49fd-86d4-8ed65f94fee6", "metadata": {}, "outputs": [], "source": [ "import torch\n", "import torch.nn as nn\n", "\n", "class ExampleDeepNeuralNetwork(nn.Module):\n", " def __init__(self, layer_sizes, use_shortcut):\n", " super().__init__()\n", " self.use_shortcut = use_shortcut\n", " # 定义多层网络，包含 5 层线性层和激活函数 GELU\n", " self.layers = nn.ModuleList([\n", " nn.Sequential(nn.Linear(layer_sizes[0], layer_sizes[1]), GELU()),\n", " nn.Sequential(nn.Linear(layer_sizes[1], layer_sizes[2]), GELU()),\n", " nn.Sequential(nn.Linear(layer_sizes[2], layer_sizes[3]), GELU()),\n", " nn.Sequential(nn.Linear(layer_sizes[3], layer_sizes[4]), GELU()),\n", " nn.Sequential(nn.Linear(layer_sizes[4], layer_sizes[5]), GELU())\n", " ])\n", " # 定义一个五层的神经网络块，其中每层包含一个线性变换和一个激活函数 GELU，\n", " # 类似 ResNet 的结构，支持添加残差连接。\n", "\n", " def forward(self, x):\n", " # 遍历每一层\n", " for layer in self.layers:\n", " # 当前层的输出\n", " layer_output = layer(x)\n", " # 检查是否可以应用残差连接\n", " if self.use_shortcut and x.shape == layer_output.shape:\n", " x = x + layer_output # 如果输入和输出维度匹配，添加残差连接\n", " else:\n", " x = layer_output # 否则直接输出当前层结果\n", " return x # 返回最终结果\n", "\n", "\n", "def print_gradients(model, x):\n", " # 前向传播\n", " output = model(x)\n", " target = torch.tensor([[0.]]) # 定义目标值\n", " # 计算损失，使用均方误差损失函数\n", " loss = nn.MSELoss()\n", " loss = loss(output, target)\n", " \n", " # 反向传播，计算梯度\n", " loss.backward()\n", "\n", " # 打印每层权重的梯度均值\n", " for name, param in model.named_parameters():\n", " if 'weight' in name:\n", " print(f\"{name} has gradient mean of {param.grad.abs().mean().item()}\")" ] }, { "cell_type": "markdown", "id": "b39bf277-b3db-4bb1-84ce-7a20caff1011", "metadata": {}, "source": [ "- 在没有残差链接的时候,梯度是这样的" ] }, { "cell_type": "code", "execution_count": 19, "id": "c75f43cc-6923-4018-b980-26023086572c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "layers.0.0.weight has gradient mean of 0.00020173587836325169\n", "layers.1.0.weight has gradient mean of 0.00012011159560643137\n", "layers.2.0.weight has gradient mean of 0.0007152039906941354\n", "layers.3.0.weight has gradient mean of 0.0013988736318424344\n", "layers.4.0.weight has gradient mean of 0.005049645435065031\n" ] } ], "source": [ "layer_sizes = [3, 3, 3, 3, 3, 1] \n", "\n", "sample_input = torch.tensor([[1., 0., -1.]])\n", "\n", "torch.manual_seed(123)\n", "model_without_shortcut = ExampleDeepNeuralNetwork(\n", " layer_sizes, use_shortcut=False\n", ")\n", "print_gradients(model_without_shortcut, sample_input)\n", "#一次一次输出梯度" ] }, { "cell_type": "markdown", "id": "837fd5d4-7345-4663-97f5-38f19dfde621", "metadata": {}, "source": [ "- **有** 残差的链接,梯度是这样的" ] }, { "cell_type": "code", "execution_count": 20, "id": "11b7c0c2-f9dd-4dd5-b096-a05c48c5f6d6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "layers.0.0.weight has gradient mean of 0.22169792652130127\n", "layers.1.0.weight has gradient mean of 0.20694106817245483\n", "layers.2.0.weight has gradient mean of 0.32896995544433594\n", "layers.3.0.weight has gradient mean of 0.2665732204914093\n", "layers.4.0.weight has gradient mean of 1.3258540630340576\n" ] } ], "source": [ "torch.manual_seed(123)\n", "model_with_shortcut = ExampleDeepNeuralNetwork(\n", " layer_sizes, use_shortcut=True\n", ")\n", "print_gradients(model_with_shortcut, sample_input)\n", "#引入了残差链接,发现梯度消失的缺点明显改善了" ] }, { "cell_type": "markdown", "id": "79ff783a-46f0-49c5-a7a9-26a525764b6e", "metadata": {}, "source": [ "- 正如我们从上述输出中看到的，残差连接有效地防止了梯度在早期层（靠近 `layer.0`）中消失。\n", "- 接下来，在实现 Transformer 块时，我们将使用这一残差连接的概念。" ] }, { "cell_type": "markdown", "id": "cae578ca-e564-42cf-8635-a2267047cdff", "metadata": {}, "source": [ "## 4.5 在 Transformer 块中连接attention层与线性层" ] }, { "cell_type": "markdown", "id": "a3daac6f-6545-4258-8f2d-f45a7394f429", "metadata": {}, "source": [ "- 在本节中，我们将把前面介绍的概念整合成一个所谓的 Transformer 块。\n", "- Transformer 块将上一章中的因果多头注意力模块与线性层以及我们之前实现的前馈神经网络相结合。\n", "- 此外，Transformer 块还包含 Dropout 和残差连接的机制。" ] }, { "cell_type": "code", "execution_count": 21, "id": "0e1e8176-e5e3-4152-b1aa-0bbd7891dfd9", "metadata": {}, "outputs": [], "source": [ "from previous_chapters import MultiHeadAttention\n", "\n", "\n", "class TransformerBlock(nn.Module):\n", " def __init__(self, cfg):\n", " super().__init__()\n", " self.att = MultiHeadAttention(\n", " d_in=cfg[\"emb_dim\"], # 输入特征维度\n", " d_out=cfg[\"emb_dim\"], # 输出特征维度\n", " context_length=cfg[\"context_length\"], # 上下文长度\n", " num_heads=cfg[\"n_heads\"], # 注意力头的数量\n", " dropout=cfg[\"drop_rate\"], # Dropout 比例\n", " qkv_bias=cfg[\"qkv_bias\"] # 查询、键和值的偏置\n", " ) # 多头注意力模块，结合各种参数\n", " self.ff = FeedForward(cfg) # 前馈神经网络模块\n", " self.norm1 = LayerNorm(cfg[\"emb_dim\"]) # 第一归一化层\n", " self.norm2 = LayerNorm(cfg[\"emb_dim\"]) # 第二归一化层\n", " self.drop_shortcut = nn.Dropout(cfg[\"drop_rate\"]) # 残差连接的 Dropout\n", "\n", " def forward(self, x):\n", " # 对注意力模块的快捷连接\n", " shortcut = x\n", " x = self.norm1(x) # 应用第一归一化层\n", " x = self.att(x) # 通过多头注意力模块，形状为 [batch_size, num_tokens, emb_size]\n", " x = self.drop_shortcut(x) # 应用 Dropout\n", " x = x + shortcut # 将原始输入加回，实现残差连接\n", "\n", " # 对前馈网络模块的残差连接\n", " shortcut = x\n", " x = self.norm2(x) # 应用第二归一化层\n", " x = self.ff(x) # 通过前馈神经网络模块\n", " x = self.drop_shortcut(x) # 应用 Dropout\n", " x = x + shortcut # 将原始输入加回，实现残差连接\n", "\n", " return x" ] }, { "cell_type": "markdown", "id": "36b64d16-94a6-4d13-8c85-9494c50478a9", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "54d2d375-87bd-4153-9040-63a1e6a2b7cb", "metadata": {}, "source": [ "- 假设我们有 2 个输入样本，每个样本包含 6 个标记，每个标记是一个 768 维的嵌入向量；然后，这个 Transformer 块会先应用自注意力机制，再通过线性层处理，生成一个相同尺寸的输出。\n", "- 你可以将输出视为我们在上一章讨论的上下文向量的增强版本。" ] }, { "cell_type": "code", "execution_count": 22, "id": "3fb45a63-b1f3-4b08-b525-dafbc8228405", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input shape: torch.Size([2, 4, 768])\n", "Output shape: torch.Size([2, 4, 768])\n" ] } ], "source": [ "torch.manual_seed(123)\n", "\n", "x = torch.rand(2, 4, 768) # Shape: [batch_size, num_tokens, emb_dim]\n", "block = TransformerBlock(GPT_CONFIG_124M)\n", "output = block(x)\n", "\n", "print(\"Input shape:\", x.shape)\n", "print(\"Output shape:\", output.shape)\n", "#经典的一系列操作" ] }, { "cell_type": "markdown", "id": "8f9e4ee4-cf23-4583-b1fd-317abb4fcd13", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "46618527-15ac-4c32-ad85-6cfea83e006e", "metadata": {}, "source": [ "## 4.6 编码GPT" ] }, { "cell_type": "markdown", "id": "dec7d03d-9ff3-4ca3-ad67-01b67c2f5457", "metadata": {}, "source": [ "- 终于要结束了：现在让我们将 Transformer 块插入到本章开头编写的架构中，这样就可以得到一个可用的 GPT 架构。\n", "- 请注意，Transformer 块会重复多次；对于最小的 124M GPT-2 模型来说，我们会重复 12 次。" ] }, { "cell_type": "markdown", "id": "9b7b362d-f8c5-48d2-8ebd-722480ac5073", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "324e4b5d-ed89-4fdf-9a52-67deee0593bc", "metadata": {}, "source": [ "- 对应的代码实现，其中 `cfg[\"n_layers\"] = 12`：" ] }, { "cell_type": "markdown", "id": "64e4e972-a13b-4733-9cf4-f54c882870cf", "metadata": {}, "source": [ "- (译者)一点语法小知识\n", "在 Python 中，* 是 解包操作符，用于将一个可迭代对象（如列表、元组）中的元素逐一解包成单独的参数。\n", "在 PyTorch 中，nn.Sequential 接受一组模块作为输入，而不是一个列表或其他容器。\n", "这里使用 * 将生成的 TransformerBlock 列表解包为独立的参数传递给 nn.Sequential。" ] }, { "cell_type": "code", "execution_count": 23, "id": "c61de39c-d03c-4a32-8b57-f49ac3834857", "metadata": {}, "outputs": [], "source": [ "class GPTModel(nn.Module):#召唤GPT!\n", " def __init__(self, cfg):\n", " super().__init__()\n", " self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n", " self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n", " self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n", " #新建字典、位置信息、还有dropout的比率设置\n", " self.trf_blocks = nn.Sequential(\n", " *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n", " #解包操作\n", "\n", " self.trf_blocks = nn.Sequential(\n", " TransformerBlock(cfg),\n", " TransformerBlock(cfg),\n", " TransformerBlock(cfg)\n", " )\n", " self.final_norm = LayerNorm(cfg[\"emb_dim\"])\n", " #归一化\n", " self.out_head = nn.Linear(\n", " cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n", " )\n", " #输出头保证维度\n", " def forward(self, in_idx):\n", " batch_size, seq_len = in_idx.shape\n", " tok_embeds = self.tok_emb(in_idx)\n", " pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n", " x = tok_embeds + pos_embeds # Shape [batch_size, num_tokens, emb_size]\n", " x = self.drop_emb(x)\n", " x = self.trf_blocks(x)\n", " x = self.final_norm(x)\n", " logits = self.out_head(x)\n", " return logits" ] }, { "cell_type": "markdown", "id": "2750270f-c45d-4410-8767-a6adbd05d5c3", "metadata": {}, "source": [ "- 用了124M模型的原始参数,我们接下啦要初始化模型参数" ] }, { "cell_type": "code", "execution_count": 24, "id": "ef94fd9c-4e9d-470d-8f8e-dd23d1bb1f64", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input batch:\n", " tensor([[6109, 3626, 6100, 345],\n", " [6109, 1110, 6622, 257]])\n", "\n", "Output shape: torch.Size([2, 4, 50257])\n", "tensor([[[ 0.3613, 0.4222, -0.0711, ..., 0.3483, 0.4661, -0.2838],\n", " [-0.1792, -0.5660, -0.9485, ..., 0.0477, 0.5181, -0.3168],\n", " [ 0.7120, 0.0332, 0.1085, ..., 0.1018, -0.4327, -0.2553],\n", " [-1.0076, 0.3418, -0.1190, ..., 0.7195, 0.4023, 0.0532]],\n", "\n", " [[-0.2564, 0.0900, 0.0335, ..., 0.2659, 0.4454, -0.6806],\n", " [ 0.1230, 0.3653, -0.2074, ..., 0.7705, 0.2710, 0.2246],\n", " [ 1.0558, 1.0318, -0.2800, ..., 0.6936, 0.3205, -0.3178],\n", " [-0.1565, 0.3926, 0.3288, ..., 1.2630, -0.1858, 0.0388]]],\n", " grad_fn=)\n" ] } ], "source": [ "torch.manual_seed(123)\n", "model = GPTModel(GPT_CONFIG_124M)\n", "\n", "out = model(batch)\n", "print(\"Input batch:\\n\", batch)\n", "print(\"\\nOutput shape:\", out.shape)\n", "print(out)\n", "#经典操作" ] }, { "cell_type": "markdown", "id": "6d616e7a-568b-4921-af29-bd3f4683cd2e", "metadata": {}, "source": [ "- 我们将在下一章训练这个模型。\n", "- 但关于其规模需要补充一点：我们之前提到它是一个拥有 1.24 亿参数的模型；我们可以通过以下方式再次确认这个数字：" ] }, { "cell_type": "code", "execution_count": 25, "id": "84fb8be4-9d3b-402b-b3da-86b663aac33a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of parameters: 163,009,536\n" ] } ], "source": [ "total_params = sum(p.numel() for p in model.parameters())\n", "#模型的总参数数量\n", "print(f\"Total number of parameters: {total_params:,}\")" ] }, { "cell_type": "markdown", "id": "b67d13dd-dd01-4ba6-a2ad-31ca8a9fd660", "metadata": {}, "source": [ "- 正如我们上面看到的，这个模型实际上有 1.63 亿参数，而不是 1.24 亿；这是为什么呢？\n", "- 在原始 GPT-2 论文中，研究人员采用了**权重共享**（weight tying）技术，即将标记嵌入层（`tok_emb`）作为输出层复用，具体表现为设置 `self.out_head.weight = self.tok_emb.weight`。\n", "- 标记嵌入层将 50,257 维的独热编码输入标记映射到 768 维的嵌入表示。\n", "- 输出层则将 768 维的嵌入表示映射回 50,257 维的表示，从而可以将其还原为单词（关于这一点，我们将在下一节详细讨论）。\n", "- 因此，嵌入层和输出层的权重参数数量相同，从它们权重矩阵的形状可以看出这一点。" ] }, { "cell_type": "code", "execution_count": 26, "id": "e3b43233-e9b8-4f5a-b72b-a263ec686982", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Token embedding layer shape: torch.Size([50257, 768])\n", "Output layer shape: torch.Size([50257, 768])\n" ] } ], "source": [ "print(\"Token embedding layer shape:\", model.tok_emb.weight.shape)\n", "print(\"Output layer shape:\", model.out_head.weight.shape)\n", "#输出格式让我们更好地理解" ] }, { "cell_type": "markdown", "id": "f02259f6-6f79-4c89-a866-4ebeae1c3289", "metadata": {}, "source": [ "- 在原始 GPT-2 论文中，研究人员将标记嵌入矩阵复用为输出矩阵。\n", "- 相应地，如果我们减去输出层的参数数量，就会得到一个拥有 1.24 亿参数的模型：" ] }, { "cell_type": "code", "execution_count": 27, "id": "95a22e02-50d3-48b3-a4e0-d9863343c164", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of trainable parameters considering weight tying: 124,412,160\n" ] } ], "source": [ "total_params_gpt2 = total_params - sum(p.numel() for p in model.out_head.parameters())\n", "print(f\"Number of trainable parameters considering weight tying: {total_params_gpt2:,}\")\n", "#Parameter- sharing" ] }, { "cell_type": "markdown", "id": "40b03f80-b94c-46e7-9d42-d0df399ff3db", "metadata": {}, "source": [ "- 在实际应用中，因为不使用权重共享更容易训练模型，所以这里没有进行权重共享。\n", "- 不过，在第 5 章加载预训练权重时，我们将重新讨论并应用权重共享的概念。\n", "- 最后，我们可以通过以下方式计算模型的内存需求，这可能是一个有用的参考点：" ] }, { "cell_type": "code", "execution_count": 28, "id": "5131a752-fab8-4d70-a600-e29870b33528", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total size of the model: 621.83 MB\n" ] } ], "source": [ "# Calculate the total size in bytes (assuming float32, 4 bytes per parameter)\n", "total_size_bytes = total_params * 4\n", "\n", "# Convert to megabytes\n", "total_size_mb = total_size_bytes / (1024 * 1024)\n", "\n", "print(f\"Total size of the model: {total_size_mb:.2f} MB\")\n", "#计算总的容量" ] }, { "cell_type": "markdown", "id": "309a3be4-c20a-4657-b4e0-77c97510b47c", "metadata": {}, "source": [ "- **练习**：你可以尝试以下其他配置，这些配置参考自 [GPT-2 论文](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=dOad5HoAAAAJ&citation_for_view=dOad5HoAAAAJ:YsMSGLbcyi4C)。\n", "\n", " - **GPT2-small**（我们已经实现的 124M 配置）：\n", " - `\"emb_dim\" = 768`\n", " - `\"n_layers\" = 12`\n", " - `\"n_heads\" = 12`\n", "\n", " - **GPT2-medium**：\n", " - `\"emb_dim\" = 1024`\n", " - `\"n_layers\" = 24`\n", " - `\"n_heads\" = 16`\n", "\n", " - **GPT2-large**：\n", " - `\"emb_dim\" = 1280`\n", " - `\"n_layers\" = 36`\n", " - `\"n_heads\" = 20`\n", "\n", " - **GPT2-XL**：\n", " - `\"emb_dim\" = 1600`\n", " - `\"n_layers\" = 48`\n", " - `\"n_heads\" = 25`" ] }, { "cell_type": "markdown", "id": "da5d9bc0-95ab-45d4-9378-417628d86e35", "metadata": {}, "source": [ "## 4.7 文本生成" ] }, { "cell_type": "markdown", "id": "48da5deb-6ee0-4b9b-8dd2-abed7ed65172", "metadata": {}, "source": [ "- GPT架构的LLM一次只能生成一个单词" ] }, { "cell_type": "markdown", "id": "caade12a-fe97-480f-939c-87d24044edff", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "a7061524-a3bd-4803-ade6-2e3b7b79ac13", "metadata": {}, "source": [ "- 以下的 `generate_text_simple` 函数实现了贪心解码（greedy decoding），这是一种简单且快速的文本生成方法。\n", "- 在贪心解码中，模型在每一步选择具有最高概率的词（或标记）作为下一个输出（由于最高的 logit 值对应最高的概率，实际上我们不需要显式地计算 softmax 函数）。\n", "- 在下一章，我们将实现一个更为复杂的 `generate_text` 函数。\n", "- 下图展示了给定输入上下文时，GPT 模型是如何生成下一个词标记的。" ] }, { "cell_type": "markdown", "id": "7ee0f32c-c18c-445e-b294-a879de2aa187", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": 29, "id": "c9b428a9-8764-4b36-80cd-7d4e00595ba6", "metadata": {}, "outputs": [], "source": [ "def generate_text_simple(model, idx, max_new_tokens, context_size):\n", " # 预测单词的模块\n", " # idx 是当前上下文中的（batch, n_tokens）索引数组\n", " for _ in range(max_new_tokens):\n", " # 每次生成一个单词后，重新将其加入序列中\n", " # 如果当前上下文长度超过模型支持的最大上下文长度，则截取\n", " # 例如，如果LLM只支持5个token，而上下文长度为10\n", " # 那么只使用最后5个token作为上下文\n", " idx_cond = idx[:, -context_size:]\n", " # 如果idx的长度超过模型支持的上下文长度size，只保留最后size个token\n", " # 避免溢出\n", " # 获取预测结果\n", " with torch.no_grad(): # 在推理阶段，不需要计算梯度，因为没有反向传播\n", " # 这样可以减少存储开销\n", " logits = model(idx_cond)\n", " # 模型输出结果\n", " # 只关注最后一个时间步的输出\n", " # (batch, n_tokens, vocab_size) 变为 (batch, vocab_size)\n", " logits = logits[:, -1, :]\n", " # 关注最后一个时间步\n", " # 使用softmax函数计算概率\n", " probas = torch.softmax(logits, dim=-1) # (batch, vocab_size)\n", " # 归一化\n", " # 获取具有最高概率值的词汇索引\n", " idx_next = torch.argmax(probas, dim=-1, keepdim=True) # (batch, 1)\n", " # 获取概率最高的词汇索引\n", " # 将采样的索引添加到序列中\n", " idx = torch.cat((idx, idx_next), dim=1) # (batch, n_tokens+1)\n", "\n", " return idx" ] }, { "cell_type": "markdown", "id": "6515f2c1-3cc7-421c-8d58-cc2f563b7030", "metadata": {}, "source": [ "- 上面的 `generate_text_simple` 实现了一个迭代过程，其中它一次生成一个token。\n", "\n", "" ] }, { "cell_type": "markdown", "id": "f682eac4-f9bd-438b-9dec-6b1cc7bc05ce", "metadata": {}, "source": [ "- 举个例子" ] }, { "cell_type": "code", "execution_count": 30, "id": "3d7e3e94-df0f-4c0f-a6a1-423f500ac1d3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "encoded: [15496, 11, 314, 716]\n", "encoded_tensor.shape: torch.Size([1, 4])\n" ] } ], "source": [ "start_context = \"Hello, I am\"\n", "#模拟\n", "encoded = tokenizer.encode(start_context)\n", "print(\"encoded:\", encoded)\n", "#进行语义理解\n", "encoded_tensor = torch.tensor(encoded).unsqueeze(0)\n", "print(\"encoded_tensor.shape:\", encoded_tensor.shape)\n", "#最终输出格式" ] }, { "cell_type": "code", "execution_count": 31, "id": "a72a9b60-de66-44cf-b2f9-1e638934ada4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Output: tensor([[15496, 11, 314, 716, 27018, 24086, 47843, 30961, 42348, 7267]])\n", "Output length: 10\n" ] } ], "source": [ "model.eval() # disable dropout\n", "#在检验的时候不需要正则化了\n", "out = generate_text_simple(\n", " model=model,\n", " #左边的参数名字,右边是函数传入的实际模型\n", " idx=encoded_tensor, #上下文的索引\n", " max_new_tokens=6, #最多运行六次,然后取结果概率最高的\n", " #初始文本➕6\n", " context_size=GPT_CONFIG_124M[\"context_length\"]\n", ")\n", "\n", "print(\"Output:\", out)\n", "print(\"Output length:\", len(out[0]))\n", "#输出长度还有每个单词的id" ] }, { "cell_type": "markdown", "id": "1d131c00-1787-44ba-bec3-7c145497b2c3", "metadata": {}, "source": [ "- 去除批次维度并将其转换回文本" ] }, { "cell_type": "code", "execution_count": 32, "id": "053d99f6-5710-4446-8d52-117fb34ea9f6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hello, I am Featureiman Byeswickattribute argue\n" ] } ], "source": [ "decoded_text = tokenizer.decode(out.squeeze(0).tolist())\n", "print(decoded_text)" ] }, { "cell_type": "markdown", "id": "9a894003-51f6-4ccc-996f-3b9c7d5a1d70", "metadata": {}, "source": [ "- 请注意，目前模型尚未经过训练，因此上述输出文本是随机的。\n", "- 我们将在下一章中训练模型。" ] }, { "cell_type": "markdown", "id": "a35278b6-9e5c-480f-83e5-011a1173648f", "metadata": {}, "source": [ "## 总结与收获\n", "\n", "- 请查看 [./gpt.py](./gpt.py) 脚本，这是一个独立的脚本，包含了我们在此 Jupyter Notebook 中实现的 GPT 模型。\n", "- 练习题的解答可以在 [./exercise-solutions.ipynb](./exercise-solutions.ipynb) 中找到。" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 5 }