{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPQMeBDjGHl7iQPy5HRtPtr"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# 推理中常见并行策略原理\n",
        "\n",
        "\n",
        "演示常见的并行策略原理\n",
        "\n",
        "\n",
        "相关文章：[大模型推理并行策略(DP/TP/PP/SP/EP)原理简介](https://zhuanlan.zhihu.com/p/2003423046342554380)\n",
        "\n",
        "Author: kaiyuan\n",
        "\n",
        "Email: kyxie@zju.edu.cn"
      ],
      "metadata": {
        "id": "CDCxkitt6PFo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1 DP(Data Parellel)策略\n",
        "\n",
        "思路：\n",
        "\n",
        "场景1：一个模型副本，我们用一个线程来运行这个模型，然后有4个数据任务，我们用一个线程池（4个线程）来同时发送数据给这个模型，但是模型处理是串行的，所以我们可以在模型内部加锁，使得同时只能有一个线程（即一个数据）被处理。\n",
        "\n",
        "场景2：四个模型副本，每个模型副本在一个线程中，然后有4个数据，我们同样用4个线程来发送数据，但是每个数据发送给不同的模型副本，这样就能并行处理。\n"
      ],
      "metadata": {
        "id": "JWEYZYClL1KT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import threading\n",
        "import time\n",
        "from queue import Queue\n",
        "import concurrent.futures\n",
        "from typing import List\n",
        "import random\n",
        "\n",
        "class FakeModel:\n",
        "    def __init__(self, model_id: int):\n",
        "        self.model_id = model_id\n",
        "        self.lock = threading.Lock()\n",
        "\n",
        "    def process(self, data: str) -> str:\n",
        "        \"\"\"模拟模型处理数据的过程\"\"\"\n",
        "        # 模拟处理时间\n",
        "        processing_time = random.uniform(0.1, 0.5)  # 随机处理时间\n",
        "        time.sleep(processing_time)\n",
        "\n",
        "        # 打印处理信息\n",
        "        with self.lock:\n",
        "            result = f\"模型副本{self.model_id} 接收数据: '{data}', 已处理 (耗时: {processing_time:.3f}s)\"\n",
        "            print(result)\n",
        "\n",
        "        return result\n",
        "\n",
        "def single_model_scenario(data_list: List[str]):\n",
        "    \"\"\"场景1: 单个模型副本处理所有数据\"\"\"\n",
        "    print(\"\\n\" + \"=\"*60)\n",
        "    print(\"场景1: 单个模型副本处理4条数据\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    model = FakeModel(1)\n",
        "\n",
        "    start_time = time.time()\n",
        "\n",
        "    # 串行处理\n",
        "    results = []\n",
        "    for data in data_list:\n",
        "        results.append(model.process(data))\n",
        "\n",
        "    end_time = time.time()\n",
        "    print(f\"\\n总耗时: {end_time - start_time:.3f}秒\")\n",
        "    return results, end_time - start_time\n",
        "\n",
        "def multi_model_scenario(data_list: List[str], num_models: int = 4):\n",
        "    \"\"\"场景2: 多个模型副本并行处理数据\"\"\"\n",
        "    print(\"\\n\" + \"=\"*60)\n",
        "    print(f\"场景2: {num_models}个模型副本并行处理4条数据\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    # 创建多个模型副本\n",
        "    models = [FakeModel(i+1) for i in range(num_models)]\n",
        "\n",
        "    start_time = time.time()\n",
        "\n",
        "    # 使用线程池并行处理\n",
        "    results = []\n",
        "    with concurrent.futures.ThreadPoolExecutor(max_workers=num_models) as executor:\n",
        "        # 为每个数据分配一个模型副本\n",
        "        future_to_data = {}\n",
        "        for i, data in enumerate(data_list):\n",
        "            model_idx = i % num_models  # 简单的数据分配策略\n",
        "            future = executor.submit(models[model_idx].process, data)\n",
        "            future_to_data[future] = data\n",
        "\n",
        "        # 收集结果\n",
        "        for future in concurrent.futures.as_completed(future_to_data):\n",
        "            results.append(future.result())\n",
        "\n",
        "    end_time = time.time()\n",
        "    print(f\"\\n总耗时: {end_time - start_time:.3f}秒\")\n",
        "    return results, end_time - start_time\n",
        "\n",
        "def data_parallel_simulation():\n",
        "    \"\"\"主模拟函数\"\"\"\n",
        "    print(\"数据并行(DP)策略模拟演示\")\n",
        "    print(\"-\" * 60)\n",
        "\n",
        "    # 模拟4条数据\n",
        "    data_list = [\n",
        "        \"数据1: 图像分类任务\",\n",
        "        \"数据2: 自然语言处理\",\n",
        "        \"数据3: 语音识别样本\",\n",
        "        \"数据4: 视频分析帧\"\n",
        "    ]\n",
        "\n",
        "    print(\"待处理数据:\")\n",
        "    for i, data in enumerate(data_list, 1):\n",
        "        print(f\"  数据{i}: {data}\")\n",
        "\n",
        "    # 场景1: 单个模型副本\n",
        "    print(\"\\n\" + \"=\"*60)\n",
        "    print(\"开始模拟: 单个模型副本 vs 多个模型副本\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    # 重置随机种子以确保公平比较\n",
        "    random.seed(42)\n",
        "    results1, time1 = single_model_scenario(data_list.copy())\n",
        "\n",
        "    # 场景2: 多个模型副本\n",
        "    random.seed(42)  # 重置随机种子\n",
        "    results2, time2 = multi_model_scenario(data_list.copy(), num_models=4)\n",
        "\n",
        "    # 性能对比\n",
        "    print(\"\\n\" + \"=\"*60)\n",
        "    print(\"性能对比总结\")\n",
        "    print(\"=\"*60)\n",
        "    print(f\"单个模型副本总耗时: {time1:.3f}秒\")\n",
        "    print(f\"4个模型副本总耗时:  {time2:.3f}秒\")\n",
        "    print(f\"加速比: {time1/time2:.2f}x\")\n",
        "\n",
        "    if time1 > time2:\n",
        "        print(f\"性能提升: {(time1 - time2)/time1*100:.1f}%\")\n",
        "    else:\n",
        "        print(\"注意: 由于线程开销，加速效果可能不明显\")\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    data_parallel_simulation()\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GzLtGUvbL1T1",
        "outputId": "824562a6-3f97-4447-a815-a7636bdf486e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "数据并行(DP)策略模拟演示\n",
            "------------------------------------------------------------\n",
            "待处理数据:\n",
            "  数据1: 数据1: 图像分类任务\n",
            "  数据2: 数据2: 自然语言处理\n",
            "  数据3: 数据3: 语音识别样本\n",
            "  数据4: 数据4: 视频分析帧\n",
            "\n",
            "============================================================\n",
            "开始模拟: 单个模型副本 vs 多个模型副本\n",
            "============================================================\n",
            "\n",
            "============================================================\n",
            "场景1: 单个模型副本处理4条数据\n",
            "============================================================\n",
            "模型副本1 接收数据: '数据1: 图像分类任务', 已处理 (耗时: 0.356s)\n",
            "模型副本1 接收数据: '数据2: 自然语言处理', 已处理 (耗时: 0.110s)\n",
            "模型副本1 接收数据: '数据3: 语音识别样本', 已处理 (耗时: 0.210s)\n",
            "模型副本1 接收数据: '数据4: 视频分析帧', 已处理 (耗时: 0.189s)\n",
            "\n",
            "总耗时: 0.867秒\n",
            "\n",
            "============================================================\n",
            "场景2: 4个模型副本并行处理4条数据\n",
            "============================================================\n",
            "模型副本2 接收数据: '数据2: 自然语言处理', 已处理 (耗时: 0.110s)\n",
            "模型副本4 接收数据: '数据4: 视频分析帧', 已处理 (耗时: 0.189s)\n",
            "模型副本3 接收数据: '数据3: 语音识别样本', 已处理 (耗时: 0.210s)\n",
            "模型副本1 接收数据: '数据1: 图像分类任务', 已处理 (耗时: 0.356s)\n",
            "\n",
            "总耗时: 0.360秒\n",
            "\n",
            "============================================================\n",
            "性能对比总结\n",
            "============================================================\n",
            "单个模型副本总耗时: 0.867秒\n",
            "4个模型副本总耗时:  0.360秒\n",
            "加速比: 2.41x\n",
            "性能提升: 58.5%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2 TP(Tensor Parallel)策略\n",
        "\n",
        "### 2.1 矩阵的列切(column split)计算原理\n",
        "\n",
        "\n",
        "矩阵分块计算结果进行拼接后，与原计算得到的结果相同。 TP和SP并行都用到了这个结论。\n",
        "\n",
        "相关文章介绍：[LLM推理并行优化的必备知识](https://zhuanlan.zhihu.com/p/1937449564509545940)\n"
      ],
      "metadata": {
        "id": "tC0Ri5ij6W0X"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ewwPTyQO6Niu",
        "outputId": "fed87111-16dc-479e-b357-cddb4c074407"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "A:\n",
            " [[0 9 5 4]\n",
            " [2 1 5 2]\n",
            " [4 6 5 3]] \n",
            "shape: (3, 4)\n",
            "\n",
            "B:\n",
            " [[8 8 8 0 4 8]\n",
            " [7 6 1 0 1 6]\n",
            " [4 3 4 4 9 1]\n",
            " [7 9 3 4 0 5]] \n",
            "shape: (4, 6)\n",
            "\n",
            "B 分块结果:\n",
            "B_0:\n",
            " [[8 8]\n",
            " [7 6]\n",
            " [4 3]\n",
            " [7 9]] \n",
            "shape: (4, 2)\n",
            "B_1:\n",
            " [[8 0]\n",
            " [1 0]\n",
            " [4 4]\n",
            " [3 4]] \n",
            "shape: (4, 2)\n",
            "B_2:\n",
            " [[4 8]\n",
            " [1 6]\n",
            " [9 1]\n",
            " [0 5]] \n",
            "shape: (4, 2)\n",
            "\n",
            "局部乘积结果:\n",
            "C_0 (A @ B_0):\n",
            " [[111 105]\n",
            " [ 57  55]\n",
            " [115 110]] \n",
            "shape: (3, 2)\n",
            "C_1 (A @ B_1):\n",
            " [[41 36]\n",
            " [43 28]\n",
            " [67 32]] \n",
            "shape: (3, 2)\n",
            "C_2 (A @ B_2):\n",
            " [[54 79]\n",
            " [54 37]\n",
            " [67 88]] \n",
            "shape: (3, 2)\n",
            "\n",
            "合并后的 C_final:\n",
            " [[111 105  41  36  54  79]\n",
            " [ 57  55  43  28  54  37]\n",
            " [115 110  67  32  67  88]] \n",
            "shape: (3, 6)\n",
            "\n",
            "标准乘法结果 (A @ B):\n",
            " [[111 105  41  36  54  79]\n",
            " [ 57  55  43  28  54  37]\n",
            " [115 110  67  32  67  88]]\n",
            "\n",
            "验证一致性: True\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "# 1. 定义整数输入矩阵 (M, N) 和 (N, K)\n",
        "M, N, K = 3, 4, 6\n",
        "A = np.random.randint(0, 10, size=(M, N))  # 随机整数矩阵 [0, 10)\n",
        "B = np.random.randint(0, 10, size=(N, K))\n",
        "print(\"A:\\n\", A, \"\\nshape:\", A.shape)\n",
        "print(\"\\nB:\\n\", B, \"\\nshape:\", B.shape)\n",
        "\n",
        "# 2. 对 B 按列切分（均分）\n",
        "num_splits = 3  # 切分块数\n",
        "B_splits = np.split(B, num_splits, axis=1)  # 沿列切分\n",
        "print(\"\\nB 分块结果:\")\n",
        "for i, B_i in enumerate(B_splits):\n",
        "    print(f\"B_{i}:\\n\", B_i, \"\\nshape:\", B_i.shape)\n",
        "\n",
        "# 3. 模拟并行计算：每个进程计算 A @ B_i\n",
        "local_results = [A @ B_i for B_i in B_splits]\n",
        "print(\"\\n局部乘积结果:\")\n",
        "for i, C_i in enumerate(local_results):\n",
        "    print(f\"C_{i} (A @ B_{i}):\\n\", C_i, \"\\nshape:\", C_i.shape)\n",
        "\n",
        "# 4. 模拟 allgather：拼接所有局部结果\n",
        "C_final = np.concatenate(local_results, axis=1)\n",
        "print(\"\\n合并后的 C_final:\\n\", C_final, \"\\nshape:\", C_final.shape)\n",
        "\n",
        "# 5. 验证结果与直接乘法的等价性\n",
        "C_ground_truth = A @ B\n",
        "print(\"\\n标准乘法结果 (A @ B):\\n\", C_ground_truth)\n",
        "print(\"\\n验证一致性:\", np.array_equal(C_final, C_ground_truth))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.2 TP过程演示\n",
        "\n",
        "展示张量并行如何通过拆分大矩阵运算到多个计算单元来提高效率。\n",
        "\n",
        "问题建模：\n",
        "\n",
        "选择大矩阵（如1024×1024）模拟真实计算场景。将输入矩阵按列分块（column-wise/column-split），计算分配：每个线程处理矩阵A与B的一个列块的乘积。最后，将所有线程的计算结果拼接成完整输出矩阵。\n",
        "\n",
        "对比机制:\n",
        "- 基准测试：使用标准numpy矩阵乘法作为性能基准\n",
        "- 并行实现：使用多线程模拟多设备并行计算\n",
        "- 结果验证：确保并行计算与串行计算数值结果一致\n",
        "\n",
        "性能对比：对比元计算与TP的速度差异\n",
        "\n",
        "注意：机器不同计算速度不一样，性能对比数据仅供参考。"
      ],
      "metadata": {
        "id": "1wUgz6dkwut4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import threading\n",
        "import time\n",
        "from typing import Tuple, List\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "class TensorParallelSimulator:\n",
        "    def __init__(self, matrix_size: int = 1024):\n",
        "        \"\"\"\n",
        "        初始化张量并行模拟器\n",
        "\n",
        "        Args:\n",
        "            matrix_size: 矩阵大小 (n×n)\n",
        "        \"\"\"\n",
        "        self.matrix_size = matrix_size\n",
        "        self.numpy_runtime = None\n",
        "        self.tp_runtime = None\n",
        "\n",
        "    def generate_matrices(self) -> Tuple[np.ndarray, np.ndarray]:\n",
        "        \"\"\"生成随机矩阵用于乘法\"\"\"\n",
        "        np.random.seed(42)  # 设置随机种子以保证可重复性\n",
        "        A = np.random.randn(self.matrix_size, self.matrix_size).astype(np.float32)\n",
        "        B = np.random.randn(self.matrix_size, self.matrix_size).astype(np.float32)\n",
        "        return A, B\n",
        "\n",
        "    def numpy_matmul(self, A: np.ndarray, B: np.ndarray) -> np.ndarray:\n",
        "        \"\"\"使用numpy的标准矩阵乘法（基准）\"\"\"\n",
        "        start_time = time.time()\n",
        "        C = np.dot(A, B)\n",
        "        end_time = time.time()\n",
        "        self.numpy_runtime = end_time - start_time\n",
        "        return C\n",
        "\n",
        "    def tp_matmul_worker(self, A_part: np.ndarray, B_part: np.ndarray,\n",
        "                         result_part: np.ndarray, worker_id: int):\n",
        "        \"\"\"张量并行工作线程：计算部分矩阵乘法\"\"\"\n",
        "        start_time = time.time()\n",
        "        # 计算部分结果\n",
        "        partial_result = np.dot(A_part, B_part)\n",
        "\n",
        "        # 将结果存入共享数组的相应部分\n",
        "        result_part[:] = partial_result\n",
        "\n",
        "        end_time = time.time()\n",
        "        print(f\"  工作线程{worker_id}: 计算完成，耗时 {end_time - start_time:.3f}秒\")\n",
        "\n",
        "    def tensor_parallel_matmul(self, A: np.ndarray, B: np.ndarray,\n",
        "                               num_workers: int = 2) -> np.ndarray:\n",
        "        \"\"\"\n",
        "        张量并行矩阵乘法\n",
        "\n",
        "        策略：将矩阵B按列分块，每个线程计算A与B的一个列块的乘积\n",
        "        最后将结果按列拼接\n",
        "        \"\"\"\n",
        "        print(f\"\\n张量并行矩阵乘法 (使用{num_workers}个工作线程)\")\n",
        "        print(f\"矩阵A形状: {A.shape}, 矩阵B形状: {B.shape}\")\n",
        "\n",
        "        # 计算每个工作线程处理的列数\n",
        "        n_cols = B.shape[1]\n",
        "        cols_per_worker = n_cols // num_workers\n",
        "\n",
        "        # 初始化结果矩阵\n",
        "        C_tp = np.zeros((A.shape[0], B.shape[1]), dtype=np.float32)\n",
        "\n",
        "        threads = []\n",
        "        start_time = time.time()\n",
        "\n",
        "        # 创建并启动工作线程\n",
        "        for i in range(num_workers):\n",
        "            # 计算当前线程处理的列范围\n",
        "            start_col = i * cols_per_worker\n",
        "            # 最后一个线程处理剩余的所有列\n",
        "            end_col = start_col + cols_per_worker if i < num_workers - 1 else n_cols\n",
        "\n",
        "            # 获取B的对应列块\n",
        "            B_part = B[:, start_col:end_col]\n",
        "\n",
        "            # 获取结果矩阵的对应部分\n",
        "            C_part = C_tp[:, start_col:end_col]\n",
        "\n",
        "            # 创建工作线程\n",
        "            thread = threading.Thread(\n",
        "                target=self.tp_matmul_worker,\n",
        "                args=(A, B_part, C_part, i+1)\n",
        "            )\n",
        "            threads.append(thread)\n",
        "\n",
        "            print(f\"  分配任务给工作线程{i+1}: 处理B的列{start_col}:{end_col}\")\n",
        "\n",
        "        # 启动所有线程\n",
        "        print(\"\\n开始并行计算...\")\n",
        "        for thread in threads:\n",
        "            thread.start()\n",
        "\n",
        "        # 等待所有线程完成\n",
        "        for thread in threads:\n",
        "            thread.join()\n",
        "\n",
        "        end_time = time.time()\n",
        "        self.tp_runtime = end_time - start_time\n",
        "\n",
        "        print(f\"\\n所有工作线程完成，总耗时: {self.tp_runtime:.3f}秒\")\n",
        "        return C_tp\n",
        "\n",
        "    def validate_result(self, C_numpy: np.ndarray, C_tp: np.ndarray) -> bool:\n",
        "        \"\"\"验证两种方法的结果是否一致（在数值误差范围内）\"\"\"\n",
        "        # 计算最大绝对误差和相对误差\n",
        "        abs_error = np.max(np.abs(C_numpy - C_tp))\n",
        "        rel_error = np.max(np.abs(C_numpy - C_tp) / (np.abs(C_numpy) + 1e-10))\n",
        "\n",
        "        print(f\"\\n结果验证:\")\n",
        "        print(f\"  最大绝对误差: {abs_error:.6e}\")\n",
        "        print(f\"  最大相对误差: {rel_error:.6e}\")\n",
        "\n",
        "        # 检查是否在可接受的误差范围内\n",
        "        tolerance = 1e-5\n",
        "        is_valid = abs_error < tolerance\n",
        "\n",
        "        if is_valid:\n",
        "            print(\"张量并行计算结果与标准numpy计算结果一致！\")\n",
        "        else:\n",
        "            print(\"计算结果存在显著差异\")\n",
        "\n",
        "        return is_valid\n",
        "\n",
        "    def compare_performance(self):\n",
        "        \"\"\"比较性能并打印结果\"\"\"\n",
        "        print(\"\\n\" + \"=\"*60)\n",
        "        print(\"性能对比\")\n",
        "        print(\"=\"*60)\n",
        "        print(f\"标准numpy矩阵乘法耗时: {self.numpy_runtime:.3f}秒\")\n",
        "        print(f\"张量并行(2线程)矩阵乘法耗时: {self.tp_runtime:.3f}秒\")\n",
        "\n",
        "        if self.numpy_runtime > self.tp_runtime:\n",
        "            speedup = self.numpy_runtime / self.tp_runtime\n",
        "            improvement = (self.numpy_runtime - self.tp_runtime) / self.numpy_runtime * 100\n",
        "            print(f\"加速比: {speedup:.2f}倍\")\n",
        "            print(f\"性能提升: {improvement:.1f}%\")\n",
        "        else:\n",
        "            print(\"注意: 由于Python GIL限制和线程开销，多线程可能不会加速CPU上的矩阵运算\")\n",
        "\n",
        "def plot_comparison():\n",
        "    \"\"\"绘制不同矩阵大小下的性能对比\"\"\"\n",
        "    print(\"\\n\" + \"=\"*60)\n",
        "    print(\"不同矩阵大小下的性能分析\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    sizes = [256, 512, 1024, 2048, 4096]\n",
        "    numpy_times = []\n",
        "    tp_times = []\n",
        "\n",
        "    for size in sizes:\n",
        "        print(f\"\\n测试矩阵大小: {size}×{size}\")\n",
        "        simulator = TensorParallelSimulator(matrix_size=size)\n",
        "        A, B = simulator.generate_matrices()\n",
        "\n",
        "        # 标准numpy\n",
        "        start = time.time()\n",
        "        _ = np.dot(A, B)\n",
        "        numpy_time = time.time() - start\n",
        "        numpy_times.append(numpy_time)\n",
        "\n",
        "        # 张量并行（2线程）\n",
        "        C_tp = simulator.tensor_parallel_matmul(A, B, num_workers=2)\n",
        "        tp_times.append(simulator.tp_runtime)\n",
        "\n",
        "        print(f\"  标准numpy: {numpy_time:.3f}秒\")\n",
        "        print(f\"  张量并行: {simulator.tp_runtime:.3f}秒\")\n",
        "\n",
        "    # 绘制图表\n",
        "    plt.figure(figsize=(10, 6))\n",
        "    x = range(len(sizes))\n",
        "\n",
        "    plt.bar([i - 0.2 for i in x], numpy_times, width=0.4, label='Standard numpy', alpha=0.8, color='blue')\n",
        "    plt.bar([i + 0.2 for i in x], tp_times, width=0.4, label='Tensor Parallel (2 workers)', alpha=0.8, color='orange')\n",
        "\n",
        "    plt.xlabel('Matrix Size')\n",
        "    plt.ylabel('Computation Time (seconds)')\n",
        "    plt.title('Tensor Parallel vs Standard Matrix Multiplication Performance')\n",
        "    plt.xticks(x, [f'{size}×{size}' for size in sizes])\n",
        "    plt.legend()\n",
        "    plt.grid(True, alpha=0.3, linestyle='--')\n",
        "\n",
        "    # 在柱状图上添加数值标签\n",
        "    for i, v in enumerate(numpy_times):\n",
        "        plt.text(i - 0.2, v + 0.01, f'{v:.2f}', ha='center', va='bottom', fontsize=9)\n",
        "    for i, v in enumerate(tp_times):\n",
        "        plt.text(i + 0.2, v + 0.01, f'{v:.2f}', ha='center', va='bottom', fontsize=9)\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "    # 计算加速比\n",
        "    print(\"\\n加速比分析:\")\n",
        "    for i, size in enumerate(sizes):\n",
        "        speedup = numpy_times[i] / tp_times[i] if tp_times[i] > 0 else 0\n",
        "        print(f\"  {size}×{size}矩阵: 加速比 = {speedup:.2f}倍\")\n",
        "\n",
        "def main():\n",
        "    \"\"\"主函数\"\"\"\n",
        "    print(\"=\"*60)\n",
        "    print(\"张量并行(TP)策略演示\")\n",
        "    print(\"模拟矩阵乘法的张量并行计算\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    # 创建模拟器\n",
        "    matrix_size = 2048  # 可以调整矩阵大小\n",
        "    simulator = TensorParallelSimulator(matrix_size=matrix_size)\n",
        "\n",
        "    # 生成测试矩阵\n",
        "    print(f\"\\n生成{matrix_size}×{matrix_size}随机矩阵...\")\n",
        "    A, B = simulator.generate_matrices()\n",
        "\n",
        "    # 1. 标准numpy矩阵乘法（基准）\n",
        "    print(\"\\n1. 标准numpy矩阵乘法 (基准):\")\n",
        "    C_numpy = simulator.numpy_matmul(A, B)\n",
        "    print(f\"   计算完成，耗时: {simulator.numpy_runtime:.3f}秒\")\n",
        "\n",
        "    # 2. 张量并行矩阵乘法\n",
        "    print(\"\\n2. 张量并行矩阵乘法:\")\n",
        "    C_tp = simulator.tensor_parallel_matmul(A, B, num_workers=2)\n",
        "\n",
        "    # 3. 验证结果正确性\n",
        "    print(\"\\n3. 验证计算结果:\")\n",
        "    simulator.validate_result(C_numpy, C_tp)\n",
        "\n",
        "    # 4. 性能对比\n",
        "    simulator.compare_performance()\n",
        "\n",
        "    # 5. 不同矩阵大小性能分析\n",
        "    plot_comparison()\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    main()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "c8s7dzbqujHp",
        "outputId": "fb477385-d78f-4b8c-da35-2b09d0456cfd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "============================================================\n",
            "张量并行(TP)策略演示\n",
            "模拟矩阵乘法的张量并行计算\n",
            "============================================================\n",
            "\n",
            "生成2048×2048随机矩阵...\n",
            "\n",
            "1. 标准numpy矩阵乘法 (基准):\n",
            "   计算完成，耗时: 0.715秒\n",
            "\n",
            "2. 张量并行矩阵乘法:\n",
            "\n",
            "张量并行矩阵乘法 (使用2个工作线程)\n",
            "矩阵A形状: (2048, 2048), 矩阵B形状: (2048, 2048)\n",
            "  分配任务给工作线程1: 处理B的列0:1024\n",
            "  分配任务给工作线程2: 处理B的列1024:2048\n",
            "\n",
            "开始并行计算...\n",
            "  工作线程1: 计算完成，耗时 0.433秒\n",
            "  工作线程2: 计算完成，耗时 0.830秒\n",
            "\n",
            "所有工作线程完成，总耗时: 0.841秒\n",
            "\n",
            "3. 验证计算结果:\n",
            "\n",
            "结果验证:\n",
            "  最大绝对误差: 1.068115e-04\n",
            "  最大相对误差: 5.074625e-02\n",
            "计算结果存在显著差异\n",
            "\n",
            "============================================================\n",
            "性能对比\n",
            "============================================================\n",
            "标准numpy矩阵乘法耗时: 0.715秒\n",
            "张量并行(2线程)矩阵乘法耗时: 0.841秒\n",
            "注意: 由于Python GIL限制和线程开销，多线程可能不会加速CPU上的矩阵运算\n",
            "\n",
            "============================================================\n",
            "不同矩阵大小下的性能分析\n",
            "============================================================\n",
            "\n",
            "测试矩阵大小: 256×256\n",
            "\n",
            "张量并行矩阵乘法 (使用2个工作线程)\n",
            "矩阵A形状: (256, 256), 矩阵B形状: (256, 256)\n",
            "  分配任务给工作线程1: 处理B的列0:128\n",
            "  分配任务给工作线程2: 处理B的列128:256\n",
            "\n",
            "开始并行计算...\n",
            "  工作线程1: 计算完成，耗时 0.007秒\n",
            "  工作线程2: 计算完成，耗时 0.005秒\n",
            "\n",
            "所有工作线程完成，总耗时: 0.010秒\n",
            "  标准numpy: 0.001秒\n",
            "  张量并行: 0.010秒\n",
            "\n",
            "测试矩阵大小: 512×512\n",
            "\n",
            "张量并行矩阵乘法 (使用2个工作线程)\n",
            "矩阵A形状: (512, 512), 矩阵B形状: (512, 512)\n",
            "  分配任务给工作线程1: 处理B的列0:256\n",
            "  分配任务给工作线程2: 处理B的列256:512\n",
            "\n",
            "开始并行计算...\n",
            "  工作线程1: 计算完成，耗时 0.008秒\n",
            "  工作线程2: 计算完成，耗时 0.013秒\n",
            "\n",
            "所有工作线程完成，总耗时: 0.019秒\n",
            "  标准numpy: 0.011秒\n",
            "  张量并行: 0.019秒\n",
            "\n",
            "测试矩阵大小: 1024×1024\n",
            "\n",
            "张量并行矩阵乘法 (使用2个工作线程)\n",
            "矩阵A形状: (1024, 1024), 矩阵B形状: (1024, 1024)\n",
            "  分配任务给工作线程1: 处理B的列0:512\n",
            "  分配任务给工作线程2: 处理B的列512:1024\n",
            "\n",
            "开始并行计算...\n",
            "  工作线程1: 计算完成，耗时 0.081秒\n",
            "  工作线程2: 计算完成，耗时 0.152秒\n",
            "\n",
            "所有工作线程完成，总耗时: 0.163秒\n",
            "  标准numpy: 0.092秒\n",
            "  张量并行: 0.163秒\n",
            "\n",
            "测试矩阵大小: 2048×2048\n",
            "\n",
            "张量并行矩阵乘法 (使用2个工作线程)\n",
            "矩阵A形状: (2048, 2048), 矩阵B形状: (2048, 2048)\n",
            "  分配任务给工作线程1: 处理B的列0:1024\n",
            "  分配任务给工作线程2: 处理B的列1024:2048\n",
            "\n",
            "开始并行计算...\n",
            "  工作线程1: 计算完成，耗时 0.590秒\n",
            "  工作线程2: 计算完成，耗时 0.904秒\n",
            "\n",
            "所有工作线程完成，总耗时: 0.919秒\n",
            "  标准numpy: 0.955秒\n",
            "  张量并行: 0.919秒\n",
            "\n",
            "测试矩阵大小: 4096×4096\n",
            "\n",
            "张量并行矩阵乘法 (使用2个工作线程)\n",
            "矩阵A形状: (4096, 4096), 矩阵B形状: (4096, 4096)\n",
            "  分配任务给工作线程1: 处理B的列0:2048\n",
            "  分配任务给工作线程2: 处理B的列2048:4096\n",
            "\n",
            "开始并行计算...\n",
            "  工作线程1: 计算完成，耗时 1.309秒\n",
            "  工作线程2: 计算完成，耗时 2.903秒\n",
            "\n",
            "所有工作线程完成，总耗时: 2.921秒\n",
            "  标准numpy: 5.182秒\n",
            "  张量并行: 2.921秒\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA90AAAJOCAYAAACqS2TfAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAt51JREFUeJzs3Xd4U9X/B/D3Sdt0p7uFMlooUKDspYAMRWXJFEHUr2xRVERAxcEoKIgDcTJUhgPZqCCoTGU52CigggwF7Kahu809vz/4JTQ0haY0hJO8X8/Doz25ufmc3Hdue3LvPVdIKSWIiIiIiIiIqMLpnF0AERERERERkavioJuIiIiIiIjIQTjoJiIiIiIiInIQDrqJiIiIiIiIHISDbiIiIiIiIiIH4aCbiIiIiIiIyEE46CYiIiIiIiJyEA66iYiIiIiIiByEg24iIiIiIiIiB+Ggm4joJtGxY0d07NjR8vOpU6cghMCiRYvsXte2bdsghMC2bdsqrD53IITAlClTbshrXbm9VXMj36uKsGjRIgghcOrUqWsuez2fH1uf2ylTpkAIYfe6rtf17ENUkpWVheHDh6NSpUoQQmDMmDHOLomIyAoH3UQE4NIf0GX55yqDOPMf1eZ/Xl5eqFmzJh5++GH8/fffzi7PpezYsQNdu3ZFlSpV4OPjg+rVq6NHjx5YsmSJZZmcnBxMmTLFZfJVkWJjYyGEwJ133mnz8Q8//NCS4z179ti9/l27dmHKlCm4cOHCdVZafsU/j5999pnNZdq2bQshBBo0aFBhr/vBBx+4xIB0yZIlmD17trPLsDJ48GCrfazBYEDjxo3x5ptvIj8/v0Jfa/r06Vi0aBEee+wxfPrpp/jf//5XoesnIrpens4ugIhuDp9++qnVz5988gk2btxYor1evXo3siyHGz16NFq2bInCwkLs27cP8+fPxzfffIPDhw8jOjra2eUpb8WKFRgwYACaNGmCp556CiEhITh58iR+/PFHfPjhh3jggQcAXBp0JyYmAoDSR38dxcfHB1u3bsV///2HSpUqWT32+eefw8fHB3l5eeVa965du5CYmIjBgwcjODi4zM/Lzc2Fp2fF/hnh4+ODJUuW4KGHHrJqP3XqFHbt2gUfH58Kfb0PPvgA4eHhGDx4sFV7+/btkZubC71eXyGv89JLL2HChAkVsi5blixZgt9++63EEd6YmBjk5ubCy8vLYa99Nd7e3vjoo48AABcuXMCqVaswfvx4/Prrr1i6dGmFvc6WLVtw6623YvLkyRW2TiKiisRBNxEBQIk/cn/66Sds3LixRLtKsrOz4e/vf9Vl2rVrh379+gEAhgwZgjp16mD06NFYvHgxnn/++XK/tpQSeXl58PX1Lfc6XMGUKVNQv359/PTTTyUGMMnJyU6q6sYoS/7Kqm3btvj111+xbNkyPPXUU5b2f//9F9u3b0efPn2watWqCnmtq9E0DQUFBfDx8anwATAAdOvWDV9//TVSU1MRHh5uaV+yZAmioqJQu3ZtZGRkVPjrXkmn01Vo/zw9PSv8C4qyEEI4ZDuVlaenp9XvkFGjRuGWW27BsmXLMGvWrOv6YrN4FpOTk1G/fv2KKBkAUFRUBE3TKuxLFyIinl5ORGWmaRpmz56NhIQE+Pj4ICoqCiNHjizxR3BsbCzuuece7NixA61atYKPjw9q1qyJTz75xGq5wsJCJCYmonbt2vDx8UFYWBhuu+02bNy40Wq5LVu2oF27dvD390dwcDB69eqFo0ePWi1jvmbyyJEjeOCBBxASEoLbbrvN7j7ecccdAICTJ08CABYuXIg77rgDkZGR8Pb2Rv369TFnzpwSzzP3+bvvvkOLFi3g6+uLefPm2bWOsjp27Bj69euH0NBQ+Pj4oEWLFvj666/tXs/KlSshhMAPP/xQ4rF58+ZBCIHffvsNAPDff/9hyJAhqFq1Kry9vVG5cmX06tXrmtfHnjhxAi1btrT5x2tkZCSAS0cxIyIiAACJiYmW01HN1wsfOnQIgwcPRs2aNeHj44NKlSph6NChSEtLs1qfOQPHjx+3HLUNCgrCkCFDkJOTY7Vsfn4+nn76aURERCAwMBA9e/bEv//+W6LG06dPY9SoUYiPj4evry/CwsJw3333lei3+XrhH374AaNGjUJkZCSqVq1qeXz+/PmIi4uDr68vWrVqhe3bt1/1fbuSj48P+vbta3VKPgB88cUXCAkJQefOnUs8pyzv25QpU/DMM88AAGrUqGF57839E0LgiSeewOeff46EhAR4e3vj22+/tTxm3ka5ubmoW7cu6tati9zcXMv609PTUblyZbRp0wYmk+ma/ezVqxe8vb2xYsUKq/YlS5agf//+8PDwsGq/2jXL17rmPDY2Fr///jt++OEHS7/NZ1nYuqa7Y8eOaNCgAfbu3Ys2bdrA19cXNWrUwNy5c6/Zr9Ku6f7ss8/QqlUr+Pn5ISQkBO3bt8f3339vefyrr75C9+7dER0dDW9vb8TFxWHatGlW72XHjh3xzTff4PTp05Z+xMbGXvX9sWefWpbPU1npdDrLe2zOWH5+PiZPnoxatWrB29sb1apVw7PPPlviFPTSsiiEwMmTJ/HNN9+UyG9ycjKGDRuGqKgo+Pj4oHHjxli8eLHVes3v0RtvvIHZs2cjLi4O3t7eOHLkiOU9+PPPP/HQQw8hKCgIERERmDhxIqSU+Oeff9CrVy8YDAZUqlQJb775ptW6CwoKMGnSJDRv3hxBQUHw9/dHu3btsHXr1lJrMO8rvL290bJlS/z6668l3sdjx46hf//+iIiIgK+vL+Lj4/Hiiy9aLXP27FkMHToUUVFR8Pb2RkJCAhYsWGDvJiOiCsIj3URUZiNHjsSiRYswZMgQjB49GidPnsR7772H/fv3Y+fOnVanMB4/fhz9+vXDsGHDMGjQICxYsACDBw9G8+bNkZCQAODSH3UzZszA8OHD0apVKxiNRuzZswf79u3DXXfdBQDYtGkTunbtipo1a2LKlCnIzc3Fu+++i7Zt22Lfvn2WPy7N7rvvPtSuXRvTp0+HlNLuPp44cQIAEBYWBgCYM2cOEhIS0LNnT3h6emLt2rUYNWoUNE3D448/bvXcP/74AwMHDsTIkSMxYsQIxMfH272Oa/n999/Rtm1bVKlSBRMmTIC/vz+WL1+O3r17Y9WqVejTp0+Z19W9e3cEBARg+fLl6NChg9Vjy5YtQ0JCguX62XvvvRe///47nnzyScTGxiI5ORkbN27EmTNnSmyD4mJiYrB582b8+++/VoPQ4iIiIjBnzhw89thj6NOnD/r27QsAaNSoEQBg48aN+PvvvzFkyBBUqlQJv//+O+bPn4/ff/8dP/30U4nBTP/+/VGjRg3MmDED+/btw0cffYTIyEjMnDnTsszw4cPx2Wef4YEHHkCbNm2wZcsWdO/evURtv/76K3bt2oX7778fVatWxalTpzBnzhx07NgRR44cgZ+fn9Xyo0aNQkREBCZNmoTs7GwAwMcff4yRI0eiTZs2GDNmDP7++2/07NkToaGhqFatWqnv3ZUeeOAB3H333Thx4gTi4uIAXBqM9uvXz+bpw2V53/r27Ys///wTX3zxBd566y3L0WXzlyDApQHa8uXL8cQTTyA8PNzm9vb19cXixYvRtm1bvPjii5g1axYA4PHHH0dmZiYWLVpUYsBsi5+fH3r16oUvvvgCjz32GADg4MGD+P333/HRRx/h0KFDZX6/rmX27Nl48sknERAQYBmwREVFXfU5GRkZ6NatG/r374+BAwdi+fLleOyxx6DX6zF06FC7Xj8xMRFTpkxBmzZtMHXqVOj1evz888/YsmUL7r77bgCXvswJCAjA2LFjERAQgC1btmDSpEkwGo14/fXXAQAvvvgiMjMz8e+//+Ktt94CAAQEBJT6uvbuU8vyebJH8X2spmno2bMnduzYgUceeQT16tXD4cOH8dZbb+HPP//El19+afXcK7NYuXJlfPrpp3j66adRtWpVjBs3DsCl/Obm5qJjx444fvw4nnjiCdSoUQMrVqzA4MGDceHCBaszRoBLX47m5eXhkUcegbe3N0JDQy2PDRgwAPXq1cOrr76Kb775Bi+//DJCQ0Mxb9483HHHHZg5cyY+//xzjB8/Hi1btkT79u0BAEajER999BEGDhyIESNG4OLFi/j444/RuXNn/PLLL2jSpIlVDUuWLMHFixcxcuRICCHw2muvoW/fvvj7778tn/FDhw6hXbt28PLywiOPPILY2FicOHECa9euxSuvvAIASEpKwq233mr5oiIiIgIbNmzAsGHDYDQaOdEckTNIIiIbHn/8cVl8F7F9+3YJQH7++edWy3377bcl2mNiYiQA+eOPP1rakpOTpbe3txw3bpylrXHjxrJ79+5XraNJkyYyMjJSpqWlWdoOHjwodTqdfPjhhy1tkydPlgDkwIEDy9S/rVu3SgBywYIFMiUlRZ47d05+8803MjY2Vgoh5K+//iqllDInJ6fEczt37ixr1qxp1Wbu87ffflti+bKuo0OHDrJDhw6Wn0+ePCkByIULF1raOnXqJBs2bCjz8vIsbZqmyTZt2sjatWuX6N/WrVuv+j4MHDhQRkZGyqKiIkvb+fPnpU6nk1OnTpVSSpmRkSEByNdff/2q67Ll448/lgCkXq+Xt99+u5w4caLcvn27NJlMVsulpKRIAHLy5Mkl1mHr/fviiy9KZMycgaFDh1ot26dPHxkWFmb5+cCBAxKAHDVqlNVyDzzwQIkabL327t27JQD5ySefWNoWLlwoAcjbbrvN6r0sKCiQkZGRskmTJjI/P9/SPn/+fAnAanuXJiYmRnbv3l0WFRXJSpUqyWnTpkkppTxy5IgEIH/44QfL65tzW1rttt63119/XQKQJ0+eLLE8AKnT6eTvv/9u87Ert9fzzz8vdTqd/PHHH+WKFSskADl79uxr9tGc1xUrVsh169ZJIYQ8c+aMlFLKZ555xvJZ6dChg0xISLA8z9ZnpLT6zO9R8X4mJCTY3Aa2Pj8dOnSQAOSbb75pacvPz7fsowoKCkqtyZxNs7/++kvqdDrZp0+fEp8FTdMs/29rG44cOVL6+flZ7QO6d+8uY2JiSixrqxZ796nX+jyVZtCgQdLf31+mpKTIlJQUefz4cTl9+nQphJCNGjWSUkr56aefSp1OJ7dv32713Llz50oAcufOnZa2q2XR/Bkpbvbs2RKA/OyzzyxtBQUFsnXr1jIgIEAajUar98hgMMjk5GSrdZjfg0ceecTSVlRUJKtWrSqFEPLVV1+1tGdkZEhfX185aNAgq2WLf+7Ny0VFRVm9r+YawsLCZHp6uqX9q6++kgDk2rVrLW3t27eXgYGB8vTp01brLZ6bYcOGycqVK8vU1FSrZe6//34ZFBRkM1dE5Fg8vZyIymTFihUICgrCXXfdhdTUVMu/5s2bIyAgoMTpcvXr10e7du0sP0dERCA+Pt5qZvDg4GD8/vvv+Ouvv2y+5vnz53HgwAEMHjzY6qhDo0aNcNddd2H9+vUlnvPoo4/a1a+hQ4ciIiIC0dHR6N69O7Kzs7F48WK0aNECAKyuyc7MzERqaio6dOiAv//+G5mZmVbrqlGjhs3TfO1Zx9Wkp6djy5Yt6N+/Py5evGjZBmlpaejcuTP++usvnD171q7+DxgwAMnJyVan0a5cuRKapmHAgAGW+vV6PbZt22b39bRDhw7Ft99+i44dO2LHjh2YNm0a2rVrh9q1a2PXrl1lWkfx9y8vLw+pqam49dZbAQD79u0rsfyVGWjXrh3S0tJgNBoBwJKb0aNHWy1n6+hP8dcuLCxEWloaatWqheDgYJuvPWLECKsjunv27EFycjIeffRRq1PsBw8ejKCgoFL7bIuHhwf69++PL774AsClCdSqVatm9TkrrfayvG+l6dChQ5mvl50yZQoSEhIwaNAgjBo1Ch06dCjxPl/L3XffjdDQUCxduhRSSixduhQDBw60ax2O4unpiZEjR1p+1uv1GDlyJJKTk7F3794yr+fLL7+EpmmYNGkSdDrrP8WKn7lRfBuaP/Pt2rVDTk4Ojh07Znf9FbFPvfLzdDXZ2dmIiIhAREQEatWqhRdeeAGtW7fGmjVrAFz6vVKvXj3UrVvX6veK+TKfK3+v2JPF9evXo1KlSlbZ8fLywujRo5GVlVXispp7773X6gyP4oYPH275fw8PD7Ro0QJSSgwbNszSHhwcXOJ3nIeHh+Vzr2ka0tPTUVRUhBYtWtj8DA4YMAAhISGWn82fbfM6U1JS8OOPP2Lo0KGoXr261XPNuZFSYtWqVejRoweklFbva+fOnZGZmWnX55+IKgZPLyeiMvnrr7+QmZlpuQ73SldOinXlHwQAEBISYjVomzp1Knr16oU6deqgQYMG6NKlC/73v/9ZTis+ffo0AFhO0y6uXr16+O6770pMVlWjRg27+jVp0iS0a9cOHh4eCA8PR7169awmPNq5cycmT56M3bt3l7iOMTMz02rgVNpr27OOqzl+/DiklJg4cSImTpxoc5nk5GRUqVKlTOsDgC5duiAoKAjLli1Dp06dAFw6tbxJkyaoU6cOgEszEM+cORPjxo1DVFQUbr31Vtxzzz14+OGHS8ykbUvnzp3RuXNn5OTkYO/evVi2bBnmzp2Le+65B8eOHSs1U2bp6elITEzE0qVLS+TM1pcWV2bP/EdsRkYGDAYDTp8+DZ1OZzlF28xWznJzczFjxgwsXLgQZ8+etbpkwdZrX5kBc4Zr165t1W6+RZ29HnjgAbzzzjs4ePAglixZgvvvv7/U+z/b+76Vxp7PlF6vx4IFC9CyZUv4+Phg4cKFdt+f2svLC/fddx+WLFmCVq1a4Z9//rHMcu9s0dHRJSbHM39OTp06ZflS41pOnDgBnU53zQHk77//jpdeeglbtmwpMci1ZxualWefeq3P09X4+Phg7dq1AC7tR2rUqGF1mclff/2Fo0ePljrYvTK39mTx9OnTqF27dokvNcx34DC/F2VZ95XvQVBQEHx8fKwm+zO3XznXxOLFi/Hmm2/i2LFjKCwsvOrrXe29Bi4Pvq9227yUlBRcuHAB8+fPx/z5820u4+qTWBLdjDjoJqIy0TQNkZGR+Pzzz20+fuUfTaVdv1l80NK+fXucOHECX331Fb7//nt89NFHeOuttzB37lyrIwv2sHe28IYNG5Z6/+MTJ06gU6dOqFu3LmbNmoVq1apBr9dj/fr1eOutt6Bp2jVf2951XI152fHjx9s8og4AtWrVKvP6gEt/CPfu3Rtr1qzBBx98gKSkJOzcuRPTp0+3Wm7MmDHo0aMHvvzyS3z33XeYOHEiZsyYgS1btqBp06Zlei0/Pz+0a9cO7dq1Q3h4OBITE7FhwwYMGjToqs/r378/du3ahWeeeQZNmjRBQEAANE1Dly5dbL5/ZcleWT355JNYuHAhxowZg9atWyMoKAhCCNx///02X9vRs9XfcsstiIuLw5gxY3Dy5MmrDkbtfd9KY2+fvvvuOwCXjq7/9ddfdn8RBlz6cmHu3LmYMmUKGjduXOrgtLQBfVkmbbvZXbhwAR06dIDBYMDUqVMRFxcHHx8f7Nu3D88995xd2/B6XM/nycPDo9T9K3Bpn9awYUPLHABXunLOA0d+vq62blvvQVnel88++wyDBw9G79698cwzzyAyMhIeHh6YMWOG5dp2e9d5LeZcPPTQQ6XuW81fbBPRjcNBNxGVSVxcHDZt2oS2bdtW6B8+oaGhGDJkCIYMGYKsrCy0b98eU6ZMwfDhwxETEwPg0gRlVzp27BjCw8Mr7JZMtqxduxb5+fn4+uuvrY5AXHnKo6PXYWY+Murl5XXVP2TtNWDAACxevBibN2/G0aNHIaW0nFpeXFxcHMaNG4dx48bhr7/+QpMmTfDmm2/is88+s/s1zafvnz9/HkDpg6eMjAxs3rwZiYmJmDRpkqW9tEsSyiImJgaapuHEiRNWR/xs5WzlypUYNGiQ1azEeXl5uHDhQplfy1yv+ZRZ4NKp6idPnkTjxo3trn/gwIF4+eWXUa9evRITMZnZ877ZeyT6ag4dOoSpU6diyJAhOHDgAIYPH47Dhw/bfSr9bbfdhurVq2Pbtm1XnbDLfCTwyu1x5VHM0tjb93PnzpU4Evznn38CwFUnFLxSXFwcNE3DkSNHSt2G27ZtQ1paGlavXm2ZmAu4fGeF4sraD2fvU68UFxeHgwcPolOnThWaQ+BSXw8dOgRN06yOdptPyze/F460cuVK1KxZE6tXr7bqX3nvJ27+HWC+q4Qt5jsymEymCv09QUTXh9d0E1GZ9O/fHyaTCdOmTSvxWFFRUZkHIcVdeRpeQEAAatWqZblVTOXKldGkSRMsXrzYav2//fYbvv/+e3Tr1s3u17SH+ajDlacUL1y48IauwywyMhIdO3bEvHnzLIPV4lJSUuxeJwDceeedCA0NxbJly7Bs2TK0atXK6uhkTk4O8vLyrJ4TFxeHwMDAErf1udLmzZtttpuvHTUPes2zgF+ZI1vvH3Bp5uny6tq1KwDgnXfeueY6PTw8Srz2u+++W+YjqS1atEBERATmzp2LgoICS/uiRYvK9ZkBLl1fOnny5BK3J7qybqBs75t5kFXeeswKCwsxePBgREdH4+2338aiRYuQlJSEp59+2u51CSHwzjvvYPLkyfjf//5X6nIGgwHh4eH48ccfrdo/+OCDMr2Ov7+/Xf0uKiqy3AoQuHRLqHnz5iEiIgLNmzcv83p69+4NnU6HqVOnljhibd5mtrZhQUGBzb75+/uX6XRzZ+9Tr9S/f3+cPXsWH374YYnHcnNzLXcAKI9u3brhv//+w7JlyyxtRUVFePfddxEQEFDijg2OYGsb/vzzz9i9e3e51hcREYH27dtjwYIFOHPmjNVjxXNz7733YtWqVTYH5+X9PUFE14dHuomoTDp06ICRI0dixowZOHDgAO6++254eXnhr7/+wooVK/D222+jX79+dq2zfv366NixI5o3b47Q0FDs2bMHK1euxBNPPGFZ5vXXX0fXrl3RunVrDBs2zHJ7m6CgoKveg7ci3H333dDr9ejRowdGjhyJrKwsfPjhh4iMjLQ56HXUOop7//33cdttt6Fhw4YYMWIEatasiaSkJOzevRv//vsvDh48aPc6vby80LdvXyxduhTZ2dl44403rB7/888/0alTJ/Tv3x/169eHp6cn1qxZg6SkJNx///1XXXevXr1Qo0YN9OjRA3FxccjOzsamTZuwdu1atGzZEj169ABw6dTO+vXrY9myZahTpw5CQ0PRoEEDNGjQAO3bt8drr72GwsJCVKlSBd9//73No31l1aRJEwwcOBAffPABMjMz0aZNG2zevBnHjx8vsew999yDTz/9FEFBQahfvz52796NTZs2WW4pdy1eXl54+eWXMXLkSNxxxx0YMGAATp48iYULF5brmm7g0hG6a2XfYDCU+X0zDxZffPFF3H///fDy8kKPHj3sPuL58ssv48CBA9i8eTMCAwPRqFEjTJo0CS+99BL69etn94CuV69e6NWr1zWXGz58OF599VUMHz4cLVq0wI8//mg5+nwtzZs3x5w5c/Dyyy+jVq1aiIyMtDoj4UrR0dGYOXMmTp06hTp16mDZsmU4cOAA5s+fb/O2baWpVasWXnzxRcvEgn379oW3tzd+/fVXREdHY8aMGWjTpg1CQkIwaNAgjB49GkIIfPrppzZPNW7evDmWLVuGsWPHomXLlggICLB8tq7kzH3qlf73v/9h+fLlePTRR7F161a0bdsWJpMJx44dw/Lly/Hdd99Zzoqx1yOPPIJ58+Zh8ODB2Lt3L2JjY7Fy5Urs3LkTs2fPRmBgYAX3pqR77rkHq1evRp8+fdC9e3ecPHkSc+fORf369ZGVlVWudb7zzju47bbb0KxZMzzyyCOoUaMGTp06hW+++QYHDhwAALz66qvYunUrbrnlFowYMQL169dHeno69u3bh02bNiE9Pb0Ce0lEZXIjp0onInVcecsws/nz58vmzZtLX19fGRgYKBs2bCifffZZee7cOcsytm7fImXJW2K9/PLLslWrVjI4OFj6+vrKunXryldeecVy6x2zTZs2ybZt20pfX19pMBhkjx495JEjR6yWMd/aJSUlpUz9K36Loqv5+uuvZaNGjaSPj4+MjY2VM2fOlAsWLChx66HS+mzPOspyyzAppTxx4oR8+OGHZaVKlaSXl5esUqWKvOeee+TKlStL9O9atwwz27hxowQghRDyn3/+sXosNTVVPv7447Ju3brS399fBgUFyVtuuUUuX778muv94osv5P333y/j4uKkr6+v9PHxkfXr15cvvvii5ZY9Zrt27ZLNmzeXer3e6nZP//77r+zTp48MDg6WQUFB8r777pPnzp0rcUuo0jJg61ZRubm5cvTo0TIsLEz6+/vLHj16yH/++afEOjMyMuSQIUNkeHi4DAgIkJ07d5bHjh2TMTExVrcGsnXLruI++OADWaNGDent7S1btGghf/zxxxLbuzRXy9bVXr+s75uUUk6bNk1WqVJF6nQ6q/cKgHz88cdtvmbx9ezdu1d6enrKJ5980mqZoqIi2bJlSxkdHS0zMjJKrb+sn8crbxkm5aXbag0bNkwGBQXJwMBA2b9/f5mcnFymW4b9999/snv37jIwMNDqFm6l3TIsISFB7tmzR7Zu3Vr6+PjImJgY+d5771nVU5ZbhpktWLBANm3aVHp7e8uQkBDZoUMHuXHjRsvjO3fulLfeeqv09fWV0dHR8tlnn5XfffddidqysrLkAw88IIODgyUAy+3DStuHXM8+1db7aIv5lmHXUlBQIGfOnCkTEhIs70Pz5s1lYmKizMzMtCx3tSyW9hlJSkqyfH71er1s2LBhiffC/B7ZuiViae9BaX27Mp+apsnp06fLmJgY6e3tLZs2bSrXrVsnBw0aZHWLt6vVYOvz+ttvv1k+2z4+PjI+Pl5OnDixRN8ff/xxWa1aNenl5SUrVaokO3XqJOfPn1/iNYjI8YSU5ZhZhoiIiMiNdOzYEampqVe9npaIiMgWXtNNRERERERE5CAcdBMRERERERE5CAfdRERERERERA7Ca7qJiIiIiIiIHIRHuomIiIiIiIgchINuIiIiIiIiIgfxdHYB10PTNJw7dw6BgYEQQji7HCIiIiIiInITUkpcvHgR0dHR0OlKP56t9KD73LlzqFatmrPLICIiIiIiIjf1zz//oGrVqqU+rvSgOzAwEMClThoMBidXQ+WhaRpSUlIQERFx1W+HiG5WzDCpjPkl1THDpDLmV31GoxHVqlWzjEtLo/Sg23xKucFg4KBbUZqmIS8vDwaDgTsbUhIzTCpjfkl1zDCpjPl1Hde61Jlbl4iIiIiIiMhBOOgmp+MkeKQ6ZphUxvyS6phhUhnz6x6ElFI6u4jyMhqNCAoKQmZmJk8vJyIiIiIiohumrONRpa/pLiuTyYTCwkJnl0E2SClRWFgILy8vftPnRry8vODh4eHsMiqElBIFBQXQ6/XMMCmH+SXVMcOkMubXfbj0oFtKif/++w8XLlxwdilUCiklNE2DTqfjzsbNBAcHo1KlSspvdyklMjIyEBkZqXxfyP0wv6Q6ZphUxvy6D5cedJsH3JGRkfDz82OYb0JSShQVFcHT05Pbx01IKZGTk4Pk5GQAQOXKlZ1cERERERGR47jsoNtkMlkG3GFhYc4uh0rBQbd78vX1BQAkJycjMjLSZU41JyIiIiK6ksvOXm6+htvPz8/JlRCRLebPpivMt+Dp6bLfX5IbYH5JdcwwqYz5dQ8uO+g249HTm5sQgpOouSlX2eY6nQ7h4eHQ6Vx+d0ouiPkl1THDNHjwYOj1egQEBFj+7d69u9Tl33vvPbRo0QLe3t7o3bt3icePHDmCTp06ISQkBJUqVcIjjzyCnJwch9TO/LoPbmFyKiklTCYTFL5zHbk58zXqzDCpiPkl1THDBACjRo1CVlaW5V/r1q1LXTY6OhovvfQSRowYYfPxBx54APHx8UhKSsLhw4dx8OBBTJs2zSF1M7/ug4NuuqZTp05BCIEDBw44ZN2enp4OWTfRjSClhNFo5C9MUhLzS6pjhsleffv2Re/evREeHm7z8b///hsPPfQQ9Ho9IiIi0LNnTxw+fNghtTC/7sMtLyJo0eLGvdaePfY/JyUlBZMmTcI333yDpKQkhISEoHHjxpg0aRLatm0L4NKpuWvWrLF5WgwRERERkbv45JNP8Mknn6By5coYOnQonn766XKfsj1+/Hh88sknaNq0KTIzM7FmzZpSj4oTlZVbDrpvdvfeey8KCgqwePFi1KxZE0lJSdi8eTPS0tKcXVq5FRQUQK/XO7sMIiIiInIho0ePxuuvv47Q0FD8+uuv6N+/P3Q6HZ5++ulyra9r164YMmQIAgMDYTKZ0Lt3bwwdOrSCqyZ3w9PLbzIXLlzA9u3bMXPmTNx+++2IiYlBq1at8Pzzz6Nnz54AgNjYWABAnz59IISw/HzixAn06tULUVFRCAgIQMuWLbFp0yar9cfGxmL69OkYOnQoAgMDUb16dcyfP99qmV9++QVNmzaFj48PWrRogf3791s9bjKZMGzYMNSoUQO+vr6Ij4/H22+/bbXM4MGD0bt3b7zyyiuIjo5GfHx8mdZty7Vq3rZtG4QQuHDhgqXtwIEDEELg1KlTAIBFixYhODgY69atQ3x8PPz8/NCvXz/k5ORg8eLFiI2NRUhICEaPHg2TyWT12tOmTcPAgQPh7++PKlWq4P3337c8PnToUNxzzz1W9RYWFiIyMhIff/zxNftG6hNCQK/Xu8zEcORemF9SHTNMzZo1Q0REBDw8PHDrrbdiwoQJWLZsWbnWlZGRgTvvvBMjRoxATk4O0tPT4e/vj4ceeqiCq76E+XUfHHTfZMyzLn755ZfIz8+3ucyvv/4KAFi4cCHOnz9v+TkrKwvdunXD5s2bsX//fnTp0gU9evTAmTNnrJ7/5ptvWga8o0aNwmOPPYY//vjDso577rkH9evXx969ezFlyhSMHz/e6vmapqFq1apYsWIFjhw5gkmTJuGFF17A8uXLrZbbvHkz/vjjD2zcuBHr1q2zue5nnnkGwLVnsr5azWWVk5ODd955B0uXLsW3336Lbdu2oU+fPli/fj3Wr1+PTz/9FPPmzcPKlSutnvf666+jcePG2L9/PyZMmICnnnoKGzduBAAMHz4c3377Lc6fP29Zft26dcjJycGAAQPsqo/UJIRAaGgof2GSkphfUh0zTFe6npnAT5w4gdzcXIwePRp6vR4hISEYOXIkvvnmmwqs8DLm131w0H2T8fT0xKJFi7B48WIEBwejbdu2eOGFF3Do0CHLMhEREQCA4OBgVKpUyfJz48aNMXLkSDRo0AC1a9fGtGnTEBcXh6+//trqNbp164ZRo0ahVq1aeO655xAeHo6tW7cCAJYsWQJN0/Dxxx8jISEB99xzj2VgbObl5YXExES0aNECNWrUwIMPPoghQ4aUGHT7+/vjo48+QkJCAhISEmyu2zygv9YEEleruawKCwsxZ84cNG3aFO3bt0e/fv2wY8cOfPzxx6hfvz7uuece3H777SXW27ZtW0yYMAF16tTBk08+iX79+uGtt94CALRp0wbx8fH49NNPLcsvXLgQ9913HwICAuyqj9QkpcTFixc5CQopifkl1THDtHz5cstkZHv27MGrr76Ke++9t9Tli4qKkJeXh6KiImiahry8PBQUFAAA6tati4CAAHzwwQcoKirCxYsX8eGHH6Jp06YOqZ35dR8cdN+E7r33Xpw7dw5ff/01unTpgm3btqFZs2ZYtGjRVZ+XlZWF8ePHo169eggODkZAQACOHj1a4kh3o0aNLP8vhEClSpWQnJwMADh69CgaNWoEHx8fyzK2brvw/vvvo3nz5oiIiEBAQADmz59f4nUaNmxodR13Wddty9VqLis/Pz/ExcVZfo6KikJsbKzV4DgqKqrEeq+ssXXr1jh69Kjl5+HDh2PhwoUAgKSkJGzYsIHX/rgRKSWys7P5C5OUxPyS6phheu+991C9enUEBgbiwQcfxKhRozBu3DjL448++igeffRRy88vv/wyfH198corr2Dt2rXw9fXF3XffDeDSGadr167FF198gfDwcMTGxuLChQtYvHixQ2pnft0HJ1K7Sfn4+OCuu+7CXXfdhYkTJ2L48OGYPHkyBg8eXOpzxo8fj40bN+KNN95ArVq14Ovri379+lm+vTPz8vKy+lkIAU3Tylzb0qVLMX78eLz55pto3bo1AgMD8frrr+Pnn3+2Ws7f37/M67yWq9VsPo2o+A6rsLCwTOu43vcCAB5++GFMmDABu3fvxq5du1CjRg20a9fOrnUQERERkf1+/PHHqz4+d+5cq5+nTJmCKVOmlLp827ZtsWPHjooojciCR7oVUb9+fWRnZ1t+9vLysprwCwB27tyJwYMHo0+fPmjYsCEqVapkmUisrOrVq4dDhw4hLy/P0vbTTz+VeJ02bdpg1KhRaNq0KWrVqoUTJ05UyLrLw3x6ffHrqivyvt9X1vjTTz+hXr16lp/DwsLQu3dvLFy4EIsWLcKQIUMq7LWJiIiIiEhtHHTfZNLS0nDHHXfgs88+w6FDh3Dy5EmsWLECr732Gnr16mVZLjY2Fps3b8Z///2HjIwMAEDt2rWxevVqHDhwAAcPHsQDDzxg91HbBx54AEIIjBgxAkeOHMH69evxxhtvWC1Tu3Zt7NmzB9999x3+/PNPTJw40TKZm73rfvPNN+2qz5ZatWqhWrVqmDJlCv766y988803FbJes507d+K1117Dn3/+iffffx8rVqzAU089ZbXM8OHDsXjxYhw9ehSDBg2qsNemm58QAr6+vpwEhZTE/JLqmGFSGfPrPjjovskEBATglltuwVtvvYX27dujQYMGmDhxIkaMGIH33nvPstybb76JjRs3olq1apbJHWbNmoWQkBC0adMGPXr0QOfOndGsWTO7X3/t2rU4fPgwmjZtihdffBEzZ860WmbkyJHo27cvBgwYgFtuuQVpaWkYNWrUda37enY2Xl5e+OKLL3Ds2DE0atQIM2fOxMsvv1zu9V1p3Lhx2LNnD5o2bYqXX34Zs2bNQufOna2WufPOO1G5cmV07twZ0dHRFfbadPMTQiAoKIi/MElJzC+pjhkmlTG/7kNIha/cNxqNCAoKQmZmJgwGg9VjeXl5OHnyJGrUqGE1cRfdXKSUMJlM8PDwuCl3OLGxsRgzZgzGjBlz1eWysrJQpUoVLFy4EH379r0xxSnOVT6jUkoYjUYYDIabMsNEV8P8kuqYYTW0aOHsCm5OQkhs2sT8quxq49HieKSbnE7h732gaRqSk5Mxbdo0BAcHo2fPns4uiW4wKSVyc3OVzjG5L+aXVMcMk9qYX3fB2cuJrsOZM2dQo0YNVK1aFYsWLYKnJz9SRERERER0GUcIRFdxrdnfY2Nj+e0kERERERGViqeXk9OZ77NNpCIhBPz9/XktFimJ+SXVMcOkMimZX3fBI93kVEIIeHh4OLsMonITQiAwMNDZZRCVC/NLqmOGSW3Mr7vgIUZyKiklioqKeIo2KUtKifT0dGaYlMT8kuqYYVKZEMyvu+Cgm5yOOxpSmZQSBQUFzDEpifkl1THDpDbm111w0E1ERERERETkIBx0ExERERERETkIB93kdBU9e3lsbCxmz55t+VkIgS+//LLMzx88eDB69+59zeX+97//Yfr06fYXeIOdOnUKQggcOHDghr1mamoqIiMj8e+//96w13QWIQQMBgNnHiUlMb+kOmaYVCYl8+su3HP28m9b3LjX6rLHrsWv9aGbPHkypkyZch0F3RhTpkxBYmIiAMDDwwNVq1ZFnz59MG3aNAQEBFiWU3X28oMHD2L9+vWYM2cOAKCwsBAvvfQS1q9fj7///htBQUG488478eqrryI6OtrJ1d544eHhePjhhzF58mR8/PHHzi7HoYQQ8PPzc3YZROXC/JLqmGFSG/PrLnik+yZz/vx5y7/Zs2fDYDBYtY0fP97ZJVopKCgo9bGEhAScP38ep06dwsyZMzF//nyMGzfOahkpJQoLC685gYR5lvObxbvvvov77rvP8gVCTk4O9u3bh4kTJ2Lfvn1YvXo1/vjjD/Ts2dOpdV5t+1yPwsLCay4zZMgQfP7550hPT3dIDTcLTdOQmpoKTdOcXQqR3ZhfUh0zTCoTgvl1F04ddE+ZMgVCCKt/devWdWZJTlepUiXLv6CgIAghrNqWLl2KevXqwcfHB3Xr1sUHH3xgea75NOLVq1fj9ttvh5+fHxo3bozdu3dbljl9+jR69OiBkJAQ+Pv7IyEhAevXr7c8/sMPP6BVq1bw9vZG5cqVMWHCBKvBbseOHfHEE09gzJgxCA8PR+fOnUvti6enJypVqoSqVatiwIABePDBB/H1118DAD799FO0aNECBoMB1apVw4MPPojk5GTLc7dt2wYhBDZs2IDmzZvD29sbO3bswIkTJ9CrVy9ERUUhICAALVu2xKZNm+x6j//55x/0798fwcHBCA0NRa9evXDq1KkyP99kMmHlypXo0aOHpS0oKAgbN25E//79ER8fj1tvvRXvvfce9u7dizNnzthcz7p16xAcHAyTyQQAOHDgAIQQmDBhgmWZ4cOH46GHHrL8vGrVKiQkJMDb2xuxsbF48803rdYZGxuLadOm4eGHH4bBYMAjjzxis/6hQ4eibt26ltq++uorNGvWDD4+PqhZsyYSExOttrsQAnPmzEHPnj3h7++PV155BRkZGXjwwQcREREBX19f1K5dGwsXLrQ8JyEhAdHR0VizZk2Z31tV3UxfCBHZi/kl1THDpDLm1z04/Ui3+Wio+d+OHTucXdJN6/PPP8ekSZPwyiuv4OjRo5g+fTomTpyIxYsXWy334osvYvz48Thw4ADq1KmDgQMHWj7Qjz/+OPLz8/Hjjz/i8OHDmDlzpuVo7dmzZ9GtWze0bNkSBw8exJw5c/Dxxx/j5Zdftlr/4sWLodfrsXPnTsydO7fM9fv6+lqOvBYWFmLatGk4cOAAVq5ciVOnTmHw4MElnjNhwgS8+uqrOHr0KBo1aoSsrCx069YNmzdvxv79+9GlSxf06NGj1IHtlQoLC9G5c2cEBgZi+/bt2LlzJwICAtClS5cyHxU+dOgQMjMz0aLF1S9TyMzMhBACwcHBNh9v164dLl68iP379wO49IVHeHg4tm3bZlnmhx9+QMeOHQEAe/fuRf/+/XH//ffj8OHDmDJlCiZOnIhFixZZrfeNN95A48aNsX//fkycONHqsfz8fNx33304cOAAtm/fjurVq2P79u14+OGH8dRTT+HIkSOYN28eFi1ahFdeecXquVOmTEGfPn1w+PBhDB06FBMnTsSRI0ewYcMGHD16FHPmzEF4eLjVc1q1aoXt27df9X0iIiIiInJlTr+m23w0lK5t8uTJePPNN9G3b18AQI0aNSyDpEGDBlmWGz9+PLp37w4ASExMREJCAo4fP245snnvvfeiYcOGAICaNWtanvfBBx+gWrVqeO+99yxnHZw7dw7PPfccJk2aZJnwrHbt2njttdfsqn3v3r1YsmQJ7rjjDgDA0KFDAVw6bbx69ep4++230apVK2RlZVld8z116lTcddddlp9DQ0PRuHFjy8/Tpk3DmjVr8PXXX+OJJ564Zh3Lli2Dpmn46KOPLNfPL1y4EMHBwdi2bRvuvvvua67j9OnT8PDwQGRkZKnL5OXl4bnnnsPAgQNhMBhsLhMUFIQmTZpg27ZtaNGiBbZt24ann34aiYmJyMrKQmZmJo4fP44OHToAAGbNmoVOnTpZBtJ16tTBkSNH8Prrr1t9YXHHHXdYncZvPoqflZWF7t27Iz8/H1u3bkVQUBCASxmZMGGCJUM1a9bEtGnT8Oyzz2Ly5MmW9TzwwAMYMmSI5eczZ86gadOmli8fYmNjS/QxOjra8qUCEREREZE7cvqg+6+//kJ0dDR8fHzQunVrzJgxA9WrV7e5bH5+PvLz8y0/G41GAJeu5zFfC2E+TV1KafXP/Jita4cFAFtXFNvTftVlbb1mabUUay/+35ycHJw4cQLDhg3DiBEjLMsXFRUhKCjIqp/mAbWU0vKFRlJSEuLj4zF69Gg89thj+P7779GpUyfce++9aNSoEYQQOHr0KFq3bm15rhACbdq0QVZWFv755x/LdmnevHmZaj98+DACAgJgMplQUFCA7t27491334WUEnv37kViYiIOHjyIjIwMy/Y7ffo0EhISLOsp/lpCCFy8eBFTpkzB+vXrcf78eRQVFSE3NxenT5+2qsnWeyilxMGDB3H8+HEEBgZa1Z6Xl4fjx49bDfDN78GVfc3JyYG3t7fV+ou/BwUFBejfvz+klPjggw9KXQ8AdOjQAdu2bcPYsWOxfft2TJ8+HcuXL8eOHTuQlpaG6Oho1KpVC1JKHD16FL169bJaT5s2bTB79mwUFRVZJqQzv2fFPwcAMHDgQFStWhVbtmyBj4+Ppf3gwYPYuXOn1ZFtk8mEvLw8ZGdnw9/f3+Z2f+yxx3Dvvfdi3759uOuuu9C7d2+0adPGqq8+Pj7Iycmx+R4U//8rr2XS6XRWtZen/cp9wfW2X1lj8eXNn0FN00pdXqU+lbWdfVK/TwCs8usKfXLF7cQ+lV4jAJv7YJX75IrbSQiJ4n8pSykAXK3dukb723UA5P+vv7zt4v/Xb7u9IvokpSixD2b21O5TaZw66L7llluwaNEixMfH4/z580hMTES7du3w22+/lRgUAcCMGTMsM2IXl5KSgry8PACXTmEOCgpCVlYWNE1DUVERioqKoNPp4OHhAZPJBF3xNxgCEDYGT1dplzbaIQQgAYkrl7+0rPm6XTMvLy+rLwsurULA09PT0m5+zGQyISsrCwAwZ84ctGrVCgAsfZJSWvoJwDL4KioqsryuebKy4cOH44477sCGDRuwceNGvPrqq3jjjTfw5JNPWl7TvB5PT09LP83rl1LC39//mn3SNA116tTBmjVr4OPjg6ioKHh6XopbZmYmunTpgs6dO+OTTz5BWFgY/vnnH3Tv3t2yHc199/b2ttp+48aNw+bNm/Hqq68iLi4OAQEB6N+/P/Lz80tcE2N+X8zvYVFRES5evIjmzZuXOCXby8sLYWFhKCoqsnofbG2n0NBQ5OTkIC8vz9In8/aTUqJ///44deoUvv/+e/j5+UHTNEv2iudGp9OhY8eOWLBgAfbu3QsvLy/UqlXLMhBPS0tDu3btrPpVvE/mfl3Z7uvra6m9eHuXLl2wZMkS7N69Gx06dLD0KSsrC5MnT0a/fv1gMpms+url5WX5fx8fH8u6dDodunbtihMnTmD9+vXYvHkz7rzzTjz22GOYNWuWZbm0tDSEh4dDykuD7uK1m/OkaZrV9fwAEBkZCZPJhLS0NKv3NyoqCgUFBcjIyLC0e3p6Ijw8HLm5uZYv4gBAr9cjNDQUWVlZyM7OtrSb9xFGoxG5ubmWdn9/fwQGBiIjI8PqUgODwQA/Pz+kp6db1R8SEgJvb2+kpqZabdewsDB4eHgo3aeUlBT2yU36VFhYiMzMTJfqkytuJ/ap9D5dvHjR5frkitspMtIIg+Fyn9LT/ZGWFojo6Az4+V3uU1KSAUajH6pXT4def7lPZ8+GICfHGzVrpkCnu9yn06fDUFjogVq1rPt0/HgkvLxMiIm53CdNEzhxIgp+fgWoUuVynwoKPHH6dDgMhlxERV3uU06OHmfPhiIsLAuhoZf7ZDT6Iimp4vpkNBpvmu3kitlzdJ9SUlJQFk4ddHft2tXy/40aNcItt9yCmJgYLF++HMOGDSux/PPPP4+xY8dafjYajahWrRoiIiIsp/Cav/UMCAhASkoKPD09rQZGHh4elwbIVxA22kprF5ceKNEmxf8P1m2so3gNZjqdzuY9qs3t5sc8PDwQFRWF6OhonD59Gg8//LDVus0BML+G+XnF++7h4WHpS40aNTBq1CiMGjUKzz//PD766COMHj0a9evXx+rVqy3LCiGwa9cuBAYGIjY2FjqdzrKOa/VJp9PB29sb8fHxVjUCwPHjx5GWloZXX30VVatWRVFRkeUe0ubabfUBAHbv3o1BgwahX79+AC4NGE+dOoUOHTqUqKd4jR4eHvD09ETz5s2xfPlyREdHW532XbxGc/3F38/i26lZs2YAgGPHjqFJkyaW9sLCQgwYMADHjx/Hli1bEBERYVWPrVujma/rfu+99yx96NixI2bOnImMjAyMHTvWUke9evWwa9cuq37+9NNPqFOnjlVb8dqLvwejRo1Cw4YN0bNnT6xbt85y2nqzZs3w559/Wo6ol8b8HhZXqVIlDB06FEOHDsW8efPw7LPPYtasWZbljhw5gg4dOlhyU/z5np6eEEJAp9OVOFXfnDVbp/Dr9Xqb7b6+vvDx8bH8XHxfYD5aX7zdYDBYfblnbg8JCSnxzSdw6cuW4sztYWFhSE1NRXh4uKVuW7Wr1Kcrs8s+uW6fzJ9Jc35doU+uuJ3Yp9L7FBAQgNzcXKt9sOp9csXtlJxsQHLy5T5dOvoLnDsXgpJHhYEzZ6z7ZG7/++8IG+0Cx49HXtGuQ0FByXbg0mDaVrvR6IuLF32KtVx6zbS0AKSn+5dor4g+CaFBSmm1D2b21OrTle2lcfrp5cUFBwejTp06OH78uM3Hvb29Laf1Fmdr8Gp+I8z/irfbUtrdse1pL3VZOwb0xduv/G9iYiJGjx6N4OBgdOnSBfn5+dizZ49lcGbreVf+/5gxY9C1a1fUqVMHGRkZ2LZtG+rVqwfg0iRrb7/9NkaPHo0nnngCf/zxB6ZMmYKxY8eWGDDaW3vx5WNiYqDX6/Huu+9i5MiROHjwoGWytqv1Abh0PfmaNWvQs2dPCCEwceJEq9PJrlaHEAIPPvggXn/9dfTu3RtTp05F1apVcfr0aaxevRrPPvssqlatWuo6zCIjI9GsWTPs3LkTTZs2BXBpwH3fffdh3759WLduHTRNQ1JSEoBLH169Xm/zPQsJCUGjRo3w+eefW66l79ChAwYMGIDCwkJ07NjR8rxx48ahZcuWePnllzFgwADs3r0b77//Pj744IMSfS8tA6NHj4amaejRowc2bNiA2267DZMmTcI999yDmJgY9OvXDzqdDgcPHsRvv/1mNYnele/xpEmT0Lx5cyQkJCA/Px/ffPONJUtCCOTk5GDv3r2YPn26zfey+P/b+vLpytdzdrutGou3X7kfcoU+laWdfVK/T+bnFH+e6n1yxe3EPlVMhlXpk6ttJ/PguOzttmu3r11YBryOaK+YPlkf8CmO2VO3TzaXK9NSN0hWVhZOnDiBypUrO7uUm9Lw4cPx0UcfYeHChWjYsCE6dOiARYsWoUaNGmVeh8lkwuOPP4569eqhS5cuqFOnjuW2Y1WqVMH69evxyy+/oHHjxnj00UcxbNgwvPTSSxXaj4iICCxatAgrVqxAQkICXn/9dbz++utleu6sWbMQEhKCNm3aoEePHujcubPlyHNZ+Pn54ccff0T16tXRt29f1KtXD8OGDUNeXl6pE57ZMnz4cHz++eeWn8+ePYuvv/4a//77L5o0aYLKlStb/u3ateuq6+rQoQNMJhM6duwI4NIgvX79+qhUqRLi4+MtyzVr1gzLly/H0qVL0aBBA0yaNAlTp061Oev71YwZMwaJiYno1q0bdu3ahc6dO2PdunX4/vvv0bJlS9x666146623EBMTc9X16PV6PP/882jUqBHat28PDw8PLF261PL4V199herVq6Ndu3Z21UdERERE5EqEvNr5pA42fvx49OjRAzExMTh37hwmT56MAwcO4MiRI2U6VG80GhEUFITMzMwSA6a8vDycPHkSNWrUsDoFgW4u5muOzacaqyI3Nxfx8fFYtmyZZfI5snbrrbdi9OjReOCBB2w+7iqfUfM16ZGRkWX+tpPoZsH8kuqYYTVc4y6rbksIDWvXMr8qu9p4tDinnl7+77//YuDAgUhLS0NERARuu+02/PTTT2U+N55cg61rnW92vr6++OSTT5CamursUm5Kqamp6Nu3LwYOHOjsUhxOCIGwsDClvjQiMmN+SXXMMKlMSubXXTh10F38VFRyT+adjIo7G/Pp4FRSeHg4nn32WWeXcUMIIawmKiRSCfNLqmOGSW3Mr7vgeQzkVObTy514lQPRdTGf2ljW+zQS3UyYX1IdM0wqE4L5dRccdBMRERERERE5CAfdRERERERERA7i8oNunq5BdHPiZ5OIiIiI3IFTJ1JzJL1eD51Oh3PnziEiIgJ6vZ6TFNykpJQwmUzOLoNuECklCgoKkJKSAp1OB71e7+ySrotOp+OtPkhZzC+pjhkmlUnJ/LoLlx1063Q61KhRA+fPn8e5c+ecXQ5dhZSSX4i4IT8/P1SvXl35XzTmL42EEMwxKYf5JdUxw6Q25tdduOygG7h0tLt69eooKirikdSblKZpSEtLQ1hYmPKDLyo7Dw8PeHp6usQvGCkl0tLSEBkZ6RL9IffC/JLqmGFSmRDMr7tw6UE3cOn+jV5eXvDy8nJ2KWSDpmnw8vKCj48PB91ERERERORyOMohIiIiIiIichAOusnpeDoNqY4ZJpUxv6Q6ZphUxvy6B5c/vZxubjqdDlFRUc4ug6jcmGFSGfNLqmOGSWVSMr/ugke6yamklMjPz4eU0tmlEJULM0wqY35JdcwwqY35dRccdJNTSSmRkZHBnQ0pixkmlTG/pDpmmFQmBPPrLjjoJiIiIiIiInIQDrqJiIiIiIiIHISDbnI6T0/O50dqY4ZJZcwvqY4ZJpUxv+6BW5mcSqfTITw83NllEJUbM0wqY35JdcwwqUxK5tdd8Eg3OZWUEjk5OZxAgpTFDJPKmF9SHTNMamN+3QUH3eRUUkoYjUbubEhZzDCpjPkl1THDpDIhmF93wUE3ERERERERkYNw0E1ERERERETkIBx0k1MJIaDX6yGEcHYpROXCDJPKmF9SHTNMamN+3QVnLyenEkIgNDTU2WUQlRszTCpjfkl1zDCpTErm113wSDc5lZQSFy9e5AQSpCxmmFTG/JLqmGFSG/PrLjjoJqeSUiI7O5s7G1IWM0wqY35JdcwwqUwI5tddcNBNRERERERE5CAcdBMRERERERE5CAfd5FRCCPj6+nLWRlIWM0wqY35JdcwwqY35dRecvZycSgiBoKAgZ5dBVG7MMKmM+SXVMcOkMimZX3fBI93kVFJKZGZmcgIJUhYzTCpjfkl1zDCpTAjm111w0E1OJaVEbm4udzakLGaYVMb8kuqYYVIb8+suOOgmIiIiIiIichAOuomIiIiIiIgchINuciohBPz9/TlrIymLGSaVMb+kOmaYVCYl8+suOHs5OZUQAoGBgc4ug6jcmGFSGfNLqmOGSW3Mr7vgkW5yKikl0tPTOYEEKYsZJpUxv6Q6ZphUJgTz6y446CanklKioKCAOxtSFjNMKmN+SXXMMKmN+XUXHHQTEREREREROQgH3UREREREREQOwkE3OZUQAgaDgbM2krKYYVIZ80uqY4ZJZVIyv+6Cs5eTUwkh4Ofn5+wyiMqNGSaVMb+kOmaY1Mb8ugse6San0jQNqamp0DTN2aUQlQszTCpjfkl1zDCpTAjm111w0E1OV1RU5OwSiK4LM0wqY35JdcwwqYz5dQ8cdBMRERERERE5CAfdRERERERERA7CQTc5lRACISEhnLWRlMUMk8qYX1IdM0wqk5L5dRecvZycSggBb29vZ5dBVG7MMKmM+SXVMcOkNubXXfBINzmVpmlISkrirI2kLGaYVMb8kuqYYVKZEMyvu+Cgm5xOSunsEoiuCzNMKmN+SXXMMKmM+XUPHHQTEREREREROQgH3UREREREREQOwkE3OZUQAmFhYZy1kZTFDJPKmF9SHTNMKpOS+XUXHHSTUwkh4OHhwZ0NKYsZJpUxv6Q6ZpjUxvy6Cw66yak0TUNycjJnbSRlMcOkMuaXVMcMk8qEYH7dBQfdRERERERERA7CQTcRERERERGRg3DQTUREREREROQgHHSTU+l0OkRGRkKnYxRJTcwwqYz5JdUxw6QyKZlfd8EtTE4lpYTJZIKU0tmlEJULM0wqY35JdcwwqY35dRccdJNTSSmRlpbGnQ0pixkmlTG/pDpmmFQmBPPrLjjoJiIiIiIiInIQDrqJiIiIiIiIHISDbnI6IYSzSyC6LswwqYz5JdUxw6Qy5tc9eDq7AHJvOp0OUVFRzi6DqNyYYVIZ80uqY4ZJZVIyv+6CR7rJqaSUyM/P5wQSpCxmmFTG/JLqmGFSG/PrLjjoJqeSUiIjI4M7G1IWM0wqY35JdcwwqUwI5tddcNBNRERERERE5CAcdBMRERERERE5CAfd5HSenpzPj9TGDJPKmF9SHTNMKmN+3QO3MjmVTqdDeHi4s8sgKjdmmFTG/JLqmGFSmZTMr7vgkW5yKiklcnJyOIEEKYsZJpUxv6Q6ZpjUxvy6Cw66yamklDAajdzZkLKYYVIZ80uqY4ZJZUIwv+6Cg24iIiIiIiIiB+Ggm4iIiIiIiMhBOOgmpxJCQK/XQwjh7FKIyoUZJpUxv6Q6ZpjUxvy6C85eTk4lhEBoaKizyyAqN2aYVMb8kuqYYVKZlMyvu+CRbnIqKSUuXrzICSRIWcwwqYz5JdUxw6Q25tddcNBNTiWlRHZ2Nnc2pCxmmFTG/JLqmGFSmRDMr7vgoJuIiIiIiIjIQTjoJiIiIiIiInIQDrrJqYQQ8PX15ayNpCxmmFTG/JLqmGFSG/PrLm6aQferr74KIQTGjBnj7FLoBhJCICgoiDsbUhYzTCpjfkl1zDCpTErm113cFIPuX3/9FfPmzUOjRo2cXQrdYFJKZGZmcgIJUhYzTCpjfkl1zDCpTAjm1104fdCdlZWFBx98EB9++CFCQkKcXQ7dYFJK5ObmcmdDymKGSWXML6mOGSa1Mb/uwumD7scffxzdu3fHnXfe6exSiIiIiIiIiCqUpzNffOnSpdi3bx9+/fXXMi2fn5+P/Px8y89GoxEAoGkaNE0DcOnaHiEEpJRW3xpdq938/PK263S6Euu2t728tbtCn8q6/VTqU1nb2Sf1+2Qrw6r3qSzt7JPr9Kn4c1ylT9dqZ59cq0/8O+Lm7pMQEsDldikFgKu1W9dof7sOgPz/9Ze3Xfz/+m23V0yfUGIfzOyp3afSOG3Q/c8//+Cpp57Cxo0b4ePjU6bnzJgxA4mJiSXaU1JSkJeXBwDw9fVFUFAQjEYjcnNzLcv4+/sjMDAQGRkZKCgosLQbDAb4+fkhPT0dRUVFlvaQkBB4e3sjJSXFaoOEhYXBw8MDycnJVjVERkbCZDIhLS3N0iaEQFRUFAoKCpCRkWFp9/T0RHh4OHJzcy1fHACAXq9HaGgosrKykJ2dbWl35T5dvHgReXl5SElJgRDCJfrkituJfSq9T6mpqVYZdoU+ueJ2Yp9s96mwsNAqv67QJ1fcTuwT/45QvU+RkUYYDJf7lJ7uj7S0QERHZ8DP73KfkpIMMBr9UL16OvT6y306ezYEOTneqFkzBTrd5T6dPh2GwkIP1Kpl3afjxyPh5WVCTMzlPmmawIkTUfDzK0CVKpf7VFDgidOnw2Ew5CIq6nKfcnL0OHs2FGFhWQgNvdwno9EXSUkV1Sc98vPzLfkFmD3V+pSSkoKyEPLKrxBukC+//BJ9+vSBh4eHpc1kMkEIAZ1Oh/z8fKvHANtHuqtVq4aMjAwYDAYA/KaGfWKf2Cf2iX1in9gn9ol9Yp9upj61bCnBI922a//ll5tnO7li9hzdpwsXLiAkJASZmZmW8agtTht0X7x4EadPn7ZqGzJkCOrWrYvnnnsODRo0uOY6jEYjgoKCrtlJunlJKZGRkYGQkBAIwdslkHqYYVIZ80uqY4bV0KKFsyu4OQkh8d13zK/Kyjoeddrp5YGBgSUG1v7+/ggLCyvTgJtcg5QSBQUFkFJyZ0NKYoZJZcwvqY4ZJrUxv+7C6bOXExEREREREbkqp85efqVt27Y5uwQiIiIiIiKiCsMj3eRUQggYDAaeUkPKYoZJZcwvqY4ZJpVJyfy6i5vqSDe5HyEE/Pz8nF0GUbkxw6Qy5pdUxwyT2phfd8Ej3eRUmqYhNTW1zDeWJ7rZMMOkMuaXVMcMk8qEYH7dBQfd5HTFb0BPpCJmmFTG/JLqmGFSGfPrHjjoJiIiIiIiInIQDrqJiIiIiIiIHISDbnIqIQRCQkI4ayMpixkmlTG/pDpmmFQmJfPrLjh7OTmVEALe3t7OLoOo3JhhUhnzS6pjhkltzK+74JFucipN05CUlMRZG0lZzDCpjPkl1THDpDIhmF93wUE3OZ2U0tklEF0XZphUxvyS6phhUhnz6x446CYiIiIiIiJyEA66iYiIiIiIiByEg25yKiEEwsLCOGsjKYsZJpUxv6Q6ZphUJiXz6y446CanEkLAw8ODOxtSFjNMKmN+SXXMMKmN+XUX1zXozs/Pr6g6yE1pmobk5GTO2kjKYoZJZcwvqY4ZJpUJwfy6C7sG3Rs2bMCgQYNQs2ZNeHl5wc/PDwaDAR06dMArr7yCc+fOOapOIiIiIiIiIuWUadC9Zs0a1KlTB0OHDoWnpyeee+45rF69Gt999x0++ugjdOjQAZs2bULNmjXx6KOPIiUlxdF1ExEREREREd30PMuy0GuvvYa33noLXbt2hU5Xcpzev39/AMDZs2fx7rvv4rPPPsPTTz9dsZUSERERERERKUZIhe/IbjQaERQUhMzMTBgMBmeXQ+WkaZrNL3OIVMEMk8qYX1IdM3zza9HC2RXcvH75hflVWVnHo9e9hU0mEw4cOICMjIzrXRW5ISklTCYTFP7uh9wcM0wqY35JdcwwqY35dRd2D7rHjBmDjz/+GMClAXeHDh3QrFkzVKtWDdu2bavo+sjFSSmRlpbGnQ0pixkmlTG/pDpmmFQmBPPrLuwedK9cuRKNGzcGAKxduxYnT57EsWPH8PTTT+PFF1+s8AKJiIiIiIiIVGX3oDs1NRWVKlUCAKxfvx733XefZWbzw4cPV3iBRERERERERKqye9AdFRWFI0eOwGQy4dtvv8Vdd90FAMjJyYGHh0eFF0iuTwjh7BKIrgszTCpjfkl1zDCpjPl1D2W6ZVhxQ4YMQf/+/VG5cmUIIXDnnXcCAH7++WfUrVu3wgsk16bT6RAVFeXsMojKjRkmlTG/pDpmmFQmJfPrLuwedE+ZMgUNGjTAP//8g/vuuw/e3t4AAA8PD0yYMKHCCyTXJqVEQUEB9Ho9v+kjJTHDpDLml1THDJPaJPLzmV93YPegGwD69etXom3QoEHXXQy5HyklMjIyEBkZyZ0NKYkZJpUxv6Q6ZphUJgTz6y7KNOh+5513yrzC0aNHl7sYIiIiIiIiIldSpkH3W2+9ZfVzSkoKcnJyEBwcDAC4cOEC/Pz8EBkZyUE3ERERERER0f8r0+zlJ0+etPx75ZVX0KRJExw9ehTp6elIT0/H0aNH0axZM0ybNs3R9ZIL8vQs11UORDcNZphUxvyS6phhUhnz6x6ElFLa84S4uDisXLkSTZs2tWrfu3cv+vXrh5MnT1ZogVdjNBoRFBSEzMxMGAyGG/a6RERERERUNi1aOLuCm9eePc6ugK5HWcejdt+n+/z58ygqKirRbjKZkJSUZO/qyM1JKZGTkwM7v/shumkww6Qy5pdUxwyT2phfd2H3oLtTp04YOXIk9u3bZ2nbu3cvHnvsMcs9u4nKSkoJo9HInQ0pixkmlTG/pDpmmFQmBPPrLuwedC9YsACVKlVCixYt4O3tDW9vb7Rq1QpRUVH46KOPHFEjERERERERkZLsvnI/IiIC69evx59//oljx44BAOrWrYs6depUeHFEREREREREKiv3dHl16tThQJuumxACer0eQghnl0JULswwqYz5JdUxw6Q25tdd2D3oNplMWLRoETZv3ozk5GRommb1+JYtWyqsOHJ9QgiEhoY6uwyicmOGSWXML6mOGSaVScn8ugu7B91PPfUUFi1ahO7du6NBgwb8Zoaui5QSWVlZCAgIYJZIScwwqYz5JdUxw6Q2iYsXmV93YPege+nSpVi+fDm6devmiHrIzUgpkZ2dDX9/f+5sSEnMMKmM+SXVMcOkMiGYX3dh9+zler0etWrVckQtRERERERERC7F7kH3uHHj8Pbbb/N+ckRERERERETXYPfp5Tt27MDWrVuxYcMGJCQkwMvLy+rx1atXV1hx5PqEEPD19eUpNaQsZphUxvyS6phhUhvz6y7sHnQHBwejT58+jqiF3JAQAkFBQc4ug6jcmGFSGfNLqmOGSWVSMr/uwu5B98KFCx1RB7kpKSWMRiMMBgO/5SMlMcOkMuaXVMcMk8qEkMjMZH7dgd3XdJulpKRgx44d2LFjB1JSUiqyJnIjUkrk5uZyjgBSFjNMKmN+SXXMMKmN+XUXdg+6s7OzMXToUFSuXBnt27dH+/btER0djWHDhiEnJ8cRNRIREREREREpye5B99ixY/HDDz9g7dq1uHDhAi5cuICvvvoKP/zwA8aNG+eIGomIiIiIiIiUZPc13atWrcLKlSvRsWNHS1u3bt3g6+uL/v37Y86cORVZH7k4IQT8/f15HQspixkmlTG/pDpmmFQmJfPrLuwedOfk5CAqKqpEe2RkJE8vJ7sJIRAYGOjsMojKjRkmlTG/pDpmmNTG/LoLu08vb926NSZPnoy8vDxLW25uLhITE9G6desKLY5cn5QS6enpnECClMUMk8qYX1IdM0wqE4L5dRd2H+l+++230blzZ1StWhWNGzcGABw8eBA+Pj747rvvKrxAcm1SShQUFEBKyVNrSEnMMKmM+SXVMcOkNubXXdg96G7QoAH++usvfP755zh27BgAYODAgXjwwQfh6+tb4QUSERERERERqcruQTcA+Pn5YcSIERVdCxEREREREZFLsfua7hkzZmDBggUl2hcsWICZM2dWSFHkPoQQMBgMPKWGlMUMk8qYX1IdM0wqk5L5dRd2D7rnzZuHunXrlmhPSEjA3LlzK6Qoch9CCPj5+XFnQ8pihkllzC+pjhkmtTG/7sLuQfd///2HypUrl2iPiIjA+fPnK6Qoch+apiE1NRWapjm7FKJyYYZJZcwvqY4ZJpUJwfy6C7sH3dWqVcPOnTtLtO/cuRPR0dEVUhS5l6KiImeXQHRdmGFSGfNLqmOGSWXMr3uweyK1ESNGYMyYMSgsLMQdd9wBANi8eTOeffZZjBs3rsILJCIiIiIiIlKV3YPuZ555BmlpaRg1ahQKCgoAAD4+Pnjuuefw/PPPV3iBRERERERERKoSUkpZnidmZWXh6NGj8PX1Re3ateHt7V3RtV2T0WhEUFAQMjMzYTAYbvjr0/WTUqKgoAB6vZ6TSJCSmGFSGfNLqmOG1dCihbMruFlJ7NzJ/KqsrONRu6/pNvvvv/+Qnp6OuLg4eHt7o5xjd3JzQgh4e3tzR0PKYoZJZcwvqY4ZJrUxv+7C7kF3WloaOnXqhDp16qBbt26WGcuHDRvGa7rJbpqmISkpibM2krKYYVIZ80uqY4ZJZUIwv+7C7kH3008/DS8vL5w5cwZ+fn6W9gEDBuDbb7+t0OLIPfAsCVIdM0wqY35JdcwwqYz5dQ92T6T2/fff47vvvkPVqlWt2mvXro3Tp09XWGFEREREREREqrP7SHd2drbVEW6z9PR0p0ymRkRERERERHSzsnvQ3a5dO3zyySeWn4UQ0DQNr732Gm6//fYKLY5cnxACYWFhnECClMUMk8qYX1IdM0wqk5L5dRd2n17+2muvoVOnTtizZw8KCgrw7LPP4vfff0d6ejp27tzpiBrJhQkh4OHhwZ0NKYsZJpUxv6Q6ZpjUxvy6C7uPdDdo0AB//vknbrvtNvTq1QvZ2dno27cv9u/fj7i4OEfUSC5M0zQkJydz1kZSFjNMKmN+SXXMMKlMCObXXdh9pBsAgoKC8OKLL1Z0LUREREREREQuxe4j3d9++y127Nhh+fn9999HkyZN8MADDyAjI6NCiyMiIiIiIiJSmd2D7meeeQZGoxEAcPjwYYwdOxbdunXDyZMnMXbs2AovkIiIiIiIiEhVdp9efvLkSdSvXx8AsGrVKvTo0QPTp0/Hvn370K1btwovkFybTqdDZGQkdDq7v/8huikww6Qy5pdUxwyTyqRkft2F3VtYr9cjJycHALBp0ybcfffdAIDQ0FDLEXCispJSwmQyQUrp7FKIyoUZJpUxv6Q6ZpjUxvy6C7sH3bfddhvGjh2LadOm4ZdffkH37t0BAH/++SeqVq1a4QWSa5NSIi0tjTsbUhYzTCpjfkl1zDCpTAjm113YPeh+77334OnpiZUrV2LOnDmoUqUKAGDDhg3o0qVLhRdIREREREREpCq7r+muXr061q1bV6L9rbfeqpCCiIiIiIiIiFxFmY50Z2dn27VSe5cn9yaEcHYJRNeFGSaVMb+kOmaYVMb8uocyDbpr1aqFV199FefPny91GSklNm7ciK5du+Kdd96psALJtel0OkRFRXHWRlIWM0wqY35JdcwwqUxK5tddlOn08m3btuGFF17AlClT0LhxY7Ro0QLR0dHw8fFBRkYGjhw5gt27d8PT0xPPP/88Ro4c6ei6yUVIKVFQUAC9Xs9v+khJzDCpjPkl1THDpDaJ/Hzm1x2UadAdHx+PVatW4cyZM1ixYgW2b9+OXbt2ITc3F+Hh4WjatCk+/PBDdO3aFR4eHo6umVyIlBIZGRmIjIzkzoaUxAyTyphfUh0zTCoTgvl1F3ZNpFa9enWMGzcO48aNc1Q9RERERERERC6DFxAQERERERG5kPz8fIwYMQI1atRAYGAg6tatiwULFpS6/N69e3HbbbfBYDCgZs2a+OSTTyyP/fnnn+jTpw8qVaqE4OBgtG3bFjt37rwR3XAZTh10z5kzB40aNYLBYIDBYEDr1q2xYcMGZ5ZETuDpafed64huKswwqYz5JdUxw6QyR+W3qKgIlStXxqZNm2A0GrFo0SKMGzcO33//fYllL1y4gG7duuGhhx5CRkYGvvjiCzz55JPYsWOH5fGuXbvi8OHDSEtLw+DBg9GtWzekpqY6pHZXJKSU0lkvvnbtWnh4eKB27dqQUmLx4sV4/fXXsX//fiQkJFzz+UajEUFBQcjMzITBYLgBFRMRERERkT1atHB2BTevPXtu3Gv17dsXDRo0wNSpU63a169fj0cffRRnzpyxtA0ZMgRSSixatMjmukJDQ7Fy5Urccccdjiz5plfW8ahTj3T36NED3bp1Q+3atVGnTh288sorCAgIwE8//eTMsugGklIiJycHTvzuh+i6MMOkMuaXVMcMk9puXH7z8vLwyy+/oFGjRiUe0zStRA2apuHQoUM213X48GFcvHgR9evXd0itruimuabbZDJh6dKlyM7ORuvWrZ1dDt0gUkoYjUb+siRlMcOkMuaXVMcMk8qEuDH5lVJi+PDhqF27Nvr27Vvi8datWyM7OxvvvfceCgsLsXPnTqxZswZGo7HEshcuXMD999+PF154AZUqVXJo3a6kXBcRbN++HfPmzcOJEyewcuVKVKlSBZ9++ilq1KiB2267za51HT58GK1bt0ZeXh4CAgKwZs2aUr81yc/PR35+vuVncxA0TYOmaQAAIQSEEJBSWgX4Wu3m55e3XafTlVi3ve3lrd0V+lTW7adSn8razj6p3ydbGVa9T2VpZ59cp0/Fn+MqfbpWO/vkWn3i3xE3d5+EkAAut0spAFyt3bpG+9t1AOT/r7+87eL/12+7vWL6hBL74IreTpqm4fHHH8cff/yBjRs32lw+NDQUa9euxTPPPIPJkyejfv36GDx4MH7++WdLjVJKZGZmokuXLmjbti2mTJmiRPZu9OepNHYPuletWoX//e9/ePDBB7F//37LIDgzMxPTp0/H+vXr7VpffHw8Dhw4gMzMTKxcuRKDBg3CDz/8YHPgPWPGDCQmJpZoT0lJQV5eHgDA19cXQUFBMBqNyM3NtSzj7++PwMBAZGRkoKCgwNJuMBjg5+eH9PR0FBUVWdpDQkLg7e2NlJQUqw0SFhYGDw8PJCcnW9UQGRkJk8mEtLQ0S5sQAlFRUSgoKEBGRoal3dPTE+Hh4cjNzbX6Bkmv1yM0NBRZWVnIzs62tLt6ny5cuAApJXQ6ncv0yRW3E/tUep+KZ9hV+uSK24l9st2n4vl1lT654nZin/h3hMp9iow0wmC43Kf0dH+kpQUiOjoDfn6X+5SUZIDR6Ifq1dOh11/u09mzIcjJ8UbNminQ6S736fTpMBQWeqBWLes+HT8eCS8vE2JiLvdJ0wROnIiCn18BqlS53KeCAk+cPh0OgyEXUVGX+5STo8fZs6EIC8tCaOjlPhmNvkhKqpg+5eZ6ITMz05JfoGK3U05ODh5//HHs3bsXK1assAwQbW2ntm3bYsOGDZbsjRw50nL2cUZGBlJTUzFw4EDUqVMHb731FoQQSEtLu+mz5+jPU0pKCsrC7onUmjZtiqeffhoPP/wwAgMDcfDgQdSsWRP79+9H165d8d9//9mzuhLuvPNOxMXFYd68eSUes3Wku1q1asjIyLBcuO7K3xK6Yp80TUNGRgaCg4OLfRuqdp9ccTuxT6XXbjKZcOHChRIZVrlPrrid2CfbtV+5D3aFPrnidmKf+HeE6n1q2bIijgq73pFuIYBvv0232gdX5HYaNWoUdu7ciU2bNiEsLOyq2+nAgQOoV68eTCYTPvvsM0yaNAn79u1DlSpVLEe4a9WqhYULF8LDw0OZ7Dn683ThwgWEhIRccyI1uwfdfn5+OHLkCGJjY60G3X///Tfq169vOeJcXnfccQeqV69e6kx5xXH2ciIiIiKimxtnLy+do2YvP336NGJjY+Ht7W11W7KHHnoIc+fORdeuXdGuXTu88MILAC7NVr5mzRoUFRWhTZs2eOuttyx3k1q8eDEGDx4MPz8/y5cDADBv3jw8+OCDjumAIso6HrX79PJKlSrh+PHjiI2NtWrfsWMHatasade6nn/+eXTt2hXVq1fHxYsXsWTJEmzbtg3fffedvWWRoqSUyMrKQkBAgNWHmEgVzDCpjPkl1THDpDaJixcdk9+YmJgSR4qL27Bhg9XPCxcuxMKFC20uO2jQIAwaNKhC63M3ds9ePmLECDz11FP4+eefIYTAuXPn8Pnnn2P8+PF47LHH7FpXcnIyHn74YcTHx6NTp0749ddf8d133+Guu+6ytyxSlJQS2dnZV90pEN3MmGFSGfNLqmOGSWVCML/uwu4j3RMmTICmaejUqRNycnLQvn17eHt7Y/z48XjyySftWtfHH39s78sTERERERERKcPuQbcQAi+++CKeeeYZHD9+HFlZWahfvz4CAgIcUR8RERERERGRssp1n27g0vTspd1Pm6ishBDw9fXldVikLGaYVMb8kuqYYVKbgO++YRAiGRA8xbyELg6aZc4J7B505+Xl4d1338XWrVuRnJxcYvr0ffv2VVhx5PqEEAgKCnJ2GUTlxgyTyphfUh0zTCqTUiBIl+TsMugGsHvQPWzYMHz//ffo168fWrVqxW8W6bpIKWE0GmEwGJglUhIzTCpjfkl1zDCpTAiJTC0KBpFc4t7h5FrsHnSvW7cO69evR9u2bR1RD7kZKSVyc3MRGBjIX5akJGaYVMb8kuqYYVKbRK40IFAkg+l1bXbfMqxKlSoIDAx0RC1ERERERERELsXuQfebb76J5557DqdPn3ZEPUREREREREQuw+7Ty1u0aIG8vDzUrFkTfn5+8PLysno8PT29wooj1yeEgL+/P08JI2Uxw6Qy5pdUxwyTyqQU8BfpEOD13K7O7kH3wIEDcfbsWUyfPh1RUVHcydF1EULwcgVSGjNMKmN+SXXMMKlNIFCX5uwi6Aawe9C9a9cu7N69G40bN3ZEPeRmpJTIyMhASEgIv8AhJTHDpDLml1THDJPKhJBI16ogRJzj7OUuzu5ruuvWrYvc3FxH1EJuSEqJgoICSMkdDamJGSaVMb+kOmaY1CZRIP14crkbsHvQ/eqrr2LcuHHYtm0b0tLSYDQarf4RERERERER0SV2n17epUsXAECnTp2s2qWUEELAZDJVTGVEREREREREirN70L1161ZH1EFuSggBg8HA67BIWcwwqYz5JdUxw6QyKQUMuiTOXu4G7B50d+jQwRF1kJsSQsDPz8/ZZRCVGzNMKmN+SXXMMKlNwE/w8lx3UKZB96FDh9CgQQPodDocOnToqss2atSoQgoj96BpGtLT0xEaGgqdzu4pBoicjhkmlTG/pDpmmFQmhIZULQah4gx0nL3cpZVp0N2kSRP8999/iIyMRJMmTSCEsDlLJK/ppvIoKipydglE14UZJpUxv6Q6ZphUViT1AK+OcHllGnSfPHkSERERlv8nIiIiIiIiomsr06A7JiYGHh4eOH/+PGJiYhxdExEREREREZFLKPPFL7ZOJye6XkIIhISEcNZRUhYzTCpjfkl1zDCpTEqBEN1Zzl7uBuyevZyoIgkh4O3t7ewyiMqNGSaVMb+kOmaY1CbgLXKcXQTdAHYNuj/66CMEBARcdZnRo0dfV0HkXjRNQ0pKCiIiIjjrKCmJGSaVMb+kOmaYVCaEhiRTHCJ0f3P2chdn16B77ty58PDwKPVxIQQH3WQ3XrpAqmOGSWXML6mOGSaVybJf7UsKs2vQvWfPHkRGRjqqFiIiIiIiIiKXUuavVjhBBREREREREZF9OHs5OZUQAmFhYfxSh5TFDJPKmF9SHTNMKpNSIEx3mrOXu4EyD7onT558zUnUiOwlhICHhwd/WZKymGFSGfNLqmOGSW0CHigE4+v67Bp0+/n5ObIWckOapiE5ORmapjm7FKJyYYZJZcwvqY4ZJpUJoSFZqwVNctTt6jhdHhEREREREZGDcNBNRERERERE5CAcdBMRERERERE5CAfd5FQ6nQ6RkZHQ6RhFUhMzTCpjfkl1zDCpTEodInXHoROcvdzV2b2HSkpKwv/+9z9ER0fD09MTHh4eVv+I7CGlhMlk4i3pSFnMMKmM+SXVMcOkNgkTvMD4uj5Pe58wePBgnDlzBhMnTkTlypV5iwa6LlJKpKWlITIyklkiJTHDpDLml1THDJPKhJBI02IQqTvOe3W7OLsH3Tt27MD27dvRpEkTB5RDRERERERE5DrsPr28WrVqPIWHiIiIiIiIqAzsHnTPnj0bEyZMwKlTpxxQDrkjng5GqmOGSWXML6mOGSaVCWjOLoFuALtPLx8wYABycnIQFxcHPz8/eHl5WT2enp5eYcWR69PpdIiKinJ2GUTlxgyTyphfUh0zTCqTUocojxPOLoNuALsH3bNnz3ZAGeSupJQoKCiAXq/nN9WkJGaYVMb8kuqYYVKbRL70gx45YHxdm92D7kGDBjmiDnJTUkpkZGRw1lFSFjNMKmN+SXXMMKlMCIkMrQpnL3cDdg+6AcBkMuHLL7/E0aNHAQAJCQno2bMn79NNREREREREVIzdg+7jx4+jW7duOHv2LOLj4wEAM2bMQLVq1fDNN98gLi6uwoskIiIiIiIiUpHds5ePHj0acXFx+Oeff7Bv3z7s27cPZ86cQY0aNTB69GhH1EguztOzXCdcEN00mGFSGfNLqmOGSWWeosDZJdANYPde6ocffsBPP/2E0NBQS1tYWBheffVVtG3btkKLI9en0+kQHh7u7DKIyo0ZJpUxv6Q6ZphUJqUO4brTzi6DbgC7j3R7e3vj4sWLJdqzsrKg1+srpChyH1JK5OTkQEpOHkFqYoZJZcwvqY4ZJrVJ5EgDGF/XZ/eg+5577sEjjzyCn3/+GVJKSCnx008/4dFHH0XPnj0dUSO5MCkljEYjf1mSsphhUhnzS6pjhkllQkgYtShIcOZ9V2f3oPudd95BXFwcWrduDR8fH/j4+KBt27aoVasW3n77bUfUSERERERERKQku6/pDg4OxldffYW//voLx44dAwDUq1cPtWrVqvDiiIiIiIiIiFRW7ukea9eujdq1a1dkLeSGhBDQ6/UQgqfVkJqYYVIZ80uqY4ZJbQJ6kcOTy91AmQbdY8eOxbRp0+Dv74+xY8deddlZs2ZVSGHkHoQQVjPhE6mGGSaVMb+kOmaYVCalQKjurLPLoBugTIPu/fv3o7Cw0PL/RBVFSomsrCwEBATwW2pSEjNMKmN+SXXMMKlN4qIWhgCRBsbXtZVp0L1161ab/090vaSUyM7Ohr+/P39ZkpKYYVIZ80uqY4ZJZUJIZMtQ+It0CHAGfldm9+zlQ4cOtXmf7uzsbAwdOrRCiiIiIiIiIiJyBXYPuhcvXozc3NwS7bm5ufjkk08qpCgiIiIiIiIiV1Dm2cuNRiOklJBS4uLFi/Dx8bE8ZjKZsH79ekRGRjqkSHJdQgj4+vrylDBSFjNMKmN+SXXMMKlNwFcYOXu5GyjzoDs4OBhCCAghUKdOnRKPCyGQmJhYocWR6xNCICgoyNllEJUbM0wqY35JdcwwqUxKgSBdkrPLoBugzIPurVu3QkqJO+64A6tWrbK6PYNer0dMTAyio6MdUiS5LikljEYjDAYDv6UmJTHDpDLml1THDJPKhJDI1KJgEMkQghOpubIyD7o7dOgAADh58iSqVasGnc7uy8GJSpBSIjc3F4GBgfxlSUpihkllzC+pjhkmtUnkSgMCRTJPMXdxZR50m8XExAAAcnJycObMGRQUFFg93qhRo4qpjIiIiIiIiEhxdg+6U1JSMGTIEGzYsMHm4yaT6bqLIiIiIiIiInIFdp8jPmbMGFy4cAE///wzfH198e2332Lx4sWoXbs2vv76a0fUSC5MCAF/f3+eEkbKYoZJZcwvqY4ZJpVJKeAv0iHA67ldnd1Hurds2YKvvvoKLVq0gE6nQ0xMDO666y4YDAbMmDED3bt3d0Sd5KKEEAgMDHR2GUTlxgyTyphfUh0zTGoTCNSlObsIugHsPtKdnZ1tuR93SEgIUlJSAAANGzbEvn37KrY6cnlSSqSnp0NKfsNHamKGSWXML6mOGSaVCSGRrlWBlDxTw9XZPeiOj4/HH3/8AQBo3Lgx5s2bh7Nnz2Lu3LmoXLlyhRdIrk1KiYKCAv6yJGUxw6Qy5pdUxwyT2iQKpB9PLncDdp9e/tRTT+H8+fMAgMmTJ6NLly74/PPPodfrsWjRooquj4iIiIiIiEhZdg+6H3roIcv/N2/eHKdPn8axY8dQvXp1hIeHV2hxRERERERERCqz+/TyqVOnIicnx/Kzn58fmjVrBn9/f0ydOrVCiyPXJ4SAwWDgrKOkLGaYVMb8kuqYYVKZlAIGXRJnL3cDdg+6ExMTkZWVVaI9JycHiYmJFVIUuQ8hBPz8/PjLkpTFDJPKmF9SHTNMahPwE0Ywvq7P7kG3lNLmju3gwYMIDQ2tkKLIfWiahtTUVGia5uxSiMqFGSaVMb+kOmaYVCaEhlQtBhpnL3d5Zb6mOyQkBEIICCFQp04dq4G3yWRCVlYWHn30UYcUSa6tqKjI2SUQXRdmmFTG/JLqmGFSWZHUAxxzu7wyD7pnz54NKSWGDh2KxMREBAUFWR7T6/WIjY1F69atHVIkERERERERkYrKPOgeNGgQAKBGjRpo06YNvLy8HFYUERERERERkSuw+5ZhNWrUsNyn25bq1atfV0HkXoQQlksXiFTEDJPKmF9SHTNMKpNSIER3lrOXuwG7B92xsbFX3bGZTKbrKojcixAC3t7ezi6DqNyYYVIZ80uqY4ZJbQLeIufai5Hy7B5079+/3+rnwsJC7N+/H7NmzcIrr7xSYYWRe9A0DSkpKYiIiIBOZ/dk+kROxwyTyphfUh0zTCoTQkOSKQ4Rur+hEzza7crsHnQ3bty4RFuLFi0QHR2N119/HX379q2Qwsh9SMmdDKmNGSaVMb+kOmaYVCbtv4MzKajCtnJ8fDx+/fXXilodERERERERkfLsPtJtNBqtfpZS4vz585gyZQpq165dYYURERERERERqc7uQXdwcHCJidSklKhWrRqWLl1aYYWRexBCICwsjLOOkrKYYVIZ80uqY4ZJZVIKhOlOc/ZyN2D3oHvr1q1WP+t0OkRERKBWrVrw9LR7deTmhBDw8PDgL0tSFjNMKmN+SXXMMKlNwAOFYHxdn92j5A4dOjiiDnJTmqYhOTkZkZGRnHWUlMQMk8qYX1IdM0wqE0JDslYLkbrjnL3cxZXr0PQff/yBd999F0ePHgUA1KtXD0888QTq1q1bocURERERERERqczurwRXrVqFBg0aYO/evWjcuDEaN26Mffv2oWHDhli1apVd65oxYwZatmyJwMBAREZGonfv3vjjjz/sLYmIiIiIiIjopmT3ke5nn30Wzz//PKZOnWrVPnnyZDz77LO49957y7yuH374AY8//jhatmyJoqIivPDCC7j77rtx5MgR+Pv721saERERERER0U1FSCntuoDAz88Phw4dQq1ataza//rrLzRu3Bg5OTnlLiYlJQWRkZH44Ycf0L59+2subzQaERQUhMzMTBgMhnK/LjmXpmm8DouUxgyTyphfUh0zfPNr0cLZFdy8fpnWktdzl6bLHmdXcE1lHY/afaS7Y8eO2L59e4lB944dO9CuXTv7Ky0mMzMTABAaGmrz8fz8fOTn51t+Nt8zXNM0aJoG4NIslkIISClR/PuEa7Wbn1/edp1OV2Ld9raXt3aV+6RpGgoLC+Hp6WlpU71Prrid2KfSazeZTCgqKiqRYZX75IrbiX2yXfuV+2BX6JMrbif2iX9HqN4nISRQ7LZYUgoAV2u3rtH+dh0A+f/rL2+7+P/1226vmD4BhVIPT1mA/98FQ0BCCECT1lOa64SElIBE+dsFAHHVduubl5lrKa39yhrtbb9m7WXI6s32eSqN3YPunj174rnnnsPevXtx6623AgB++uknrFixAomJifj666+tli0rTdMwZswYtG3bFg0aNLC5zIwZM5CYmFiiPSUlBXl5eQAAX19fBAUFwWg0Ijc317KMv78/AgMDkZGRgYKCAku7wWCAn58f0tPTUVRUZGkPCQmBt7c3UlJSrDZIWFgYPDw8kJycbFVDZGQkTCYT0tLSLG1CCERFRaGgoAAZGRmWdk9PT4SHhyM3N9fyxQEA6PV6hIaGIisrC9nZ2ZZ2V+5TZmYm/vvvPwQFBUGn07lEn1xxO7FPpfcpOTkZFy5csGTYFfrkituJfbLdp/z8fJw6dcqSX1fokytuJ/aJf0eo3qfISCMMhst9Sk/3R1paIKKjM+Dnd7lPSUkGGI1+qF49HXr95T6dPRuCnBxv1KyZAp3ucp9Onw5DYaEHatWy7tPx45Hw8jIhJuZynzRN4MSJKPj5FaBKlct9KijwxOnT4TAYchEVdblPOTl6nD0birCwLISGXu6T0eiLpKSK6VNurhdOmm6BQfcfdP+/rcJ0p+EhC5GsWR/gjNQdhwleSNNiLG0CGqI8TqAAfsjQqljaPUUBwsVp5MIAoxZladeLHISKs8iSYciWlw9y+gojgkQSjDISufLy0Vp/kY5AkYYMGY0C6WdpN+iS4Acj0mV1FEm9pT1EdxbeyEGKVhOy2NRh5e5TsazerJ+nlJQUlIXdp5eX9fQdIS4dASqrxx57DBs2bMCOHTtQtWpVm8vYOtJdrVo1ZGRkWA7nu/K3hK7YJ5PJhOTkZERERECn07lEn1xxO7FPpddeVFSElJSUEhlWuU+uuJ3YJ9u1X7kPdoU+ueJ2Yp/4d4TqfWrZsrSjv6W1u8eRbiEkvp78ECKK3TKMR7qLtXf+xbr9Jvw8XbhwASEhIRV/enlZD6Hb44knnsC6devw448/ljrgBgBvb294e3uXaNfpdCW+DDC/EVcqrb20LxPsabf3NR3drlKfrtyGrtCnsrazT+r3yVaGVe9TWdvZJ9foU1n3wSr16Ua3s0/8O4LbqfR288Cz7O22a7ev3Txodkx7xfTp0oBe9///irN1nbcQlwawjmu/cvh79fbSrkW3p/2qNTohqxXxebKlXPfprihSSjz55JNYs2YNtm3bhho1ajizHHISW4EnUgkzTCpjfkl1zDCpTKDiD2jSzadcg+5ff/0VW7duRXJycokj37NmzSrzeh5//HEsWbIEX331FQIDA/Hff/8BAIKCguDr61ue0kgxOp0OUVFR116Q6CbFDJPKmF9SHTNMKpNShyiPE84ug24Auwfd06dPx0svvYT4+HhERUVZfbto7zeNc+bMAXBpRvTiFi5ciMGDB9tbGilISomCggLo9Xp+U01KYoZJZcwvqY4ZJrVJ5Es/6JEDxte12T3ofvvtt7FgwYIKGRRfOTEDuR8pJTIyMhAZGclflqQkZphUxvyS6phhUpkQEhlaFUTqjtu8rplcR9mu/C7+BJ0Obdu2dUQtRERERERERC7F7kH3008/jffff98RtRARERERERG5FLtPLx8/fjy6d++OuLg41K9fH15eXlaPr169usKKI/fg6enUSfSJrhszTCpjfkl1zDCpzFMUOLsEugHs3kuNHj0aW7duxe23346wsDBeP0PXRafTITw83NllEJUbM0wqY35JdcwwqUxKHcJ1p51dBt0Adg+6Fy9ejFWrVqF79+6OqIfcjJQSubm58PX15Rc4pCRmmFTG/JLqmGFSm0SONMAXRs5e7uLsvqY7NDQUcXFxjqiF3JCUEkajkTPZk7KYYVIZ80uqY4ZJZUJIGLUoSHDE7ersHnRPmTIFkydPRk5OjiPqISIiIiIiInIZdp9e/s477+DEiROIiopCbGxsiYnU9u3bV2HFEREREREREanM7kF37969HVAGuSshBPR6Pa/DImUxw6Qy5pdUxwyT2gT0Iocnl7sBuwfdkydPdkQd5KaEEAgNDXV2GUTlxgyTyphfUh0zTCqTUiBUd9bZZdANUO4bG+7duxdHjx4FACQkJKBp06YVVhS5DyklsrKyEBAQwG+pSUnMMKmM+SXVMcOkNomLWhgCRBpnL3dxdg+6k5OTcf/992Pbtm0IDg4GAFy4cAG33347li5dioiIiIqukVyYlBLZ2dnw9/fnL0tSEjNMKmN+SXXMMKlMCIlsGQp/kQ4BzsDvyuyevfzJJ5/ExYsX8fvvvyM9PR3p6en47bffYDQaMXr0aEfUSERERERERKQku490f/vtt9i0aRPq1atnaatfvz7ef/993H333RVaHBEREREREZHK7D7SrWlaiduEAYCXlxc0TauQosh9CCHg6+vLU8JIWcwwqYz5JdUxw6Q2AV9h5OzlbsDuQfcdd9yBp556CufOnbO0nT17Fk8//TQ6depUocWR6xNCICgoiL8sSVnMMKmM+SXVMcOkMikFgnRJEILXc7s6uwfd7733HoxGI2JjYxEXF4e4uDjUqFEDRqMR7777riNqJBcmpURmZiak5M6G1MQMk8qYX1IdM0wqE0IiU4uClPzSyNXZfU13tWrVsG/fPmzatAnHjh0DANSrVw933nlnhRdHrk9KidzcXAQGBvJbalISM0wqY35JdcwwqU0iVxoQKJJ5irmLK9d9uoUQuOuuu3DXXXdVdD1ERERERERELqPMp5dv2bIF9evXh9FoLPFYZmYmEhISsH379gotjoiIiIiIiEhlZR50z549GyNGjIDBYCjxWFBQEEaOHIlZs2ZVaHHk+oQQ8Pf35ylhpCxmmFTG/JLqmGFSmZQC/iIdApyTwNWVedB98OBBdOnSpdTH7777buzdu7dCiiL3IYTgdVikNGaYVMb8kuqYYVKbQKAuDYyv6yvzoDspKcnm/bnNPD09kZKSUiFFkfuQUiI9PZ2zjpKymGFSGfNLqmOGSWVCSKRrVTh7uRso86C7SpUq+O2330p9/NChQ6hcuXKFFEXuQ0qJgoIC/rIkZTHDpDLml1THDJPaJAqkH08udwNlHnR369YNEydORF5eXonHcnNzMXnyZNxzzz0VWhwRERERERGRysp8y7CXXnoJq1evRp06dfDEE08gPj4eAHDs2DG8//77MJlMePHFFx1WKBEREREREZFqyjzojoqKwq5du/DYY4/h+eeft5zGI4RA586d8f777yMqKsphhZJrEkLAYDBwAhRSFjNMKmN+SXXMMKlMSgGDLomzl7uBMg+6ASAmJgbr169HRkYGjh8/DiklateujZCQEEfVRy5OCAE/Pz9nl0FUbswwqYz5JdUxw6Q2AT9hdHYRdAOU+Zru4kJCQtCyZUu0atWKA266LpqmITU1FZqmObsUonJhhkllzC+pjhkmlQmhIVWLgcbZy11euQbdRBWpqKjI2SUQXRdmmFTG/JLqmGFSWZHUO7sEugE46CYiIiIiIiJyEA66iYiIiIiIiByEg25yKiEEQkJCOOsoKYsZJpUxv6Q6ZphUJqVAiO4sZy93A3bNXk5U0YQQ8Pb2dnYZROXGDJPKmF9SHTNMahPwFjnOLoJuAB7pJqfSNA1JSUmcdZSUxQyTyphfUh0zTCoTQkOSKY6zl7sBDrrJ6aTkKTWkNmaYVMb8kuqYYVKZ5HDMLXArExERERERETkIB91EREREREREDsJBNzmVEAJhYWGcdZSUxQyTyphfUh0zTCqTUiBMd5qzl7sBDrrJqYQQ8PDw4C9LUhYzTCpjfkl1zDCpTcADhWB8XR8H3eRUmqYhOTmZs46SsphhUhnzS6pjhkllQmhI1mpx9nI3wEE3ERERERERkYNw0E1ERERERETkIBx0ExERERERETkIB93kVDqdDpGRkdDpGEVSEzNMKmN+SXXMMKlMSh0idcehE5y93NVxD0VOJaWEyWSClNzZkJqYYVIZ80uqY4ZJbRImeIHxdX0cdJNTSSmRlpbGX5akLGaYVMb8kuqYYVKZEBJpWgwkOHu5q+Ogm4iIiIiIiMhBOOgmIiIiIiIichAOusnphOApNaQ2ZphUxvyS6phhUpmA5uwS6AbwdHYB5N50Oh2ioqKcXQZRuTHDpDLml1THDJPKpNQhyuOEs8ugG4BHusmppJTIz8/nBCikLGaYVMb8kuqYYVKbRL704+zlboCDbnIqKSUyMjL4y5KUxQyTyphfUh0zTCoTQiJDq8LZy90AB91EREREREREDsJBNxEREREREZGDcNBNTufpyfn8SG3MMKmM+SXVMcOkMk9R4OwS6AbgXoqcSqfTITw83NllEJUbM0wqY35JdcwwqUxKHcJ1p51dBt0APNJNTiWlRE5ODidAIWUxw6Qy5pdUxwyT2iRypIGzl7sBDrrJqaSUMBqN/GVJymKGSWXML6mOGSaVCSFh1KI4e7kb4KCbiIiIiIiIyEE46CYiIiIiIiJyEA66yamEENDr9RCCp9WQmphhUhnzS6pjhkltAnqRw5PL3QBnLyenEkIgNDTU2WUQlRszTCpjfkl1zDCpTEqBUN1ZZ5dBNwCPdJNTSSlx8eJFToBCymKGSWXML6mOGSa1SVzUwjh7uRvgoJucSkqJ7Oxs/rIkZTHDpDLml1THDJPKhJDIlqGcvdwNcNBNRERERERE5CAcdBMRERERERE5CAfd5FRCCPj6+nLWUVIWM0wqY35JdcwwqU3AVxh5crkb4Ozl5FRCCAQFBTm7DKJyY4ZJZcwvqY4ZJpVJKRCkS3J2GXQD8Eg3OZWUEpmZmZwAhZTFDJPKmF9SHTNMKhNCIlOLgpQ81u3qOOgmp5JSIjc3l78sSVnMMKmM+SXVMcOkNolcaQDT6/o46CYiIiIiIiJyEA66iYiIiIiIiByEg25yKiEE/P39OesoKYsZJpUxv6Q6ZphUJqWAv0iH4AnmLo+zl5NTCSEQGBjo7DKIyo0ZJpUxv6Q6ZpjUJhCoS3N2EXQD8Eg3OZWUEunp6ZwAhZTFDJPKmF9SHTNMKhNCIl2rwtnL3QAH3eRUUkoUFBTwlyUpixkmlTG/pDpmmNQmUSD9eHK5G+Cgm4iIiMiNFRYW4oknnkBISAhCQ0Px5JNPoqioyOayJ06cQNeuXRESEoIqVargtddeK7HMRx99hPj4ePj7+yM2NhZfffWVo7tARHRT46CbiIiIyI29/PLL2LFjB44cOYLff/8d27dvx/Tp00ssZzKZ0LNnTzRr1gzJycnYsmUL3nvvPSxZssSyzPz58/Hmm29i6dKlyMrKws8//4yGDRveyO4QEd10OOgmpxJCwGAwcNZRUhYzTCpjfgkAFixYgJdeegmVK1dG5cqV8eKLL+Ljjz8usdwff/yBP/74A5MnT4aXlxfi4+MxbNgwzJ8/H8ClQfmkSZPw9ttvo2nTphBCICoqCjVr1nRY7cwwqUxKAYMuibOXuwEOusmphBDw8/PjL0tSFjNMKmN+KSMjA//++y+aNGliaWvSpAnOnDmDzMxMq2U1TQMAq+unNU3DoUOHAFwalCclJWHfvn2IjY1F1apVMWLECBiNRofVzwyT2gT8hBGMr+vjoJucStM0pKamWn6RE6mGGSaVMb+UlZUFAAgODra0mf//4sWLVsvGx8cjNjYWkyZNQn5+Pn7//XcsWLDAMqhOT08HAGzatAl79uzBgQMHcPLkSTz99NMOq9/RGa6o692Tk5Px4IMPomrVqjAYDGjatCm+/vprh9RM6hBCQ6oWA42zl7s8pw66f/zxR/To0QPR0dEQQuDLL790ZjnkJKX98iJSBTNMKmN+3VtAQAAAWB3VNv//lfe/9vLywldffYX9+/ejSpUqePDBBzFkyBCEhYVZrev5559HeHg4wsPD8fzzz2Pt2rUO7YMjM1xR17tnZWWhadOm+Omnn3DhwgVMnToVAwcOxJEjRxxWO6mhSOqdXQLdAE4ddGdnZ6Nx48Z4//33nVkGERERkVsKCQlB1apVceDAAUvbgQMHUK1aNQQFBZVYPiEhAd9//z1SU1Nx4MAB5Ofno0OHDgAuHQn38fG5UaXfEBV1vXvNmjUxfvx4VK1aFTqdDj169EB8fDx++umnG90lInICT2e+eNeuXdG1a1dnlkBERETk1oYMGYJXXnkFbdu2BQBMnz4dw4cPt7nsoUOHEBcXBy8vL6xbtw4LFizA5s2bAQC+vr546KGHMHPmTDRr1gxCCMycORO9evW6YX2pSNe63r34lxLXut79SsnJyTh69CgaNWrkmOKJ6Kbi1EG3vfLz85Gfn2/52XwNkaZplp2dEAJCCEgprXZ812q/8loge9t1Ol2JddvbXt7aVe4TAAQFBUFKCU3TXKJPrrid2KfSa5dS2sywyn1yxe3EPtmuHbDeB7tCn1xxOzm6Ty+99BJSU1NRr149AMCDDz6IF154AZqm4bHHHgMAzJkzB0IILF++HHPmzEFeXh4aN26M1atXW24JpmkaZs2ahSeeeAI1atSAt7c3evTogTfeeMOqv6r8HWG+3t1gMFgeN1/vnpmZaXX6vfl694kTJyIxMRHHjx+3XO9+ZS2FhYW4//770b9/fzRr1swt/oYVQgLFZuiWUgC4Wrt1jfa36wDI/19/edvF/6/fdntF9ElKgSBxDlICGoR57RACJa7z1gkJKQGJ8rcLAOKq7dbzqJtrKa39yhrtbb9m7WXI6s22Ly+NkFd+kpxECIE1a9agd+/epS4zZcoUJCYmlmj/888/LTs+X19fBAUFITMzE7m5uZZl/P39ERgYiPT0dBQUFFjaDQYD/Pz8kJqaanVNUEhICLy9vZGUlGS1QcLCwuDh4YHk5GSrGiIjI2EymZCWlmbVp6ioKOTn5yMjI8PS7unpifDwcOTk5FjN6KnX6xEaGoqLFy8iOzvb0s4+sU/sE/vEPrFP7BP7xD7d2D4ZjUaEh4dj9+7diI2NBXDpgE98fDz++OMPGAwGqz7t378fY8aMweHDhxEdHY0uXbrg008/xcmTJy19KigowMiRI+Hh4YHFixejsLDQLbZT9+6ZMBgu9yk93R9paYGoUiUdfn6X+5SUZIDR6IeYmFTo9Zf7dPZsCHJyvBEXlwSd7nKfTp8OQ2GhB2rVsu7T8eOR8PIyISbmcp80TeDEiSj4+eWjSpXLfSoo8MTp0+EwGHIQFXW5Tzk5epw9G4qwsIsIDb3cJ6PRF0lJQYiKqpg+fZM4ALLYFb9hutPwQCGStVpWfYrUHYcJXkjTYixtAhqiPE4gX/ohQ6tiafcUBQjXnUaONMCoRVna9SIHobqzuKiFIVuGWtp9hRFBuiRkalHIlQZLu79IR6AuDelaFRRIP0u7QZcEP2FEqhZjdU16iO4svEUOkkxxFdOnpssv9+km3Ed4eHjgxIkTqFOnDjIzMy37BFuUGnTbOtJdrVo1ZGRkWO34XPVbQlfsk8lkQkpKCsLDw6HT6VyiT664ndin0msvKipCampqiQyr3CdX3E7sk+3ar9wHu0KfXHE7sU/O+zuiWrVqmDVrFu69914AwOrVqzF27FicOnXqmrVPmDABp06dwrJlyyClREFBAfr374+CggJ89dVX0Ov1brOdWrYs7ehvae3ucaRbCIm1kx9AuO5v6P7/NXiku1h751+s22/C/d6FCxcQEhJyzUG3UqeXe3t7w9vbu0S7Tqez/LFgZn4jrlRa+5XPL0+7va/p6HZV+mR+7eKvr3qfXHE7sU9Xr/3KDLtCn8rSzj6p3yfzc8qyD1alT664ndgn5/wdMWTIEMyYMQPt2rUDcPl6d1vLHz582Op694ULF2Lz5s0Q4tIXtPfffz9ycnKwbt06y9+z7rKdzAPPsrfbrt2+dvOg2THtFdMnCQgBnZCWQbfZlT8DgBCXBrCOa79y+Hv1dls12tt+1RqdkNWK+DzZotSgm4iIiIjoRpk4cSLS0tIs17s/9NBDeOGFFwAAjz76KABg7ty5AFDievcvv/zSMlHarl278NVXX8HHxwfh4eGW9b/wwguW9RGR63LqoDsrKwvHjx+3/Hzy5EkcOHAAoaGhqF69uhMrIyIiIrp5tGjh7ApuTkIAjrwNuJeXF95//32bt7c1D7bNXn75Zbz88ss219OhQ4cSp2kTkftw6qB7z549uP322y0/jx07FgAwaNAgLFq0yElV0Y0khEBYWJjN0zuIVMAMk8qYX1KdlMwwqUtKgTDdaZunV5Nrceqgu2PHjvzWz80JIeDh4cFflqQsZphUxvyS+phhUpmABwrB+Lo+XtNNTqVpGpKTkxEZGVnmiQiIbibMMKmM+SXVCaEhecODiNQdL3XyJrfVZY+zK6BrEEJDslaL+XUD/A1LRERERERE5CAcdBMRERERERE5CAfdRERERERERA7CQTc5lU6n47WEpDRmmFTG/JLqpNTxelhSFvPrPvhblpxKSgmTycRZ7ElZzDCpjPkl9UmY4AVGmNTE/LoLDrrJqaSUSEtL4x98pCxmmFTG/JLqhJBI02IgwXsukXqYX/fBQTcRERERERGRg3DQTUREREREROQgHHST0wnBU2pIbcwwqYz5JdUJaM4ugajcmN//a+++o6I80/6Bf58BqTogIkhRwEbsjdUQY9RoLNHEltXji7G7PxNdS7K2jSVrNFlLbFFjl815LdFVo/EYEsXViLFEBbEFIxZsqNRBkDrX7w+XeR1nqGEcHuf7OYdz9H7uebgGvgxz8dxzj22wt3YBZNs0Gg28vb2tXQZRuTHDpGbML6mdiAbedvHWLoOoXJhf28Er3WRVIoKcnBxu4kOqxQyTmjG/pH6CHHHh7s+kUsyvrWDTTVYlIkhNTeUTPlItZpjUjPkltVMUQarej7s/kyoxv7aDTTcRERERERGRhbDpJiIiIiIiIrIQNt1kdfb23M+P1I0ZJjVjfknt7JVca5dAVG7Mr23gb1qyKo1GA09PT2uXQVRuzDCpGfNLaieigafmlrXLICoX5td28Eo3WZWIICsri5v4kGoxw6RmzC+pnyBLtNz9mVSK+bUVbLrJqkQEOp2OT/hItZhhUjPml9ROUQQ6vTd3fyZVYn5tB5tuIiIiIiIiIgth001ERERERERkIWy6yaoURYGDgwMUhctqSJ2YYVIz5pfUT4GDksXFuaRSzK+t4O7lZFWKosDDw8PaZRCVGzNMasb8ktqJKPDQ3LV2GUTlwvzaDl7pJqsSEWRkZHATH1ItZpjUjPkl9RNk6Gtw92dSKebXVrDpJqsSEWRmZvIJH6kWM0xqxvyS2imKIFM8uPszqRLzazvYdBMRERERERFZCJtuIiIiIiIiIgth001WpSgKnJ2duXMuqRYzTGrG/JL6KXBWdFycSyrF/NoK7l5OVqUoCtzc3KxdBlG5McOkZswvqZ2IAjfNA2uXQVQuzK/t4JVusioRQXp6OjfxIdVihknNmF9SO0URpOu9IcJrhaQ+zK/tYNNNViUiePLkCZ/wkWoxw6RmzC+pn+CJaMEEkzoxv7aCTTcRERERERGRhbDpJiIiIiIiIrIQNt1kVYqiwNXVlTvnkmoxw6RmzC+pnYgCVyUFChfokgoxv7aDu5eTVSmKgmrVqlm7DKJyY4ZJzZhfUj8F1TTJ1i6CqJyYX1vBK91kVSKClJQUbuJDqsUMk5oxv6R2iiJI0ftx92dSJebXdrDpJqsSEeTm5vIJH6kWM0xqxvyS+glyxYWLc0mlmF9bwaabiIiIiIiIyELYdBMRERERERFZCJtusipFUaDVarlzLqkWM0xqxvyS2oko0GoecPdnUiXm13Zw93KyKkVR4OLiYu0yiMqNGSY1Y35J/RS4KDprF0FUTsyvreCVbrIqvV6PpKQk6PV6a5dCVC7MMKkZ80tqpyh6JOkDoOfuz6RCzK/tYNNNVpefn2/tEoj+EGaY1Iz5JbXLFwdrl0BUbsyvbWDTTURERERERGQhbLqJiIiIiIiILIRNN1mVoiioXr06d84l1WKGSc2YX1I7EQXVNXe5+zOpEvNrO7h7OVmVoihwdHS0dhlE5cYMk5oxv6R+ChyVLGsXQVROzK+t4JVusiq9Xo8HDx5w51xSLWaY1Iz5JbVTFD0eFNTj7s+kSsyv7WDTTVYnwiU1pG7MMKkZ80tqJ3w6SyrG/NoGfpeJiIiIiIiILIRNNxEREREREZGFsOkmq1IUBTVq1ODOuaRazDCpGfNLaieioIbmFnd/JlVifm0Hm26yKkVRYGdnxyd8pFrMMKkZ80vqp8AOeWCESZ2YX1vBppusSq/X4+HDh9w5l1SLGSY1Y35J7RRFj4f6+tz9mVSJ+bUdbLqJiIiIiIiILIRNNxEREREREZGFsOkmIiIiIiIishA23WRVGo0GXl5e0GgYRVInZpjUjPkltRPRwEtzDRqFuz+T+jC/toO/ZcmqRAQFBQUQ4YMNqRMzTGpm6fzm5eVh/PjxqF69Ojw8PPDXv/4V+fn5ZueuXLkSISEhcHR0RN++fc3O2bBhA4KDg+Hq6orAwEDs3bvXInWTmggKUAV8CCZ1Yn5tBZtusioRQXJyMhsWUi1mmNTM0vmdN28eoqKicPnyZVy6dAnHjh3D559/bnaur68vZs6ciTFjxpg9vm7dOnz55ZfYvn07Hj9+jFOnTqFZs2YWqZvUQ1EEyfoACLj7M6kP82s72HQTERGRRWzatAkzZ86Ej48PfHx88Mknn2Djxo1m5/bv3x99+/aFp6enybGCggLMnj0by5cvR6tWraAoCry9vVG3bl1L3wUiIqI/jE03ERERVbjU1FTcuXMHLVu2NIy1bNkSCQkJSE9PL9O54uLi8ODBA5w7dw6BgYHw9/fHmDFjoNPpKrhqIiKiisemm6xOUbikhtSNGSY1s1R+Hz9+DABwd3c3jBX+OyMjo0znSklJAQAcOnQIZ86cQUxMDG7cuIHJkydXSK2kbgr01i6BqNyYX9vAppusSqPRwNvbmzvnkmoxw6Rmlsxv1apVAcDoqnbhv6tVq1auc82YMQOenp7w9PTEjBkz8P3331dQtaRWIhp428Vz92dSJebXdvBZIlmViCAnJ4ebUJFqMcMElG2X7pLmxsfHo2fPnqhevTr8/PywcOFCi9VtyfxWr14d/v7+iImJMYzFxMSgdu3acHNzK9O5goOD4eTkVMEV0stBkCMu3P2ZVIr5tRVsusmqRASpqalsWEi1mGECyrZLd3FzCwoK8O6776J169Z4+PAhDh8+jJUrV2Lr1q0WqdvS+R0xYgTmz5+PxMREJCYm4vPPP8fo0aPNzs3Pz0d2djby8/Oh1+uRnZ2N3NxcAICzszOGDBmCBQsWIDU1FWlpaViwYAH69OljkbpJPRRFkKr34+7PpErMr+1g001ERPQHlWWX7uLmxsXFIS4uDnPmzEGVKlUQHByMUaNGYd26dS/y7lSYWbNmITQ0FI0aNUKjRo3Qvn17/P3vfwcAjB07FmPHjjXMnTdvHpydnTF//nx8//33cHZ2Rrdu3QzHly1bBl9fXwQFBSE4OBgBAQFYsmTJC79PREREZWVv7QKIiIjUrKRdup9dSl3SXL3+6YY6z1551uv1iI2Ntfj9sIQqVapg1apVWLVqlcmxNWvWGP3/008/xaefflrkuVxdXREeHl7BFRIREVker3ST1dnb828/pG7MsG0ryy7dJc0NDg5GYGAgZs+ejZycHFy6dAmbNm2y6FtjMb+kdvZKrrVLICo35tc2sOkmq9JoNPD09OTOz6RazDCVZZfukuZWqVIFe/fuRXR0NPz8/BAWFoYRI0agRo0aFqmd+SW1E9HAU3OLuz+TKjG/toO/ZcmqRARZWVnchIpUixmmsuzSXZq5TZo0wU8//YSkpCTExMQgJycHHTt2tEjtzC+pnyBLtNz9mVSK+bUVXFNGViUi0Ol0cHJygqJw50ZSH0tnOC8vD5MnT8aWLVugKArCwsKwdOlSs0uCi5ubk5OD8ePH49ChQ0hKSoKfnx+mTp2KkSNHVnjNtqhwl+727dsDQLG7dJc0NzY2FvXq1UOVKlWwf/9+bNq0CZGRkRapW0Sg+88oOGmuQeGVFlM9zli7AiqBogh0em84aTKggBkmdWF+bQevdNNLrSLfO3flypUICQmBo6Mj+vbt+4LugW3g96loFfVWVPn5+fDx8cGhQ4eg0+kQHh6Ojz/+GD/99NOLvDsvrbLs0l3cXADYsWMH6tSpg+rVq2Px4sX47rvv0Lx58xd+n4iIiKhisOl+SVRk01KWc1V2FdWwAICvry9mzpyJMWPGvKjybQa/T0WrqLeicnV1xdy5c1GvXj0oioJXX30VnTt3RlRU1Iu8Oy+twl26U1NTkZqaiq+++sqwGmHNmjVGO3UXNxd4mvHk5GRkZmbil19+MVwRJyIiInVi0/2SqMimpSzn+qMURYGDg4PFlpZXVMMCAP3790ffvn3h6elpkVptmZq/T5bMcElvL1XeuQCQnZ2N06dP8wqqjVMUBQ5KFvjiHlIvZpjUjPm1FWy6XxIV2bSU5Vx/lKIo8PDwUF3DQhVH7d8nS2a4It+K6lkigtGjR6NBgwbo379/xRZNqqIoCjw0d/l6blItEWaY1Iv5tR1sul8CFdm0vOimRkSQkZFhkZ1zLdWwUMVS+/fJkhmuyLeierbeDz/8EHFxcfjuu+/4VlE2TkSQoa/BnXNJxZhhUjPm11Zw9/KXQEmNyLNvWVPS3MLGoTTnqggigszMTLi6ulb4lcJnm5DCpcalaVhKmksVS+3fJ0tm+Nm3l6pXrx6A0r0VVVFzRQTjxo3DqVOnEBkZWeE/z5VZSIi1K6icFEXw/acecFVSuHMuqZKiCDKFGSZ1Yn5tBy9xvAQq8mpYWc5V2VX0e+eSZfD7VLzCt5dKTExEYmJiqd6Kqqi548ePx/Hjx3Hw4EFUr179Rd0FIiIiIpvGpvslUJFNy8vW1FRkw5Kfn4/s7Gzk5+dDr9cjOzsbubm5L+quvNT4fSpaRb0V1a1bt7B69WrExcUhICAAVatWRdWqVY1uT0REREQVj8vLXxKFjUjhW8uUpmkpam5ZzvVHKYoCZ2dni+1ePmvWLCQnJ6NRo0YAgCFDhhg1LAAMb+VT3Fzg6a7u//jHPwz/d3Z2RseOHXHkyBGL1G5L1Px9snSGC99eatWqVSbHnn0bqpLmBgQEWOR156R2CpwVHXfOJRVjhknNmF9boUgleBa2atUqLFq0CImJiWjRogW++uortG3btsTb6XQ6uLm5IT09HVqt9gVUWnnl5eVh0qRJ2Lp1K4CnjcjSpUthb29v0rQUN7c0x4mI1Iav6S7amXn84hSpxxlrV2DADBeNGS4C86sKzG8xKlGGi1LaftTqTfe3336LoUOHYs2aNWjXrh2WLVuGnTt3Ii4uDl5eXsXelk23+okIdDodtFqtxa4UElkSM6wOfMJnnqIIDv2jN7TKQ75ljTmV6AkfM2weM1wM5rfSY35LUIkyXJTS9qNWf033kiVLMGbMGIwYMQKNGzfGmjVr4OLigk2bNlm7NHoBRARPnjzhsldSLWaY1E3wRLTcM5dUjBkmNWN+bYVV1wvn5ubi7NmzmDFjhmFMo9Gga9euOHHihBUrIyIqg7OTAc01gH+lNqaCv1ATERERWZpVm+6kpCQUFBTA29vbaNzb2xu//fabyfycnBzk5OQY/l/4VlZpaWnQ6/UAnm5qpCgKRMToylNJ44W3L++4RqMxOXdZx8tbu5rvU0FBAXQ//xUOmuvQKAIFAgWAPPduhYXj+ue2mijruAYC+e/5yzuu/Pf8RY+br71c96nLYZOvmbWy17nz/42LPL23T5dCmRs3rrHs4xoAYrLUqmzjyn/Pb3686NrLdp80mnxsm5YPB43eKMOqz14pxkusPS3NeLwSPe69DNmriJ8nRSmALvP/8gu8JNmrqJ+nZzJs7d+5ev3Llb2K+nnSaIwz/NJkz0wtZb5Pzz0GW/N5xNP8vlzZq4ifJ0URk8fglyJ7NvA8onA87b81lrTiUVU7Y33xxRdGuxIXCggIsEI1RC8C30tZDYIHWbuCyor5VQPmtzjMsBoww0VhftWA+S2OejKckZFR7NsrW7Xp9vT0hJ2dHR48eGA0/uDBA9SqVctk/owZM/DRRx8Z/q/X65GSkoIaNWpwAyOV0ul0qF27Nm7fvs3N8EiVmGFSM+aX1I4ZJjVjftVPRJCRkQFfX99i51m16XZwcECbNm0QGRmJvn37AnjaSEdGRmL8+PEm8x0dHeHo6Gg05u7u/gIqJUvTarV8sCFVY4ZJzZhfUjtmmNSM+VW34q5wF7L68vKPPvoIw4YNQ0hICNq2bYtly5YhMzMTI0aMsHZpRERERERERH+I1ZvuQYMG4dGjR5g9ezYSExPRsmVLREREmGyuRkRERERERKQ2Vm+6AWD8+PFml5PTy8/R0RFz5swxedkAkVoww6RmzC+pHTNMasb82g5FStrfnIiIiIiIiIjKRWPtAoiIiIiIiIheVmy6iYiIiIiIiCyETTcRERERERGRhbDpJgDAF198gT/96U+oVq0avLy80LdvX8TFxRnN6dSpExRFMfoYO3asybnCw8PRvHlzODk5wcvLC+PGjXvp6qLK5dNPPzXJwCuvvGI4vm7dOnTq1AlarRaKoiAtLc3o9jdv3sSoUaMQFBQEZ2dn1KtXD3PmzEFubq7FarJWXWQdP//8M9555x34+vpCURR89913RsdFBLNnz4aPjw+cnZ3RtWtX/P7774bjZc3CtWvXUK1aNbi7u/+hukvKKACkpKQgLCwMWq0W7u7uGDVqFB4/fmw4fuTIEfTp0wc+Pj5wdXVFy5YtsWXLliI/5/bt26EoCvr27fuHaqeKU5rfxdnZ2Rg3bhxq1KiBqlWrYsCAAXjw4IHZ8yUnJ8Pf399sprZs2YIWLVrAxcUFPj4+GDlyJJKTk8tVd0pKCv76178iODgYzs7OqFOnDiZMmID09HSjeQkJCejVqxdcXFzg5eWFKVOmID8/3+w5jx8/Dnt7e7Rs2dJovKCgALNmzTL6Gf3ss8/ArZMqn3/+859QFAWTJk0yjJUmv5GRkXjttddQrVo11KpVC9OmTTPJiYhg8eLFaNiwIRwdHeHn54f58+dXSN1jx46FoihYtmyZ0XhJj8EAsGPHDrRs2RIuLi4ICAjAokWLTM6fk5ODTz75BAEBAXB0dERgYCA2bdpUIbUTm276r6NHj2LcuHE4efIkDh48iLy8PHTr1g2ZmZlG88aMGYP79+8bPhYuXGh0fMmSJfjkk08wffp0XLp0CYcOHUL37t2N5jx8+BCXL182qSEvLw9RUVFWq4vUrUmTJkYZeDZLWVlZ6NGjB/7+97+bve1vv/0GvV6PtWvX4tKlS1i6dCnWrFljMr8s2S2ppoqsiyq/zMxMtGjRAqtWrTJ7fOHChVixYgXWrFmDU6dOwdXVFd27d0d2djaAsmUhLy8PgwcPRocOHcx+rnPnzkGn05mM3717F1evXjUaKymjABAWFoZLly7h4MGD2L9/P37++Wf85S9/MRz/5Zdf0Lx5c+zatQuxsbEYMWIEhg4div3795uc6+bNm/jb3/5WZO1kHaX5XTx58mR8//332LlzJ44ePYp79+6hf//+Zs83atQoNG/e3GT8+PHjGDp0KEaNGoVLly5h586dOH36NMaMGWM0r7QZvnfvHu7du4fFixfj4sWLCA8PR0REBEaNGmWYU1BQgF69eiE3Nxe//PIL/vWvfyE8PByzZ882OX9aWhqGDh2KLl26mBxbsGABvv76a6xcuRJXrlzBggULsHDhQnz11VdmvwZkHb/++ivWrl1rkr+S8nv+/Hm8/fbb6NGjB6Kjo/Htt99i3759mD59utF5Jk6ciA0bNmDx4sX47bffsG/fPrRt29ZoztWrV3Hv3j2T2nQ6Hc6ePWu27j179uDkyZPw9fU1OVbSY/APP/yAsLAwjB07FhcvXsTq1auxdOlSrFy50ug8AwcORGRkJDZu3Ii4uDhs27YNwcHBRXwlqcyEyIyHDx8KADl69KhhrGPHjjJx4sQib5OSkiLOzs5y6NChYs+9ePFi8fb2lt9++80wlpeXJwMGDJCmTZtKfn6+Veoi9ZozZ460aNGixHn/+c9/BICkpqaWOHfhwoUSFBRkNFaW7Ja2poqoi9QFgOzZs8fwf71eL7Vq1ZJFixYZxtLS0sTR0VG2bdtW5HmKysLUqVNlyJAhsnnzZnFzczM53rt3b2nfvr08fvzYMJaYmCjBwcHywQcfmP1cRWX08uXLAkB+/fVXw9gPP/wgiqLI3bt3i6z97bfflhEjRhiN5efny2uvvSYbNmyQYcOGSZ8+fYq8PVnX87+L09LSpEqVKrJz507DnCtXrggAOXHihNFtV69eLR07dpTIyEiTTC1atEjq1q1rNH/FihXi5+dnNFaeDBfasWOHODg4SF5enoiIHDhwQDQajSQmJhrmfP3116LVaiUnJ8fotoMGDZKZM2eafXzv1auXjBw50misf//+EhYWVmw99OJkZGRIgwYN5ODBg0bPHUuT3xkzZkhISIjR+fbt2ydOTk6i0+lE5Onjob29vdFzBHPGjx8vwcHBRpl7/PixtG/fXnr27Gky/86dO+Ln5ycXL16UgIAAWbp0qeFYaR6DBw8eLO+9957ROVesWCH+/v6i1+sNt3Fzc5Pk5ORia6fy45VuMqtw6ZWHh4fR+JYtW+Dp6YmmTZtixowZyMrKMhw7ePAg9Ho97t69i0aNGsHf3x8DBw7E7du3jc7x8ccfY8CAAejSpQvi4+Oh1+sxdOhQREdHIyIiAnZ2dlapi9Tt999/h6+vL+rWrYuwsDAkJCT8ofOlp6eb5Kys2a3omoqqi9Ttxo0bSExMRNeuXQ1jbm5uaNeuHU6cOFHk7cxl4fDhw9i5c2eRV9QBYOvWrcjPz8c777yDJ0+eICkpCV26dEGDBg2wfPnyMtV+4sQJuLu7IyQkxDDWtWtXaDQanDp1qky1z507F15eXkZXIalyev538dmzZ5GXl2eU4VdeeQV16tQxyvDly5cxd+5cfPPNN9BoTJ+ChoaG4vbt2zhw4ABEBA8ePMC///1vvP3220bz/kiG09PTodVqYW9vD+Bphps1awZvb2/DnO7du0On0+HSpUuGsc2bN+P69euYM2eO2fO+9tpriIyMNFxpP3/+PKKiotCzZ89i66EXZ9y4cejVq5dRToHS5TcnJwdOTk5Gt3N2dkZ2drbh6vT333+PunXrYv/+/QgKCkJgYCBGjx6NlJQUo9stWbIEwcHB6Nq1K5KSkvDkyRP07t0bOTk52L59u9FcvV6P999/H1OmTEGTJk1M7lNpHoOLqv3OnTu4desWAGDfvn0ICQnBwoUL4efnh4YNG+Jvf/sbnjx5UvIXlkrF3toFUOWj1+sxadIktG/fHk2bNjWM/8///A8CAgLg6+uL2NhYTJs2DXFxcdi9ezcA4Pr169Dr9fj888+xfPlyuLm5YebMmXjrrbcQGxsLBwcHw7lWrlyJJ0+e4M0338Srr76KEydO4Oeff4afn59V6yJ1ateuHcLDwxEcHIz79+/jH//4Bzp06ICLFy+iWrVqZT7ftWvX8NVXX2Hx4sUmx0qb3YquqaS6SL0SExMBwOhJf+H/C489z1wWkpOTMXz4cPzv//4vtFptkZ+vWrVqiIiIwJtvvok+ffrg0aNH8PX1xb///W9UqVKlzLV7eXkZjdnb28PDw6PI2nfs2GFY4lkoKioKGzduRExMTJk+P7145n4XJyYmwsHBwWQPgWcznJOTg8GDB2PRokWoU6cOrl+/bnLu9u3bY8uWLRg0aBCys7MNjfXzf0Qqb4aTkpLw2WefGS29TUxMNPuzV3gMePoH1OnTp+PYsWOGZv1506dPh06nwyuvvAI7OzsUFBRg/vz5CAsLK7IeenG2b9+Oc+fO4ddffzU5Vpr8du/eHcuWLcO2bdswcOBAJCYmYu7cuQCA+/fvA3j6fPPWrVvYuXMnvvnmGxQUFGDy5Ml47733cPjwYcN5q1Spgh07dqBPnz5466234OnpidTUVPznP/8xeexesGAB7O3tMWHCBLP3qzSPwd27d8fkyZMxfPhwdO7cGdeuXcOXX35pqD0wMBDXr19HVFQUnJycsGfPHiQlJeHDDz9EcnIyNm/eXNovMxXH2pfaqfIZO3asBAQEyO3bt4udV7g07Nq1ayIiMn/+fAEgP/74o2HOw4cPRaPRSEREhMnt8/PzpX79+gJADh8+XGnqIvVLTU0VrVYrGzZsMBovzTLuO3fuSL169WTUqFFFzilrdourqSLrInXAc8vLjx8/LgDk3r17RvP+/Oc/y8CBA01uX1QW+vXrJ9OmTTP8v6jl5YXi4+PF3t5etFptiS9tKCqj8+fPl4YNG5rMr1mzpqxevdpk/PDhw+Li4iL/+te/DGM6nU4CAwPlwIEDhjEuL6+8zP0u3rJlizg4OJjM/dOf/iRTp04VEZHJkyfLoEGDDMfMZerSpUvi4+MjCxculPPnz0tERIQ0a9bMZNl2obJkOD09Xdq2bSs9evSQ3Nxcw/iYMWOkW7duRnMzMzMFgBw4cEDy8/MlJCREvv76a8Nxc8vLt23bJv7+/rJt2zaJjY2Vb775Rjw8PCQ8PLzYusjyEhISxMvLS86fP28Ye3Z5eWnyKyLy5ZdfilarFTs7O3FxcZEvvvhCAMj27dtF5GmWAEhcXJzhNmfPnhUAZpecp6Wlibu7u9jZ2cnVq1dNjp85c0a8vb2NXqrz/PLy0jwG6/V6mTp1qjg5OYmdnZ1Ur15dPv30UwEgJ0+eFBGRt956S5ycnCQtLc1wjl27domiKJKVlWVyfio7Nt1kZNy4ceLv7y/Xr18vce7jx48FgKFx3bRpkwAwaYq9vLxk3bp1JrefOHGi1KpVS3r37i0NGjQwecJprbro5RASEiLTp083Giupub179640aNBA3n//fSkoKCjy3GXJbkk1VWRdpA7PN93x8fECQKKjo43mvfHGGzJhwgSjseKy4ObmJnZ2doYPjUYjAMTOzk42btxoNLfw9YOvv/66NGrUSPr27Wt4jas5RWV048aN4u7ubjSWl5cndnZ2snv3bqPxI0eOiKurq6xdu9ZoPDo62lBn4YeiKKIoitjZ2Rn+gErWV9TvYnOvzxYRqVOnjixZskRERFq0aCEajcZsPmfPni0iIkOGDDF57emxY8fM/lGqLBnW6XQSGhoqXbp0kSdPnhgdmzVrlkkDff36dQEg586dk9TUVLP5LByLjIwUERF/f39ZuXKl0Xk+++wzCQ4OLuKrSS/Knj17TL6HAAyPMYcOHSoxv4X0er3cvXtXsrKyDK+nPn36tIiIzJ49W+zt7Y3mZ2VlCQD56aefjMYL94Np1KiRvP766xIaGioZGRlGc5YuXWqo8dm6NRqNBAQEiEjZHoPz8/Plzp07kpOTIwcOHBAA8vDhQxERGTp0qNSrV89ofuH9M/cHASo7vqabADx9i4Px48djz549OHz4MIKCgkq8TeEyQB8fHwBPl4UBMHobkZSUFCQlJSEgIMDottOmTcPWrVtx+PBh7NmzB40bN0aXLl3w6NEjq9ZFL4fHjx8jPj7ekIHSuHv3Ljp16oQ2bdpg8+bNZl9vCJQ+uxVRU1nqIvUKCgpCrVq1EBkZaRjT6XQ4deoUQkNDDWMlZeHEiROIiYkxfMydOxfVqlVDTEwM+vXrZ5hX+PrBgoIC/PDDD4iMjMTFixcRFhaGgoKCMtUeGhqKtLQ0ox13Dx8+DL1ej3bt2hnGjhw5gl69emHBggVGS3uBp6+dvHDhglHt7777Ljp37oyYmBjUrl27TDVRxSvpd3GbNm1QpUoVowzHxcUhISHBkOFdu3bh/Pnzhu/xhg0bAADHjh0zvIVnVlaWSa4L98qQZ956qywZ1ul06NatGxwcHLBv3z6T17aGhobiwoULePjwoWHs4MGD0Gq1aNy4MbRarUk+x44di+DgYMTExBhyXlTter2+lF9lspQuXbqYfA9DQkIQFhZm+HdJ+S2kKAp8fX3h7OyMbdu2oXbt2mjdujWAp8838/PzER8fb5hf+Br/Z59vFu4HExsbi8jISMMeBr169TLak+j9999HbGysUd2+vr6YMmUKfvzxRwClfwwGnubRz88PDg4O2LZtG0JDQ1GzZk1D7ffu3TN6q7GrV69Co9HA39+/fF94MmbVlp8qjQ8++EDc3NzkyJEjcv/+fcNH4ZKSa9euydy5c+XMmTNy48YN2bt3r9StW1feeOMNo/P06dNHmjRpIsePH5cLFy5I7969pXHjxkZLuZYvXy4eHh5Gy3xycnKkZ8+e0qpVK6MdoF9kXaReH3/8sRw5ckRu3Lghx48fl65du4qnp6fhL7j379+X6OhoWb9+vQCQn3/+WaKjow27dN65c0fq168vXbp0kTt37hhl7VllyW5JNVVkXVT5ZWRkSHR0tOGq7pIlSyQ6Olpu3bolIiL//Oc/xd3dXfbu3SuxsbHSp08fCQoKMlyVK08Wilpe3q9fP2nTpo3RMsKEhAQJDAw0ubJeUkZFRHr06CGtWrWSU6dOSVRUlDRo0EAGDx5sOF64pHzGjBlGdRe3Sy6Xl1cuJf0uFnm67LxOnTpy+PBhOXPmjISGhkpoaGiR5zS3emLz5s1ib28vq1evlvj4eImKipKQkBBp27at0W1Lm+H09HRp166dNGvWTK5du2ZUe+HjdX5+vjRt2lS6desmMTExEhERITVr1pQZM2YUWbu55eXDhg0TPz8/2b9/v9y4cUN2794tnp6eRsuTqfJ4/p1vSpPfhQsXSmxsrFy8eFHmzp0rVapUMVq1VFBQIK1bt5Y33nhDzp07J2fOnJF27drJW2+9ZXSeyZMnS1BQkCQkJBjG0tLSpE2bNvLuu+8WW/fzy8tFSn4MfvTokXz99ddy5coViY6OlgkTJoiTk5OcOnXKMCcjI0P8/f3lvffek0uXLsnRo0elQYMGMnr06JK+lFRKbLpJRJ4udzT3sXnzZhF5+svsjTfeEA8PD3F0dJT69evLlClTJD093eg86enpMnLkSHF3dxcPDw/p16+f0YOKyNMnj+fOnTOp4cmTJybLb15kXaRegwYNEh8fH3FwcBA/Pz8ZNGiQ0ZLUOXPmFJujzZs3F5m1Z5UluyXVVJF1UeVX2GA8/zFs2DARebpkcdasWeLt7S2Ojo7SpUsXo9cFlicLRTXdUVFRZhve69evS2xsrNFYSRkVEUlOTpbBgwdL1apVRavVyogRI4yWSQ4bNszsOTp27Fhk7Wy6K5eSfheLPH0c/PDDD6V69eri4uIi/fr1K/aPQkW9ZGHFihXSuHFjcXZ2Fh8fHwkLC5M7d+4YzSlthov6uQMgN27cMMy7efOm9OzZU5ydncXT01M+/vjjYl9uYa7p1ul0MnHiRKlTp444OTlJ3bp15ZNPPjF52zGqHJ5vukuT386dO4ubm5s4OTlJu3btjPahKHT37l3p37+/VK1aVby9vWX48OEmWb1w4YJR/golJydLVFRUsXWba7pLegx+9OiRvPrqq+Lq6iouLi7SpUsXw2u5n3XlyhXp2rWrODs7i7+/v3z00Ud8PXcFUkSeWa9DRERERERERBWGLw4kIiIiIiIishA23UREREREREQWwqabiIiIiIiIyELYdBMRERERERFZCJtuIiIiIiIiIgth001ERERERERkIWy6iYiIiIiIiCyETTcRERERERGRhbDpJiIishGBgYFYtmyZRT/H8OHD0bdvX4t+DiIiIjVh001ERPSCDB8+HIqiYOzYsSbHxo0bB0VRMHz48FKf7+bNm1AUBTExMaWa/+uvv+Ivf/lLqc9vzvr169GiRQtUrVoV7u7uaNWqFb744gvD8eXLlyM8PPwPfQ4iIqKXCZtuIiKiF6h27drYvn07njx5YhjLzs7G1q1bUadOHYt8ztzcXABAzZo14eLiUu7zbNq0CZMmTcKECRMQExOD48ePY+rUqXj8+LFhjpubG9zd3f9oyURERC8NNt1EREQvUOvWrVG7dm3s3r3bMLZ7927UqVMHrVq1MpobERGB119/He7u7qhRowZ69+6N+Ph4w/GgoCAAQKtWraAoCjp16gTg/5Z4z58/H76+vggODgZgvLz8yJEjcHBwwLFjxwznW7hwIby8vPDgwQOzte/btw8DBw7EqFGjUL9+fTRp0gSDBw/G/PnzDXOeXV5eeCX++Y/COgEgKioKHTp0gLOzM2rXro0JEyYgMzOzbF9UIiKiSoxNNxER0Qs2cuRIbN682fD/TZs2YcSIESbzMjMz8dFHH+HMmTOIjIyERqNBv379oNfrAQCnT58GABw6dAj37983auQjIyMRFxeHgwcPYv/+/Sbn7tSpEyZNmoT3338f6enpiI6OxqxZs7BhwwZ4e3ubrbtWrVo4efIkbt26Var7Wbt2bdy/f9/wER0djRo1auCNN94AAMTHx6NHjx4YMGAAYmNj8e233yIqKgrjx48v1fmJiIjUQBERsXYRREREtmD48OFIS0vD+vXrUbt2bcTFxQEAXnnlFdy+fRujR4+Gu7t7ka+JTkpKQs2aNXHhwgU0bdoUN2/eRFBQEKKjo9GyZUujzxMREYGEhAQ4ODgYxgMDAzFp0iRMmjQJwNNl5+3atUPDhg1x8eJFtG/fHuvWrSuy/vv376N///44efIkGjZsiNDQULz99tt47733oNFojO7jd999Z3Tb7OxsdOrUCTVr1sTevXuh0WgwevRo2NnZYe3atYZ5UVFR6NixIzIzM+Hk5FSGry4REVHlxCvdREREL1jNmjXRq1cvhIeHY/PmzejVqxc8PT1N5v3+++8YPHgw6tatC61Wi8DAQABAQkJCiZ+jWbNmRg23OQ4ODtiyZQt27dqF7OxsLF26tNj5Pj4+OHHiBC5cuICJEyciPz8fw4YNQ48ePQxX34sycuRIZGRkYOvWrYYG/fz58wgPD0fVqlUNH927d4der8eNGzdKvI9ERERqYG/tAoiIiGzRyJEjDcuoV61aZXbOO++8g4CAAKxfvx6+vr7Q6/Vo2rSpYWO04ri6upaqjl9++QUAkJKSgpSUlFLdrmnTpmjatCk+/PBDjB07Fh06dMDRo0fRuXNns/PnzZuHH3/8EadPn0a1atUM448fP8b/+3//DxMmTDC5jaU2lSMiInrR2HQTERFZQY8ePZCbmwtFUdC9e3eT48nJyYiLi8P69evRoUMHAE+XXj+r8Ep2QUFBuWqIj4/H5MmTsX79enz77bcYNmwYDh06ZLgSXRqNGzcGgCI3P9u1axfmzp2LH374AfXq1TM61rp1a1y+fBn169cvV/1ERERqwKabiIjICuzs7HDlyhXDv59XvXp11KhRA+vWrYOPjw8SEhIwffp0ozleXl5wdnZGREQE/P394eTkBDc3t1J9/oKCAgwZMgTdu3fHiBEj0KNHDzRr1gxffvklpkyZYvY2H3zwAXx9ffHmm2/C398f9+/fx7x581CzZk2EhoaazL948SKGDh2KadOmoUmTJkhMTATw9I8FHh4emDZtGl599VWMHz8eo0ePhqurKy5fvoyDBw9i5cqVpbofRERElR1f001ERGQlWq0WWq3W7DGNRoPt27fj7NmzaNq0KSZPnoxFixYZzbG3t8eKFSuwdu1a+Pr6ok+fPqX+3PPnz8etW7cMm5j5+Phg3bp1mDlzJs6fP2/2Nl27dsXJkyfx5z//GQ0bNsSAAQPg5OSEyMhI1KhRw2T+mTNnkJWVhXnz5sHHx8fw0b9/fwBA8+bNcfToUVy9ehUdOnRAq1atMHv2bPj6+pb6fhAREVV23L2ciIiIiIiIyEJ4pZuIiIiIiIjIQth0ExEREREREVkIm24iIiIiIiIiC2HTTURERERERGQhbLqJiIiIiIiILIRNNxEREREREZGFsOkmIiIiIiIishA23UREREREREQWwqabiIiIiIiIyELYdBMRERERERFZCJtuIiIiIiIiIgth001ERERERERkIf8ffiODxOQ4UfMAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "加速比分析:\n",
            "  256×256矩阵: 加速比 = 0.07倍\n",
            "  512×512矩阵: 加速比 = 0.59倍\n",
            "  1024×1024矩阵: 加速比 = 0.56倍\n",
            "  2048×2048矩阵: 加速比 = 1.04倍\n",
            "  4096×4096矩阵: 加速比 = 1.77倍\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3 SP（Seqeunce Parallel）策略\n",
        "\n",
        "使用线程模拟多设备，并且使用简单的全连接层。对比：\n",
        "\n",
        "- 不切序列：整个序列数据通过一个完整的模型（多个层）进行计算。\n",
        "\n",
        "- 切序列（序列并行）：将序列分成多个部分，每个部分通过一个设备（用线程模拟）上的子模型计算，然后将结果合并。\n",
        "\n",
        "步骤：\n",
        "\n",
        "* 定义模型\n",
        "* 生成输入数据\n",
        "* 运行不切序列版本\n",
        "* 运行切序列版本（序列并行）\n",
        "* 比较结果和时间"
      ],
      "metadata": {
        "id": "jyrPWqbVysCb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import time\n",
        "import threading\n",
        "import numpy as np\n",
        "import copy\n",
        "\n",
        "class SimpleFeedForwardBlock(nn.Module):\n",
        "    def __init__(self, d_model: int = 512, hidden_dim: int = 2048):\n",
        "        super().__init__()\n",
        "        self.d_model = d_model\n",
        "        self.ffn = nn.Sequential(\n",
        "            nn.Linear(d_model, hidden_dim),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(hidden_dim, d_model)\n",
        "        )\n",
        "\n",
        "        self.norm = nn.LayerNorm(d_model)\n",
        "\n",
        "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
        "        ffn_output = self.ffn(x)\n",
        "        x = x + ffn_output\n",
        "        x = self.norm(x)\n",
        "\n",
        "        return x\n",
        "\n",
        "class SequenceParallelSimulator:\n",
        "    def __init__(self, seq_len: int = 1024, d_model: int = 512):\n",
        "        \"\"\"\n",
        "        初始化序列并行模拟器\n",
        "\n",
        "        Args:\n",
        "            seq_len: 序列长度\n",
        "            d_model: 模型维度\n",
        "        \"\"\"\n",
        "        self.seq_len = seq_len\n",
        "        self.d_model = d_model\n",
        "\n",
        "        # 固定随机种子\n",
        "        torch.manual_seed(42)\n",
        "        np.random.seed(42)\n",
        "\n",
        "        # 创建主模型\n",
        "        self.model = SimpleFeedForwardBlock(d_model)\n",
        "\n",
        "        # 生成测试数据\n",
        "        self.input_data = None\n",
        "        self.generate_data()\n",
        "\n",
        "        # 时间记录\n",
        "        self.serial_time = None\n",
        "        self.parallel_time = None\n",
        "\n",
        "    def generate_data(self):\n",
        "        \"\"\"生成测试数据\"\"\"\n",
        "        # 固定随机种子生成输入序列\n",
        "        self.input_data = torch.randn(1, self.seq_len, self.d_model)\n",
        "        print(f\"生成测试数据: batch_size=1, seq_len={self.seq_len}, d_model={self.d_model}\")\n",
        "        print(f\"输入数据范围: [{self.input_data.min():.4f}, {self.input_data.max():.4f}]\")\n",
        "\n",
        "    def serial_processing(self) -> torch.Tensor:\n",
        "        \"\"\"串行处理：整个序列一次性处理\"\"\"\n",
        "        print(f\"\\n串行处理 - 完整序列 ({self.seq_len} tokens)\")\n",
        "\n",
        "        # 确保模型在评估模式\n",
        "        self.model.eval()\n",
        "\n",
        "        # 预热\n",
        "        with torch.no_grad():\n",
        "            _ = self.model(self.input_data[:, :10, :])\n",
        "\n",
        "        # 正式计时\n",
        "        start_time = time.time()\n",
        "\n",
        "        # 完整序列一次性处理\n",
        "        with torch.no_grad():\n",
        "            output = self.model(self.input_data)\n",
        "\n",
        "        end_time = time.time()\n",
        "        self.serial_time = end_time - start_time\n",
        "\n",
        "        print(f\"串行处理完成，耗时: {self.serial_time:.4f}秒\")\n",
        "        return output\n",
        "\n",
        "    def parallel_worker(self,\n",
        "                       model: nn.Module,\n",
        "                       input_chunk: torch.Tensor,\n",
        "                       output_chunk: List,\n",
        "                       worker_id: int):\n",
        "        \"\"\"并行工作线程：处理序列的一个片段\"\"\"\n",
        "        start_time = time.time()\n",
        "\n",
        "        # 处理序列片段\n",
        "        with torch.no_grad():\n",
        "            chunk_output = model(input_chunk)\n",
        "\n",
        "        end_time = time.time()\n",
        "\n",
        "        # 存储结果\n",
        "        output_chunk[worker_id] = chunk_output\n",
        "\n",
        "        print(f\"  工作线程{worker_id+1}: 处理{input_chunk.shape[1]}个token，耗时: {end_time - start_time:.4f}秒\")\n",
        "\n",
        "    def sequence_parallel_processing(self, num_chunks: int = 2) -> torch.Tensor:\n",
        "        \"\"\"序列并行处理：将序列切分成多个片段并行处理\"\"\"\n",
        "        print(f\"\\n序列并行处理 - 将序列分成{num_chunks}个片段\")\n",
        "\n",
        "        # 计算每个片段的大小\n",
        "        chunk_size = self.seq_len // num_chunks\n",
        "\n",
        "        print(f\"每个片段大小: {chunk_size} tokens\")\n",
        "\n",
        "        # 创建多个模型副本，确保使用相同的权重\n",
        "        models = []\n",
        "        for i in range(num_chunks):\n",
        "            # 深拷贝模型，确保权重相同\n",
        "            model_copy = copy.deepcopy(self.model)\n",
        "            model_copy.eval()  # 设置为评估模式\n",
        "            models.append(model_copy)\n",
        "\n",
        "            # 验证权重是否相同\n",
        "            if i > 0:\n",
        "                params1 = list(models[0].parameters())\n",
        "                params2 = list(model_copy.parameters())\n",
        "                for p1, p2 in zip(params1, params2):\n",
        "                    if not torch.allclose(p1, p2):\n",
        "                        print(f\"警告: 模型{i}的权重与模型0不同！\")\n",
        "\n",
        "        # 切分输入序列\n",
        "        input_chunks = []\n",
        "        for i in range(num_chunks):\n",
        "            start_idx = i * chunk_size\n",
        "            end_idx = (i + 1) * chunk_size if i < num_chunks - 1 else self.seq_len\n",
        "            chunk = self.input_data[:, start_idx:end_idx, :]\n",
        "            input_chunks.append(chunk)\n",
        "            print(f\"  片段{i+1}数据范围: [{chunk.min():.4f}, {chunk.max():.4f}]\")\n",
        "\n",
        "        # 准备存储结果的列表\n",
        "        output_chunks = [None] * num_chunks\n",
        "\n",
        "        # 创建并启动线程\n",
        "        threads = []\n",
        "        start_time = time.time()\n",
        "\n",
        "        for i in range(num_chunks):\n",
        "            thread = threading.Thread(\n",
        "                target=self.parallel_worker,\n",
        "                args=(models[i], input_chunks[i], output_chunks, i)\n",
        "            )\n",
        "            threads.append(thread)\n",
        "\n",
        "            print(f\"  分配任务给工作线程{i+1}: 处理token范围 [{i*chunk_size}:{(i+1)*chunk_size if i < num_chunks-1 else self.seq_len}]\")\n",
        "\n",
        "        print(\"\\n开始并行处理...\")\n",
        "        for thread in threads:\n",
        "            thread.start()\n",
        "\n",
        "        # 等待所有线程完成\n",
        "        for thread in threads:\n",
        "            thread.join()\n",
        "\n",
        "        # 合并结果\n",
        "        print(\"合并结果...\")\n",
        "        parallel_output = torch.cat(output_chunks, dim=1)\n",
        "\n",
        "        end_time = time.time()\n",
        "        self.parallel_time = end_time - start_time\n",
        "\n",
        "        print(f\"序列并行处理完成，总耗时: {self.parallel_time:.4f}秒\")\n",
        "        return parallel_output\n",
        "\n",
        "    def validate_result(self, serial_output: torch.Tensor, parallel_output: torch.Tensor) -> bool:\n",
        "        \"\"\"验证两种方法的结果是否一致\"\"\"\n",
        "        print(\"\\n结果验证:\")\n",
        "\n",
        "        # 打印输出范围以供参考\n",
        "        print(f\"  串行输出范围: [{serial_output.min():.4f}, {serial_output.max():.4f}]\")\n",
        "        print(f\"  并行输出范围: [{parallel_output.min():.4f}, {parallel_output.max():.4f}]\")\n",
        "\n",
        "        # 检查形状是否相同\n",
        "        if serial_output.shape != parallel_output.shape:\n",
        "            print(f\"  形状不匹配: 串行{serial_output.shape} vs 并行{parallel_output.shape}\")\n",
        "            return False\n",
        "\n",
        "        # 计算差异\n",
        "        abs_diff = torch.max(torch.abs(serial_output - parallel_output)).item()\n",
        "        rel_diff = torch.max(torch.abs(serial_output - parallel_output) / (torch.abs(serial_output) + 1e-10)).item()\n",
        "\n",
        "        print(f\"  最大绝对差异: {abs_diff:.6e}\")\n",
        "        print(f\"  最大相对差异: {rel_diff:.6e}\")\n",
        "\n",
        "        # 检查是否在可接受的误差范围内（浮点数计算误差）\n",
        "        tolerance = 1e-6\n",
        "        is_valid = abs_diff < tolerance\n",
        "\n",
        "        if is_valid:\n",
        "            print(\"序列并行计算结果与串行计算结果一致！\")\n",
        "        else:\n",
        "            print(\"计算结果存在显著差异\")\n",
        "\n",
        "        return is_valid\n",
        "\n",
        "    def compare_performance(self):\n",
        "        \"\"\"比较性能并打印结果\"\"\"\n",
        "        print(\"\\n\" + \"=\"*60)\n",
        "        print(\"性能对比\")\n",
        "        print(\"=\"*60)\n",
        "        print(f\"串行处理耗时: {self.serial_time:.4f}秒\")\n",
        "        print(f\"序列并行(2线程)处理耗时: {self.parallel_time:.4f}秒\")\n",
        "\n",
        "        if self.serial_time and self.parallel_time:\n",
        "            if self.serial_time > self.parallel_time:\n",
        "                speedup = self.serial_time / self.parallel_time\n",
        "                improvement = (self.serial_time - self.parallel_time) / self.serial_time * 100\n",
        "                print(f\"加速比: {speedup:.2f}倍\")\n",
        "                print(f\"性能提升: {improvement:.1f}%\")\n",
        "            else:\n",
        "                print(\"注意: 由于线程开销和小矩阵运算，多线程可能不会加速\")\n",
        "\n",
        "def main():\n",
        "    \"\"\"主函数\"\"\"\n",
        "    print(\"=\"*60)\n",
        "    print(\"序列并行(SP)策略演示\")\n",
        "    print(\"模拟逐位置操作的序列并行（如前馈网络）\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    # 创建模拟器\n",
        "    seq_len = 2048  # 使用较长的序列\n",
        "    d_model = 512\n",
        "    simulator = SequenceParallelSimulator(seq_len=seq_len, d_model=d_model)\n",
        "\n",
        "    # 1. 串行处理\n",
        "    print(\"\\n1. 串行处理 (基准):\")\n",
        "    serial_output = simulator.serial_processing()\n",
        "\n",
        "    # 2. 序列并行处理\n",
        "    print(\"\\n2. 序列并行处理:\")\n",
        "    parallel_output = simulator.sequence_parallel_processing(num_chunks=2)\n",
        "\n",
        "    # 3. 验证结果\n",
        "    print(\"\\n3. 验证计算结果:\")\n",
        "    is_valid = simulator.validate_result(serial_output, parallel_output)\n",
        "\n",
        "\n",
        "    # 4. 性能对比\n",
        "    simulator.compare_performance()\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    main()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SrDsmETNysMK",
        "outputId": "587e39ac-8962-4ebe-df94-f45a64df94df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "============================================================\n",
            "序列并行(SP)策略演示\n",
            "模拟逐位置操作的序列并行（如前馈网络）\n",
            "============================================================\n",
            "生成测试数据: batch_size=1, seq_len=2048, d_model=512\n",
            "输入数据范围: [-4.5161, 4.5343]\n",
            "\n",
            "1. 串行处理 (基准):\n",
            "\n",
            "串行处理 - 完整序列 (2048 tokens)\n",
            "串行处理完成，耗时: 0.1974秒\n",
            "\n",
            "2. 序列并行处理:\n",
            "\n",
            "序列并行处理 - 将序列分成2个片段\n",
            "每个片段大小: 1024 tokens\n",
            "  片段1数据范围: [-4.5161, 4.4615]\n",
            "  片段2数据范围: [-4.4507, 4.5343]\n",
            "  分配任务给工作线程1: 处理token范围 [0:1024]\n",
            "  分配任务给工作线程2: 处理token范围 [1024:2048]\n",
            "\n",
            "开始并行处理...\n",
            "  工作线程2: 处理1024个token，耗时: 0.1608秒\n",
            "  工作线程1: 处理1024个token，耗时: 0.1624秒\n",
            "合并结果...\n",
            "序列并行处理完成，总耗时: 0.1641秒\n",
            "\n",
            "3. 验证计算结果:\n",
            "\n",
            "结果验证:\n",
            "  串行输出范围: [-4.4356, 4.6646]\n",
            "  并行输出范围: [-4.4356, 4.6646]\n",
            "  最大绝对差异: 0.000000e+00\n",
            "  最大相对差异: 0.000000e+00\n",
            "序列并行计算结果与串行计算结果一致！\n",
            "\n",
            "============================================================\n",
            "性能对比\n",
            "============================================================\n",
            "串行处理耗时: 0.1974秒\n",
            "序列并行(2线程)处理耗时: 0.1641秒\n",
            "加速比: 1.20倍\n",
            "性能提升: 16.9%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4 PP（Pipeline Parallelism）策略\n",
        "\n",
        "构建一个流水线并行的演示：假设模型有两层，我们将这两层分别放在两个线程（或设备）上。流水线并行中，数据被分成多个微批次（micro-batches），每个微批次依次通过模型的各个阶段（层）。在这个例子中，有两个阶段（两个线程），每个线程处理模型的一层。模拟：将一个批次的数据分成两个微批次，然后通过流水线的方式处理。"
      ],
      "metadata": {
        "id": "ZV2WOjF64ouA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import time\n",
        "import threading\n",
        "import numpy as np\n",
        "from queue import Queue\n",
        "from typing import Tuple, List\n",
        "\n",
        "class SimpleNNLayer1(nn.Module):\n",
        "    \"\"\"第一层神经网络\"\"\"\n",
        "    def __init__(self, input_dim: int = 512, hidden_dim: int = 1024):\n",
        "        super().__init__()\n",
        "        self.linear = nn.Linear(input_dim, hidden_dim)\n",
        "        self.norm = nn.LayerNorm(hidden_dim)\n",
        "\n",
        "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
        "        x = self.linear(x)\n",
        "        x = self.norm(x)\n",
        "        x = F.relu(x)\n",
        "        return x\n",
        "\n",
        "class SimpleNNLayer2(nn.Module):\n",
        "    \"\"\"第二层神经网络\"\"\"\n",
        "    def __init__(self, hidden_dim: int = 1024, output_dim: int = 512):\n",
        "        super().__init__()\n",
        "        self.linear = nn.Linear(hidden_dim, output_dim)\n",
        "        self.norm = nn.LayerNorm(output_dim)\n",
        "\n",
        "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
        "        x = self.linear(x)\n",
        "        x = self.norm(x)\n",
        "        x = torch.tanh(x)\n",
        "        return x\n",
        "\n",
        "class PipelineParallelSimulator:\n",
        "    def __init__(self, batch_size: int = 8, seq_len: int = 32, feature_dim: int = 512):\n",
        "        \"\"\"\n",
        "        初始化流水线并行模拟器\n",
        "\n",
        "        Args:\n",
        "            batch_size: 批次大小\n",
        "            seq_len: 序列长度\n",
        "            feature_dim: 特征维度\n",
        "        \"\"\"\n",
        "        self.batch_size = batch_size\n",
        "        self.seq_len = seq_len\n",
        "        self.feature_dim = feature_dim\n",
        "\n",
        "        # 固定随机种子\n",
        "        torch.manual_seed(42)\n",
        "        np.random.seed(42)\n",
        "\n",
        "        # 创建完整模型\n",
        "        self.full_model = nn.Sequential(\n",
        "            SimpleNNLayer1(feature_dim, 1024),\n",
        "            SimpleNNLayer2(1024, feature_dim)\n",
        "        )\n",
        "\n",
        "        # 创建分层的模型（用于流水线并行）\n",
        "        self.layer1 = SimpleNNLayer1(feature_dim, 1024)\n",
        "        self.layer2 = SimpleNNLayer2(1024, feature_dim)\n",
        "\n",
        "        # 生成测试数据\n",
        "        self.input_data = None\n",
        "        self.generate_data()\n",
        "\n",
        "        # 同步两个模型的权重\n",
        "        self.sync_weights()\n",
        "\n",
        "    def generate_data(self):\n",
        "        \"\"\"生成测试数据\"\"\"\n",
        "        # 固定随机种子生成输入数据\n",
        "        self.input_data = torch.randn(self.batch_size, self.seq_len, self.feature_dim)\n",
        "        print(f\"生成测试数据: batch_size={self.batch_size}, seq_len={self.seq_len}, feature_dim={self.feature_dim}\")\n",
        "\n",
        "    def sync_weights(self):\n",
        "        \"\"\"同步完整模型和分层模型的权重\"\"\"\n",
        "        # 从完整模型中提取第一层权重\n",
        "        self.layer1.load_state_dict({\n",
        "            'linear.weight': self.full_model[0].linear.weight.data.clone(),\n",
        "            'linear.bias': self.full_model[0].linear.bias.data.clone(),\n",
        "            'norm.weight': self.full_model[0].norm.weight.data.clone(),\n",
        "            'norm.bias': self.full_model[0].norm.bias.data.clone()\n",
        "        })\n",
        "\n",
        "        # 从完整模型中提取第二层权重\n",
        "        self.layer2.load_state_dict({\n",
        "            'linear.weight': self.full_model[1].linear.weight.data.clone(),\n",
        "            'linear.bias': self.full_model[1].linear.bias.data.clone(),\n",
        "            'norm.weight': self.full_model[1].norm.weight.data.clone(),\n",
        "            'norm.bias': self.full_model[1].norm.bias.data.clone()\n",
        "        })\n",
        "\n",
        "        # 设置为评估模式\n",
        "        self.full_model.eval()\n",
        "        self.layer1.eval()\n",
        "        self.layer2.eval()\n",
        "\n",
        "    def sequential_processing(self) -> torch.Tensor:\n",
        "        \"\"\"顺序处理：整个模型一次性计算\"\"\"\n",
        "        print(f\"\\n顺序处理 (基准):\")\n",
        "        print(f\"  批次大小: {self.batch_size}, 一次性通过完整的两层模型\")\n",
        "\n",
        "        with torch.no_grad():\n",
        "            output = self.full_model(self.input_data)\n",
        "\n",
        "        print(f\"  顺序处理完成\")\n",
        "        return output\n",
        "\n",
        "    def pipeline_worker_stage1(self,\n",
        "                              input_queue: Queue,\n",
        "                              output_queue: Queue,\n",
        "                              num_microbatches: int):\n",
        "        \"\"\"流水线工作线程 - 阶段1（第一层）\"\"\"\n",
        "        print(\"  阶段1线程: 启动\")\n",
        "\n",
        "        for i in range(num_microbatches):\n",
        "            # 从输入队列获取数据\n",
        "            data = input_queue.get()\n",
        "            if data is None:  # 结束信号\n",
        "                break\n",
        "\n",
        "            # 处理数据（第一层）\n",
        "            with torch.no_grad():\n",
        "                output = self.layer1(data)\n",
        "\n",
        "            # 将结果放入输出队列（传递给阶段2）\n",
        "            output_queue.put(output)\n",
        "\n",
        "            print(f\"    阶段1: 处理微批次{i+1}/{num_microbatches}完成\")\n",
        "\n",
        "        # 发送结束信号\n",
        "        output_queue.put(None)\n",
        "        print(\"  阶段1线程: 结束\")\n",
        "\n",
        "    def pipeline_worker_stage2(self,\n",
        "                              input_queue: Queue,\n",
        "                              output_queue: Queue,\n",
        "                              num_microbatches: int):\n",
        "        \"\"\"流水线工作线程 - 阶段2（第二层）\"\"\"\n",
        "        print(\"  阶段2线程: 启动\")\n",
        "\n",
        "        final_outputs = []\n",
        "\n",
        "        for i in range(num_microbatches):\n",
        "            # 从输入队列获取数据（来自阶段1）\n",
        "            data = input_queue.get()\n",
        "            if data is None:  # 结束信号\n",
        "                break\n",
        "\n",
        "            # 处理数据（第二层）\n",
        "            with torch.no_grad():\n",
        "                output = self.layer2(data)\n",
        "\n",
        "            # 收集最终输出\n",
        "            final_outputs.append(output)\n",
        "\n",
        "            print(f\"    阶段2: 处理微批次{i+1}/{num_microbatches}完成\")\n",
        "\n",
        "        # 合并所有微批次的输出\n",
        "        if final_outputs:\n",
        "            final_output = torch.cat(final_outputs, dim=0)\n",
        "        else:\n",
        "            final_output = torch.tensor([])\n",
        "\n",
        "        # 将最终结果放入输出队列\n",
        "        output_queue.put(final_output)\n",
        "\n",
        "        print(\"  阶段2线程: 结束\")\n",
        "\n",
        "    def pipeline_parallel_processing(self, num_microbatches: int = 4) -> torch.Tensor:\n",
        "        \"\"\"流水线并行处理：将批次数据分成微批次，通过流水线处理\"\"\"\n",
        "        print(f\"\\n流水线并行处理:\")\n",
        "        print(f\"  将批次分成{num_microbatches}个微批次\")\n",
        "        print(f\"  微批次大小: {self.batch_size // num_microbatches}\")\n",
        "\n",
        "        # 计算微批次大小\n",
        "        microbatch_size = self.batch_size // num_microbatches\n",
        "\n",
        "        # 创建队列用于线程间通信\n",
        "        input_queue_stage1 = Queue()\n",
        "        intermediate_queue = Queue()\n",
        "        output_queue_stage2 = Queue()\n",
        "\n",
        "        # 创建并启动阶段1线程\n",
        "        stage1_thread = threading.Thread(\n",
        "            target=self.pipeline_worker_stage1,\n",
        "            args=(input_queue_stage1, intermediate_queue, num_microbatches)\n",
        "        )\n",
        "\n",
        "        # 创建并启动阶段2线程\n",
        "        stage2_thread = threading.Thread(\n",
        "            target=self.pipeline_worker_stage2,\n",
        "            args=(intermediate_queue, output_queue_stage2, num_microbatches)\n",
        "        )\n",
        "\n",
        "        print(\"启动流水线...\")\n",
        "        stage1_thread.start()\n",
        "        stage2_thread.start()\n",
        "\n",
        "        # 主线程：将微批次数据送入阶段1\n",
        "        for i in range(num_microbatches):\n",
        "            # 获取当前微批次数据\n",
        "            start_idx = i * microbatch_size\n",
        "            end_idx = start_idx + microbatch_size\n",
        "            microbatch = self.input_data[start_idx:end_idx]\n",
        "\n",
        "            # 放入阶段1的输入队列\n",
        "            input_queue_stage1.put(microbatch)\n",
        "\n",
        "            print(f\"  主线程: 发送微批次{i+1}/{num_microbatches}到阶段1\")\n",
        "\n",
        "        # 发送结束信号到阶段1\n",
        "        input_queue_stage1.put(None)\n",
        "\n",
        "        # 等待阶段2完成并获取最终结果\n",
        "        pipeline_output = output_queue_stage2.get()\n",
        "\n",
        "        # 等待所有线程结束\n",
        "        stage1_thread.join()\n",
        "        stage2_thread.join()\n",
        "\n",
        "        print(\"流水线并行处理完成\")\n",
        "        return pipeline_output\n",
        "\n",
        "    def validate_result(self, sequential_output: torch.Tensor, pipeline_output: torch.Tensor) -> bool:\n",
        "        \"\"\"验证两种方法的结果是否一致\"\"\"\n",
        "        print(\"\\n结果验证:\")\n",
        "\n",
        "        # 检查形状是否相同\n",
        "        if sequential_output.shape != pipeline_output.shape:\n",
        "            print(f\"  形状不匹配: 顺序处理{sequential_output.shape} vs 流水线{pipeline_output.shape}\")\n",
        "            return False\n",
        "\n",
        "        # 计算差异\n",
        "        abs_diff = torch.max(torch.abs(sequential_output - pipeline_output)).item()\n",
        "        rel_diff = torch.max(torch.abs(sequential_output - pipeline_output) / (torch.abs(sequential_output) + 1e-10)).item()\n",
        "\n",
        "        print(f\"  最大绝对差异: {abs_diff:.6e}\")\n",
        "        print(f\"  最大相对差异: {rel_diff:.6e}\")\n",
        "\n",
        "        # 检查是否在可接受的误差范围内（浮点数计算误差）\n",
        "        tolerance = 1e-6\n",
        "        is_valid = abs_diff < tolerance\n",
        "\n",
        "        if is_valid:\n",
        "            print(\"流水线并行计算结果与顺序计算结果一致！\")\n",
        "        else:\n",
        "            print(\"计算结果存在显著差异\")\n",
        "\n",
        "        return is_valid\n",
        "\n",
        "\n",
        "def main():\n",
        "    \"\"\"主函数\"\"\"\n",
        "    print(\"=\"*60)\n",
        "    print(\"流水线并行(PP)策略演示\")\n",
        "    print(\"模拟两层模型的流水线并行处理\")\n",
        "    print(\"=\"*60)\n",
        "\n",
        "    # 创建模拟器\n",
        "    batch_size = 8\n",
        "    seq_len = 32\n",
        "    feature_dim = 512\n",
        "\n",
        "    simulator = PipelineParallelSimulator(\n",
        "        batch_size=batch_size,\n",
        "        seq_len=seq_len,\n",
        "        feature_dim=feature_dim\n",
        "    )\n",
        "\n",
        "    # 1. 顺序处理\n",
        "    print(\"\\n1. 顺序处理 (基准):\")\n",
        "    sequential_output = simulator.sequential_processing()\n",
        "\n",
        "    # 2. 流水线并行处理\n",
        "    print(\"\\n2. 流水线并行处理:\")\n",
        "    pipeline_output = simulator.pipeline_parallel_processing(num_microbatches=4)\n",
        "\n",
        "    # 3. 验证结果\n",
        "    print(\"\\n3. 验证计算结果:\")\n",
        "    simulator.validate_result(sequential_output, pipeline_output)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    main()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SIIjOJSZ4pKb",
        "outputId": "8785a0f0-1b98-4e6d-bb86-358ef601ed44"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "============================================================\n",
            "流水线并行(PP)策略演示\n",
            "模拟两层模型的流水线并行处理\n",
            "============================================================\n",
            "生成测试数据: batch_size=8, seq_len=32, feature_dim=512\n",
            "\n",
            "1. 顺序处理 (基准):\n",
            "\n",
            "顺序处理 (基准):\n",
            "  批次大小: 8, 一次性通过完整的两层模型\n",
            "  顺序处理完成\n",
            "\n",
            "2. 流水线并行处理:\n",
            "\n",
            "流水线并行处理:\n",
            "  将批次分成4个微批次\n",
            "  微批次大小: 2\n",
            "启动流水线...\n",
            "  阶段1线程: 启动\n",
            "  阶段2线程: 启动\n",
            "  主线程: 发送微批次1/4到阶段1\n",
            "  主线程: 发送微批次2/4到阶段1\n",
            "  主线程: 发送微批次3/4到阶段1\n",
            "  主线程: 发送微批次4/4到阶段1\n",
            "    阶段1: 处理微批次1/4完成\n",
            "    阶段1: 处理微批次2/4完成\n",
            "    阶段2: 处理微批次1/4完成\n",
            "    阶段1: 处理微批次3/4完成\n",
            "    阶段2: 处理微批次2/4完成\n",
            "    阶段1: 处理微批次4/4完成\n",
            "  阶段1线程: 结束\n",
            "    阶段2: 处理微批次3/4完成\n",
            "    阶段2: 处理微批次4/4完成\n",
            "  阶段2线程: 结束\n",
            "流水线并行处理完成\n",
            "\n",
            "3. 验证计算结果:\n",
            "\n",
            "结果验证:\n",
            "  最大绝对差异: 0.000000e+00\n",
            "  最大相对差异: 0.000000e+00\n",
            "流水线并行计算结果与顺序计算结果一致！\n"
          ]
        }
      ]
    }
  ]
}