{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9d39848f",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"https://mng.bz/lZ5B\">Build a Reasoning Model (From Scratch)</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/reasoning-from-scratch\">https://github.com/rasbt/reasoning-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"https://mng.bz/lZ5B\"><img src=\"https://sebastianraschka.com/images/reasoning-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aedcb1d",
   "metadata": {},
   "source": [
    "# Chapter 8: Exercise Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e745d0",
   "metadata": {},
   "source": [
    "Packages that are being used in this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f8c0639",
   "metadata": {},
   "outputs": [],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "used_libraries = [\n",
    "    \"reasoning_from_scratch\",\n",
    "    \"torch\",\n",
    "    \"tokenizers\"  # Used by reasoning_from_scratch\n",
    "]\n",
    "\n",
    "for lib in used_libraries:\n",
    "    print(f\"{lib} version: {version(lib)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "891543d5",
   "metadata": {},
   "source": [
    "&nbsp;\n",
    "## Exercise 8.1: Training and validation set lengths      "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ece56f7",
   "metadata": {},
   "source": [
    "- The following cells recreate the end of chapter section 8.4.3 and compute the training and validation length statistics using reusable helpers from `reasoning_from_scratch.ch08`\n",
    "- If you still have the chapter 8 notebook open, you could also ignore the code below and just add excute the following:\n",
    "\n",
    "```python\n",
    "print(\"Training stats:\")\n",
    "compute_length(train_examples)\n",
    "\n",
    "print(\"Validation stats:\")\n",
    "compute_length(val_examples)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0cae3987-c443-4fe7-aa47-b7d95f4a5ccd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "from reasoning_from_scratch.ch08 import (\n",
    "    build_examples,\n",
    "    compute_length,\n",
    "    filter_examples_by_max_len,\n",
    "    load_distill_data,\n",
    "    load_reasoning_tokenizer,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4b11ba88-c212-4dbc-87d8-90530cff6fd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deepseek-r1-math-train.json: 107538.0 KB (cached)\n",
      "✓ qwen3/tokenizer-reasoning.json already up-to-date\n",
      "Number of examples: 12000\n",
      "Number of skipped examples: 0\n",
      "Average: 2946 tokens\n",
      "Shortest: 236 tokens (index 10846)\n",
      "Longest: 42005 tokens (index 2529)\n",
      "Original: 12000\n",
      "Filtered: 6695\n",
      "Removed: 5305\n",
      "Average: 1180 tokens\n",
      "Shortest: 236 tokens (index 5971)\n",
      "Longest: 2048 tokens (index 5587)\n"
     ]
    }
   ],
   "source": [
    "math_train = load_distill_data(partition=\"deepseek-r1-math-train\")\n",
    "tokenizer = load_reasoning_tokenizer()\n",
    "\n",
    "examples, skipped = build_examples(math_train, tokenizer)\n",
    "print(\"Number of examples:\", len(examples))\n",
    "print(\"Number of skipped examples:\", skipped)\n",
    "\n",
    "compute_length(examples)\n",
    "\n",
    "filtered_examples = filter_examples_by_max_len(examples, max_len=2048)\n",
    "compute_length(filtered_examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "468791d3-d526-4337-9a7d-9a4cb78975c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = random.Random(123)\n",
    "rng.shuffle(filtered_examples)\n",
    "\n",
    "train_examples = filtered_examples[25:]\n",
    "val_examples = filtered_examples[:25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f051b95c-49d5-4c6b-92f4-c708f430c95e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training stats:\n",
      "Average: 1180 tokens\n",
      "Shortest: 236 tokens (index 5730)\n",
      "Longest: 2048 tokens (index 1319)\n"
     ]
    }
   ],
   "source": [
    "print(\"Training stats:\")\n",
    "compute_length(train_examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7d656a19-056c-42ea-b190-0a1a7f973199",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation stats:\n",
      "Average: 1106 tokens\n",
      "Shortest: 481 tokens (index 15)\n",
      "Longest: 1918 tokens (index 12)\n"
     ]
    }
   ],
   "source": [
    "print(\"Validation stats:\")\n",
    "compute_length(val_examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11c484b7-fcf6-47eb-b18e-c8d5a9f220fb",
   "metadata": {},
   "source": [
    "- As we can see, the average token length (1180 versus 1106) is fairly similar, and the datasets should be relatively balanced\n",
    "- As a bonus, we can also plot histograms to visualize the distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4d26d1e0-8fdc-40b9-ae81-d64600278e41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAGGCAYAAACHemKmAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAARBJJREFUeJzt3Ql4FFXa//07JGEJCAGBLIAQAdn3TRBFJCMoKrgNMDyCyKIIDoiABllkMwqCgPCIDMOmIssMogKiyOYMIPsqyIAiixCWgbCvSf+v+zxv1dtNOiEhCd3V+X6uq2m66nTVSXcn+eXUqbuCXC6XSwAAAACHyeXrDgAAAAC3gyALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJEIsgAAAHCkEF93ICdLTk6Wo0ePyl133SVBQUG+7g4AAIDP6bW6zp8/L9HR0ZIrV9pjrgRZH9IQW6pUKV92AQAAwC8dPnxYSpYsmWYbgqwP6Uis9UYVLFjQl10BAADwC+fOnTMDfVZOSgtB1oes6QQaYgmyAAAA/7/0TLvkZC8AAAA4EkEWAAAAjkSQBQAAgCMxRxYAAPi9pKQkuX79uq+7gSwQGhoqwcHBWbEpgiwAAPDvmqIJCQmSmJjo664gC4WHh0tkZGSm6+gzIgsAAPyWFWKLFy8uYWFhXEAoAP4wuXTpkpw4ccI8joqKytT2CLIAAMBvpxNYIfbuu+/2dXeQRfLly2fuNczqe5uZaQac7AUAAPySNSdWR2IRWML+v/c0s/OeCbIAAMCvZXYeJQL3PSXIAgAAwJEIsgAAAA5QpkwZGTdunK+74Vc42QsAADhO3IKdd3R/8c9Uy7LD5kOGDJF33nknw33YuHGj5M+fP8PPC2QEWQAAgCx07Ngx+/9z586VwYMHy969e+1lBQoU8ChHpdUZQkJuHcmKFSvG+3QTgiwA4P980yvzr8ST43k1keNpoX9LoUKFzAittWzVqlXStGlTWbJkiQwcOFB27twp33//vZQqVUr69OkjP/30k1y8eFEqVaok8fHxEhsb6zG1oHfv3uamgoKC5G9/+5ssXrxYvvvuOylRooSMGTNGnnrqqRzzHjBHFgAA4A5766235L333pM9e/ZI9erV5cKFC/L444/L8uXLZevWrdKiRQt58skn5dChQ2luZ+jQofLnP/9ZduzYYZ7fvn17OX36tOQUBFkAAIA7bNiwYfKnP/1JypYtK0WKFJEaNWrIyy+/LFWrVpXy5cvL8OHDzbqvv/46ze28+OKL0q5dOylXrpy8++67JhBv2LBBcgqCLAAAwB1Wt25dj8caQPv27WumFISHh5t5tDpae6sR2erVq9v/1xPBChYsaF/+NSdgjiwAAMAddnP1AQ2xy5Ytkw8++MCMruplXJ977jm5du1amtsJDQ31eKzzZpOTkyWnIMgCAAD42Jo1a8w0gaefftoeof3999993S2/x9QCAAAAH9N5sQsWLJBt27bJ9u3b5S9/+UuOGlm9XQRZAAAAHxs7dqwULlxYGjVqZKoVNG/eXGrXru3rbvm9IJdW4oVPnDt3ztSXO3v2rJmcDQA+RR1Z+JkrV67IgQMHJCYmRvLmzevr7uAOvbcZyUeMyAIAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEciyAIAAMCRCLIAAABwJIIsAAAAHMnnQXbSpElSpkwZU0OsQYMGsmHDhjTbz58/XypWrGjaV6tWTZYsWeKxXsviDh48WKKiosx1imNjY2Xfvn0ebUaOHGkKDoeFhUl4eHiKfcyYMcNcq9jb7cSJE6bNqlWrvK5PSEjIktcFAAAAfhxk586dK3369JEhQ4bIli1bpEaNGuZKFlZYvNnatWulXbt20rlzZ9m6dau0bt3a3Hbt2mW3GTVqlEyYMEEmT54s69evl/z585ttauFdy7Vr1+T555+X7t27e91PmzZt5NixYx433UaTJk2kePHiHm337t3r0e7m9QAAABn18MMPS+/eve3HOug3bty4NJ8TFBQkCxcuzPSLnVXbCfggq5dj69q1q3Tq1EkqV65swqeOkk6bNs1r+/Hjx0uLFi2kX79+UqlSJRk+fLi5fNvEiRPt0Vh9kwcOHCitWrWS6tWry6xZs+To0aMeb8jQoUPl9ddfNyO63uhIbmRkpH0LDg6WFStWmAB9Mw2u7m1z5fL5IDcAAPAhvcSs5hVv/vWvf5mguGPHjgxtc+PGjdKtWzfJSu+8847UrFkzxXIdmHvsscfECUJ8tWMdFd28ebPExcXZyzQE6lSAdevWeX2OLtcRXHc6UmqFVL3UmR7a121Y9BJnOmVBn9u2bdvb6quGYQ3Yzz33XIp1+gG4evWqVK1a1XwgHnjggVS3o+305n4JNgAA4KNLKmfEk+PT3VQHvp599lk5cuSIlCxZ0mPd9OnTpW7dumawLSOKFSsmd0pkZKQ4hc+GD0+dOiVJSUkSERHhsVwfpzbPVJen1d66z8g20+Pvf/+7/OUvfzEjtRadg6sjyP/85z/NrVSpUuYwgE6RSE18fLwJ1tZNnwMAAALLE088YYKnnnPj7sKFC+ZcH50WqVMlS5QoYQbK9AjxF198keY2b55aoOf/PPTQQ+acIT2qvWzZshTPefPNN+W+++4z+7j33ntl0KBBcv36dbNO+6ZHqLdv326f52P19+apBTt37pRHHnnE5KC7777bjAzr12J58cUXzdf0wQcfmHykbXr06GHvKyBHZJ1CR3L37Nkjn376qcfyChUqmJtFTx779ddf5cMPP0zR1qKjz+4jyjoiS5gFACCwhISESIcOHUwwfPvtt00wVBpidRDvf/7nf8z/NWgWLFhQFi9eLC+88IKULVtW6tevf8vtJycnyzPPPGMG6vR8oLNnz3rMp7Xcddddpg/R0dEmjOp0Tl3Wv39/cz6QnmO0dOlS+eGHH0x7HWS72cWLF83R74YNG5rpDXoeU5cuXaRnz54eQX3lypUmxOr9/v37zfb1qLXuMyBHZIsWLWrmnh4/ftxjuT5ObUhbl6fV3rrPyDZvZerUqeaNqFOnzi3b6odP37zU5MmTx3xg3W8AACDwvPTSS2aAa/Xq1R7TCnTKQenSpaVv374mX+hI6WuvvWbm1M6bNy9d29bg+csvv5ipj3qivI7Mvvvuuyna6TlDOtCmo7k6b1f3ae1DR1cLFChgQrd1no/7kWfL7NmzzQnzui+dRqkjs3pukg7aueetwoULm+VaWUpHpFu2bCnLly+X7OazIJs7d24TDt2/SP0LQx9r6vdGl9/8ouhQutU+JibGvBHubXTUU/9aSW2badFhc33DvZ3k5c22bdvMXyMAACBn00CnIdI6gV0HuvREL80UOiqrJ6zrlIIiRYqYQPndd9/JoUOH0rVtPVKsR3Sjo6PtZd5yjlaH0nN3NBvpPjTYpncf7vvSsKxVoCy6Tc1sWrnJUqVKFTNAadE8lFoVqoCZWqCH2Tt27GgmPetops790CFsrWKgdFhe54/o3FLVq1cvUwJrzJgxJunPmTNHNm3aJFOmTDHrdeheh9ZHjBgh5cuXN8FW54PoG61zNyz6Jp4+fdrc64dJA6gqV66ceaPdPwA3btwwhwBupn3V7esbp3+p6MitVjb4/vvvs/11AwAA/k9Dq462as18HY3VqQOaY95//31TiUmzhIZZDYmaX/RE+KycGtm+fXszD1anBui0Ac1NmqGyQ2hoqMdjzWQadgM6yOr8iZMnT5oLGOjJWDrErnM1rJO1NGi6l7PSv2x0iFv/ohgwYIAJqzoZWYe6LTrvQ8OwTkROTEyUxo0bm23qZGiL7m/mzJn241q1apl7ndehJ2y5n+Slc1C8XTRBP2xvvPGG/PHHH2YStZ59qEP9TZs2zYZXCgAAOM2f//xnMwin2UUPzWv9eg14a9asMWVCrYEyDXz/+c9/zElb6aElSA8fPmzKZFlHgn/66acUtfd1CoPO0bUcPHgwxdFxHdC71b50LqxmK2tUVvuv+cz9XCFf8fnJXjpZWG/e6NWzbqYXMtBbavQDMmzYMHNLjb4hN59J6I1+CFKjgVlvAAAA3uhRXh2005O9daqjnt2vdCDuH//4h8kZOrdU6+rrfNP0BlktM6rVCDp27CijR48223YPrNY+dEBQR2Hr1atnTij78ssvPdro3FktXapHprVMmJ4IpufzuNNRXb1wle5Ly4zqAKSOMuvJaTdXifIFqvcDAABk4/SCM2fOmMP71pxWPbKsF3TSZXokWOewuk+BvBUdDdVQevnyZTM1U6sIjBw50qPNU089ZS7+pIOFesRbQ7NOt3SnJ57pSWZ6NFnLhXkrAaZHnXX+rk7J1ECsNfWbNWtmX4zK14Jcejks+IT+BaVzVrRsBhUMAAREgfkMFI0HbkXPQdERQz0nxX2KIAL7vT2XgXzEiCwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAciSALAAD8GgWWAo8ri4pmEWQBAIBfsi57eunSJV93BVnMek9vvrSt467sBQAA4E1wcLC5TPyJEyfs4vx6BU84eyT20qVL5j3V91bf48wgyAIAAL+lV71SVphFYAgPD7ff28wgyAIAAL+lI7BRUVFSvHhxuX79uq+7gyyg0wkyOxJrIcgCAAC/p8Enq8IPAgcnewEAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEciyAIAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEciyAIAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEciyAIAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEciyAIAAMCRfB5kJ02aJGXKlJG8efNKgwYNZMOGDWm2nz9/vlSsWNG0r1atmixZssRjvcvlksGDB0tUVJTky5dPYmNjZd++fR5tRo4cKY0aNZKwsDAJDw/3up+goKAUtzlz5ni0WbVqldSuXVvy5Mkj5cqVkxkzZtz26wAAAAAHBdm5c+dKnz59ZMiQIbJlyxapUaOGNG/eXE6cOOG1/dq1a6Vdu3bSuXNn2bp1q7Ru3drcdu3aZbcZNWqUTJgwQSZPnizr16+X/Pnzm21euXLFbnPt2jV5/vnnpXv37mn2b/r06XLs2DH7pvuyHDhwQFq2bClNmzaVbdu2Se/evaVLly7y3XffZclrAwAAgLQFuXQI00d0BLZevXoyceJE8zg5OVlKlSolr732mrz11lsp2rdp00YuXrwoixYtspfdf//9UrNmTRNc9UuJjo6WN954Q/r27WvWnz17ViIiIsxoadu2bT22p8s0gCYmJqbYl47Afvnllx7h1d2bb74pixcv9gjRun3d1tKlS9P19Z87d04KFSpk+liwYMF0PQcAss03vTK/jSfHZ0VPAORg5zKQj3w2Iqujops3bzaH/u3O5MplHq9bt87rc3S5e3ulo61Wex0lTUhI8GijL4QG5tS2mZYePXpI0aJFpX79+jJt2jQTlNPbFwAAAGSvEPGRU6dOSVJSkhktdaePf/nlF6/P0ZDqrb0ut9Zby1Jrk17Dhg2TRx55xMyj/f777+XVV1+VCxcuyF//+tc0+6J/RVy+fNnMz73Z1atXzc2ibQEAAOCwIOvvBg0aZP+/Vq1aZkrD6NGj7SB7O+Lj42Xo0KFZ1EMAAICczWdTC/SQfXBwsBw/ftxjuT6OjIz0+hxdnlZ76z4j20wvnZ5w5MgRe0Q1tb7oXA5vo7EqLi7OzPewbocPH85UnwAAAHIynwXZ3LlzS506dWT58uX2Mj3ZSx83bNjQ63N0uXt7tWzZMrt9TEyMCZjubfTwvVYvSG2b6aWVCQoXLmxKbaWnL97oczXout8AAADgwKkFWnqrY8eOUrduXXNC1bhx48wh/E6dOpn1HTp0kBIlSphD8qpXr17SpEkTGTNmjCl9pXVdN23aJFOmTLErDWgVghEjRkj58uVNsNUpAlrJwL36wKFDh+T06dPmXufpakhVWgu2QIEC8s0335jRVa2IoPVqNaC+++67diUE9corr5hqC/3795eXXnpJVqxYIfPmzTOVDAAAABDgQVbLaZ08edJcwEBPntIyWlq6yjqJSoOmVjKw6EUMZs+eLQMHDpQBAwaYsLpw4UKpWrWq3UaDpYbhbt26mVJYjRs3NtvUQGrR/c2cOdNjDqxauXKlPPzwwxIaGmou1PD666+bSgUacMeOHStdu3a1n6MhWUOrthk/fryULFlSpk6daioXAAAAIMDryOZ01JEF4FeoIwvADziijiwAAACQGQRZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI7k8yA7adIkKVOmjOTNm1caNGggGzZsSLP9/PnzpWLFiqZ9tWrVZMmSJR7rXS6XDB48WKKioiRfvnwSGxsr+/bt82gzcuRIadSokYSFhUl4eHiKfWzfvl3atWsnpUqVMtuoVKmSjB8/3qPNqlWrJCgoKMUtISEhU68HAAAAHBBk586dK3369JEhQ4bIli1bpEaNGtK8eXM5ceKE1/Zr1641AbNz586ydetWad26tbnt2rXLbjNq1CiZMGGCTJ48WdavXy/58+c327xy5Yrd5tq1a/L8889L9+7dve5n8+bNUrx4cfnss8/k559/lrffflvi4uJk4sSJKdru3btXjh07Zt/0eQAAAMh+QS4dwvQRHYGtV6+eHRCTk5PNKOhrr70mb731Vor2bdq0kYsXL8qiRYvsZffff7/UrFnTBFf9UqKjo+WNN96Qvn37mvVnz56ViIgImTFjhrRt29Zje7qsd+/ekpiYeMu+9ujRQ/bs2SMrVqywR2SbNm0qZ86c8Tqqmx7nzp2TQoUKmT4WLFjwtrYBAFnmm16Z38aTnkevACA785HPRmR1VFRHPvXQv92ZXLnM43Xr1nl9ji53b690tNVqf+DAAXNo372NvhAamFPbZnrpi1mkSJEUyzVE6zSGP/3pT7JmzZo0t3H16lXz5rjfAAAAcHtCxEdOnTolSUlJZrTUnT7+5ZdfvD5HQ6q39ta8VOs+rTa3Q6c06DSIxYsX28s0vOoocN26dU1AnTp1qjz88MNmOkPt2rW9bic+Pl6GDh162/0AsnT0LFBGzvxhFNEf+pBV/QgE/vC94S+fCSDA+SzIOoXOv23VqpWZx/voo4/ayytUqGBuFj157Ndff5UPP/xQPv30U6/b0nm2OifYoiOyOpUCAAAAGeezqQVFixaV4OBgOX78uMdyfRwZGen1Obo8rfbWfUa2mZbdu3dLs2bNpFu3bjJw4MBbtq9fv77s378/1fV58uQxcz3cbwAAAHBYkM2dO7fUqVNHli9fbi/Tk730ccOGDb0+R5e7t1fLli2z28fExJjA6t5GRz31cH9q20yNVivQk7k6duxoynWlx7Zt28yUAwAAAAT41AI9zK5BUeeZ6mjmuHHjTFWCTp06mfUdOnSQEiVKmLmlqlevXtKkSRMZM2aMtGzZUubMmSObNm2SKVOmmPVax1WrEIwYMULKly9vgu2gQYNMJQMt02U5dOiQnD592tzrPF0NoKpcuXJSoEABM53gkUceMSeSaR+t+bU6glysWDHzf+2rbr9KlSqmtJfOkdWKBt9///0dfx0BAAByIp8GWS2ndfLkSXMBAw2LWgFg6dKl9slaGjS1koH7PNTZs2ebw/wDBgwwYXXhwoVStWpVu03//v1NGNbpAFpWq3HjxmabegEFi+5v5syZ9uNatWqZ+5UrV5oTtv7xj3+YfmkdWb1ZSpcuLb///rtddUHLfP3xxx/mwgrVq1eXH374wYziAgAAIMDryOZ01JGF48/M9gf+cHa4P/Qhq/qRWf7wufKH7w1/+UwADuSIOrIAAABAZhBkAQAA4EgEWQAAADgSQRYAAACORJAFAACAIxFkAQAA4EgEWQAAADgSQRYAAACORJAFAACAIxFkAQAA4EgEWQAAADgSQRYAAACORJAFAACAIxFkAQAAkHOC7G+//Zb1PQEAAACyO8iWK1dOmjZtKp999plcuXLldjYBAAAA3Pkgu2XLFqlevbr06dNHIiMj5eWXX5YNGzZkricAAABAdgfZmjVryvjx4+Xo0aMybdo0OXbsmDRu3FiqVq0qY8eOlZMnT97OZgEAAIA7c7JXSEiIPPPMMzJ//nx5//33Zf/+/dK3b18pVaqUdOjQwQRcAAAAwO+C7KZNm+TVV1+VqKgoMxKrIfbXX3+VZcuWmdHaVq1aZV1PAQAAADchchs0tE6fPl327t0rjz/+uMyaNcvc58r1f7k4JiZGZsyYIWXKlLmdzQMAAADZE2Q//vhjeemll+TFF180o7HeFC9eXP7+97/fzuYBAACA7AmyOnXgnnvusUdgLS6XSw4fPmzW5c6dWzp27Hg7mwcAAACyZ45s2bJl5dSpUymWnz592kwrAAAAAPwyyOrIqzcXLlyQvHnzZrZPAAAAQNZOLdALIKigoCAZPHiwhIWF2euSkpJk/fr1psYsAAAA4FdBduvWrfaI7M6dO808WIv+v0aNGqYEFwAAAOBXQXblypXmvlOnTubKXgULFsyufgEAAABZX7VAa8gCAAAAjgiyeilavciBjsLq/9OyYMGCrOgbAAAAkPmqBYUKFTIneVn/T+uWEZMmTTJXANNqBw0aNJANGzak2X7+/PlSsWJF075atWqyZMkSj/U6f1dPRNMLNeTLl09iY2Nl3759Hm1GjhwpjRo1MierhYeHe93PoUOHpGXLlqaNXtyhX79+cuPGDY82q1atktq1a0uePHmkXLlyJugDAADAz0Zk3acTZNXUgrlz55pKCJMnTzYhdty4cdK8eXNz6VsNjzdbu3attGvXTuLj4+WJJ56Q2bNnS+vWrWXLli1StWpV02bUqFEyYcIEmTlzpqlpO2jQILPN3bt326XBrl27Js8//7w0bNjQ69XHtAKDhtjIyEizz2PHjkmHDh0kNDRU3n33XdPmwIEDps0rr7win3/+uSxfvly6dOliArTuDwAAAH5YR/by5cty6dIl+/HBgwdNCP3+++8ztJ2xY8dK165dzcljlStXNoFWR0CnTZvmtb2eYNaiRQszOlqpUiUZPny4GRGdOHGiPRqr/Rg4cKC0atVKqlevLrNmzZKjR4/KwoUL7e0MHTpUXn/9dTOi641+HRp8P/vsM1NO7LHHHjP70tFjDcFK+6pBecyYMaYvPXv2lOeee04+/PDDDL0GAAAAuINBVkOiBkSVmJgo9evXN4FOl3/88cfp2oYGws2bN5tD/3ZncuUyj9etW+f1Obrcvb3S0U+rvY6SJiQkeLTRqQ462pvaNlPbj4bciIgIj/2cO3dOfv7553T1BQAAAH4YZPVQ/oMPPmj+/49//MMcgtdRWQ23elg/PfQSt3oI3z0sKn2sYdQbXZ5We+s+I9vMyH7c95FaGw27OmLtzdWrV8169xsAAADuYJDVaQV33XWXfRheqxjoaOr9999vAi2807m97ifFlSpVipcKAADgTgZZPUNf55wePnxYvvvuO3n00UfN8hMnTqT7IglFixaV4OBgOX78uMdyfawjvN7o8rTaW/cZ2WZG9uO+j9Ta6Nev1RK8iYuLk7Nnz9o3ff0AAABwB4OslrfSS9Fq2Sydf6pn/1ujs7Vq1UrXNvSStnXq1DFn+1uSk5PNY2t7N9Pl7u3VsmXL7PZ68pUGTPc2evh+/fr1qW4ztf3oJXg1mLvvR0OqnpSWnr54o2W6dBvuNwAAANzBK3vp2fmNGzc2Zalq1KhhL2/WrJk8/fTT6d6Olt7q2LGj1K1b15wwphUHLl68aKoYKC15VaJECXNIXvXq1UuaNGliTizT0ldz5syRTZs2yZQpU8x6rXPbu3dvGTFihJQvX94uvxUdHW3KdLnXiD19+rS513m627Zts0eaCxQoYEaYNbC+8MILppyXzofVSgg9evQwYVRp2S2tltC/f3956aWXZMWKFTJv3jxZvHjx7bykAAAAuBNBVunI582H6zWMZkSbNm3k5MmTZoRXw6KWulq6dKl9EpUGTZ17a9GLGGjtWA2VAwYMMGFVpzhYNWSVBksNw926dTMVFTRw6zatGrJK96d1Zi3WKPLKlSvl4YcfNlMeFi1aJN27dzcjrPnz5zeBe9iwYfZzNCRraNUyXloWrGTJkjJ16lRqyAIAANwhQS4tvppBGhTfe+89c2hdD7/rlAB3v/32W1b2MWDptAc96UvnyzLNABn2Ta/MvWhPjg+MFz2zr0NWvBb+0Ies6kdm+cPnyh++N/zlMwEEeD66rRFZvYLV6tWrzaF3vZKVdelaAAAA4E65rSD77bffmsPqDzzwQNb3CAAAAMiuqgWFCxeWIkWK3M5TAQAAAN8F2eHDh5sTpvTCCAAAAIBjphZo+atff/3VVBfQWrKhoaEpLmELAAAA+F2Qda/JCgAAADgmyA4ZMiTrewIAAABk9xxZpRcb0AsAxMXFmatkWVMK/vjjj9vdJAAAAJC9I7I7duyQ2NhYU6z2999/l65du5oqBgsWLDBX45o1a9btbBYAAADI3hHZPn36yIsvvij79u3zuPTr448/Lj/++OPtbBIAAADI/iC7ceNGefnll1MsL1GihCQkJNzOJgEAAIDsD7J58uQx18G92X/+8x8pVqzY7WwSAAAAyP4g+9RTT8mwYcPk+vXr5nFQUJCZG/vmm2/Ks88+ezubBAAAALI/yOoFES5cuGBGXy9fvixNmjSRcuXKyV133SUjR468nU0CAAAA2V+1QKsVLFu2TNasWSPbt283obZ27dqmkgEAAADgl0E2OTlZZsyYYUptaektnVYQExMjkZGR4nK5zGMAAADAr6YWaFDV+bFdunQxFz6oVq2aVKlSRQ4ePGjKcT399NPZ11MAAADgdkdkdSRW68QuX75cmjZt6rFuxYoV0rp1a3MxhA4dOmRkswDgU3ELdmbq+fGhWdYVAEB2jch+8cUXMmDAgBQhVj3yyCPy1ltvyeeff56RTQIAAADZPyKrl6YdNWpUqusfe+wxmTBhwu31BACQKesPnM7U8xvEFOEdABC4Qfb06dMSERGR6npdd+bMmazoFwDkKJmd3qBaZ0lPACBApxYkJSVJSEjq2Tc4OFhu3LiRFf0CAAAAsm5EVqsWaHUCvUStN1evXs3I5gAAASbTJ849Uy3L+gIg8GUoyHbs2PGWbahYAAA5c46tUTIregIA2RBkp0+fnpHmAJAjEAABwAFzZAEAAAB/QZAFAABA4E8tAADA38uQcaU1IOdgRBYAAACORJAFAACAIxFkAQAA4Eh+EWQnTZokZcqUkbx580qDBg1kw4YNabafP3++VKxY0bSvVq2aLFmyJMWFGwYPHixRUVGSL18+iY2NlX379qW43G779u2lYMGCEh4eLp07d5YLFy7Y69955x0JCgpKccufP7/dZsaMGSnWa58A3NnSV5m9AQCcyedBdu7cudKnTx8ZMmSIbNmyRWrUqCHNmzeXEydOeG2/du1aadeunQmeW7duldatW5vbrl277DajRo2SCRMmyOTJk2X9+vUmfOo2r1y5YrfREPvzzz/LsmXLZNGiRfLjjz9Kt27d7PV9+/aVY8eOedwqV64szz//vEd/NAi7tzl48GC2vE4AAADwsyA7duxY6dq1q3Tq1MkERQ2fYWFhMm3aNK/tx48fLy1atJB+/fpJpUqVZPjw4VK7dm2ZOHGiPRo7btw4GThwoLRq1UqqV68us2bNkqNHj8rChQtNmz179sjSpUtl6tSpZgS4cePG8tFHH8mcOXNMO1WgQAGJjIy0b8ePH5fdu3ebAO1OR2Hd20VERGT7awYAAAAfB9lr167J5s2bzaF/S65cuczjdevWeX2OLndvr3S01Wp/4MABSUhI8GhTqFAhE1itNnqv0wnq1q1rt9H2um8dwfVGQ+99990nDz74oMdynY5QunRpKVWqlAnOOsqbmqtXr8q5c+c8bgAAAHBgkD116pQkJSWlGMXUxxpGvdHlabW37m/Vpnjx4h7rQ0JCpEiRIl73q1MSPv/88xSjsRUqVDAjx1999ZV89tlnkpycLI0aNZIjR4547Xt8fLwJ1dZNwy8AAABuDxdESIcvv/xSzp8/Lx07dvRY3rBhQ3OzaIjV6Q6ffPKJmfJws7i4ODMf2KIjsoRZAPAvXJQBcA6fBtmiRYtKcHCwmX/qTh/rfFNvrPmqqbW37nWZVi1wb1OzZk27zc0nk924ccNUMvC2X51W8MQTT9xy/mtoaKjUqlVL9u/f73V9njx5zA1A1oWG1ryYAJBj+XRqQe7cuaVOnTqyfPlye5kentfH7iOd7nS5e3ullQes9jExMSaMurfRkU+d+2q10fvExEQzP9eyYsUKs2+dS+tO59yuXLkyxbQCb3SaxM6dOz0CNAAAAAJ0aoEeatdD9nriVf369U3FgYsXL5oqBqpDhw5SokQJM79U9erVS5o0aSJjxoyRli1bmkoDmzZtkilTpthVBHr37i0jRoyQ8uXLm2A7aNAgiY6ONmW6lB7+18oHWi1BqyRcv35devbsKW3btjXt3OkcWA2mjz32WIq+Dxs2TO6//34pV66cCcajR4825be6dOlyB145AACAnM3nQbZNmzZy8uRJcwEDPdFKD/9raSzrMP6hQ4dMNQH3eaizZ8825bUGDBhgwqqW1apatardpn///iYMa11YDZhaXku36X6xAj15S8Nrs2bNzPafffZZU3vWnY7Q6kUPXnzxRTMF4mZnzpwxYVj7XbhwYTO6rHVutYwYAAAAsleQSwuvwid0yoNWLzh79qy5sAKQId/0ytwL9uT4wJgje2RUpvuwsGR/x/chq/rhD19HZsWHTs3U8+Oud/F5H/zpexTw53zk8wsiAAAAALeDIAsAAABHIsgCAADAkXx+shcAAFlp/YHTmdtAyazqCYDsxogsAAAAHIkgCwAAAEciyAIAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEei/BYAAP5WAkxEGmRJT4DAxogsAAAAHIkRWQAA/FDcgp2Zen78M9WyrC+Av2JEFgAAAI7EiCwAn2p9ZJTP3wH6EFivZaDI/Gv5aeY78U0v8bknx/v+6wiUPgQgRmQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSCG+7gAA54pbsDPT22idJT0BAOREBFkAAODV+gOnM/XKNIgpwiuLwJ9aMGnSJClTpozkzZtXGjRoIBs2bEiz/fz586VixYqmfbVq1WTJkiUe610ulwwePFiioqIkX758EhsbK/v27fNoc/r0aWnfvr0ULFhQwsPDpXPnznLhwgV7/e+//y5BQUEpbj/99FOG+gIAAIAAHZGdO3eu9OnTRyZPnmxC7Lhx46R58+ayd+9eKV68eIr2a9eulXbt2kl8fLw88cQTMnv2bGndurVs2bJFqlatatqMGjVKJkyYIDNnzpSYmBgZNGiQ2ebu3btN4FQaYo8dOybLli2T69evS6dOnaRbt25me+5++OEHqVKliv347rvvzlBfAADwBab+ICfw+Yjs2LFjpWvXriZIVq5c2QTasLAwmTZtmtf248ePlxYtWki/fv2kUqVKMnz4cKldu7ZMnDjRHo3VMDxw4EBp1aqVVK9eXWbNmiVHjx6VhQsXmjZ79uyRpUuXytSpU014bty4sXz00UcyZ84c086dBtfIyEj7Fhoamu6+AAAAIECD7LVr12Tz5s3m0L/doVy5zON169Z5fY4ud2+vdLTVan/gwAFJSEjwaFOoUCETWK02eq/TCerWrWu30fa67/Xr13ts+6mnnjIjwxp2v/766wz1BQAAAAE6teDUqVOSlJQkERERHsv18S+//OL1ORpSvbXX5dZ6a1labW6ethASEiJFihSx2xQoUEDGjBkjDzzwgAm4//znP820AR3V1XCbnr7c7OrVq+ZmOXfuXJqvDwAAAPx4jqy/Klq0qJm7a6lXr56ZdjB69Gg7yGaUzqUdOnRoFvYSAABke/UFXmK/FeLrsBgcHCzHjx/3WK6PdT6qN7o8rfbWvS7TqgXubWrWrGm3OXHihMc2bty4YSoZpLZfpdMT9OSw9PblZnFxcR7hWEdkS5Uqler+AADI6ThpDX47RzZ37txSp04dWb58ub0sOTnZPG7YsKHX5+hy9/ZKw6XVXqsUaJB0b6OBUee+Wm30PjEx0czPtaxYscLsW8NqarZt2+YRjm/Vl5vlyZPHlPtyvwEAAMChUwt0hLJjx47mxKv69eubigMXL140VQxUhw4dpESJEuawvOrVq5c0adLEzF9t2bKlqTSwadMmmTJlilmvtV579+4tI0aMkPLly9vlt6Kjo80cV6UVBrTagFZL0CoJWn6rZ8+e0rZtW9NOaekuDdq1atUyjxcsWGAqKWilA8ut+gIAAKCY3hCgQbZNmzZy8uRJcwEDPUlKD/9raSzrJKpDhw6Zk60sjRo1MvVatbzWgAEDTFjVE7Dc67b279/fhGGtC6sjr1pxQLdp1ZBVn3/+uQmvzZo1M9t/9tlnTe1Zd1pO6+DBg+ZEML3ogda8fe655zLUFwAAAGSPIJcWXoVP6JQHLQ129uxZphkg477plblX7cnx/jF37cioTG8DyEoLS/YPiM+1P3wdWXGJ2rjrXXz/dfz100z3Yf2EF3zeh0DMRz6/IAIAAADgyKkFAAAgMGV2XqhRMit6gkDFiCwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkTvYCAABIA5fJ9V+MyAIAAMCRCLIAAABwJIIsAAAAHIkgCwAAAEciyAIAAMCRqFoA5FBZcRYuAAC+RJAFAADIAYMP8c9Uk0DD1AIAAAA4EkEWAAAAjkSQBQAAgCMRZAEAAOBIBFkAAAA4EkEWAAAAjkSQBQAAgCNRRxYAACAHiMtkLVp/rEPLiCwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAcyS+C7KRJk6RMmTKSN29eadCggWzYsCHN9vPnz5eKFSua9tWqVZMlS5Z4rHe5XDJ48GCJioqSfPnySWxsrOzbt8+jzenTp6V9+/ZSsGBBCQ8Pl86dO8uFCxfs9atWrZJWrVqZbeTPn19q1qwpn3/+ucc2ZsyYIUFBQR437RMAAABywCVq586dK3369JHJkyebEDtu3Dhp3ry57N27V4oXL56i/dq1a6Vdu3YSHx8vTzzxhMyePVtat24tW7ZskapVq5o2o0aNkgkTJsjMmTMlJiZGBg0aZLa5e/duO2hqiD127JgsW7ZMrl+/Lp06dZJu3bqZ7Vn7qV69urz55psSEREhixYtkg4dOkihQoXMfi0ahLWvFg2zwJ2w/sDpzG2gZFb1BACAHDoiO3bsWOnatasJkpUrVzaBNiwsTKZNm+a1/fjx46VFixbSr18/qVSpkgwfPlxq164tEydOtEdjNQwPHDjQjKhqGJ01a5YcPXpUFi5caNrs2bNHli5dKlOnTjXhuXHjxvLRRx/JnDlzTDs1YMAAs+1GjRpJ2bJlpVevXma/CxYs8OiPBtfIyEj7pqEXAAAAAR5kr127Jps3bzaH/u0O5cplHq9bt87rc3S5e3ulo61W+wMHDkhCQoJHGx1F1cBqtdF7nU5Qt25du422132vX78+1f6ePXtWihQp4rFMpyOULl1aSpUqZYLzzz//nOrzr169KufOnfO4AQAAwIFB9tSpU5KUlJRiFFMfaxj1Rpen1d66v1Wbm6cthISEmJCa2n7nzZsnGzduNCPHlgoVKpiR46+++ko+++wzSU5ONiO4R44c8boNnQ6hodq6afgFAACAQ6cWOMHKlStNgP3b3/4mVapUsZc3bNjQzJvVE8GaNGliph0UK1ZMPvnkE6/biYuLM6O61u3w4cN38KsAAAAILD4NskWLFpXg4GA5fvy4x3J9rPNNvdHlabW37m/V5sSJEx7rb9y4YSoZ3Lzf1atXy5NPPikffvihCa1pCQ0NlVq1asn+/fu9rs+TJ485Ocz9BgAAAAcG2dy5c0udOnVk+fLl9jI9PK+PdbTTG13u3l5p5QGrvVYp0DDq3kbnourcV6uN3icmJpr5uZYVK1aYfetcWvcSXC1btpT333/fVDS4FZ0msXPnTlOyCwAAAAFefktLb3Xs2NGceFW/fn1TceDixYv2XFQdBS1RooSZX6q0eoAexh8zZowJmVppYNOmTTJlyhS7ikDv3r1lxIgRUr58ebv8VnR0tCnTpbTagVYg0GoJWiVBy2/17NlT2rZta9pZ0wm0zJbu79lnn7Xnzmr4tk74GjZsmNx///1Srlw5E4xHjx4tBw8elC5duvjktQQAAMhJfB5k27RpIydPnjQXMNCwqPNNtTSWdbLWoUOHTDUBi55MpbVetbyWlsjSsKpltawasqp///4mDOsoqgZMLa+l23S/WIFe3EDDa7Nmzcz2Naxq7VmL1qC9dOmSCdBWiFYaonWkVp05c8aEYe134cKFzeiy1p/VMmIAAADIXkEuLbwKn9ApD1q9QE/8Yr4sMmr9hBcy9aItLNnfL1701kdG+boLQJZ/b/jD55qvI+veD394Lf3hZ3b8M9X8Lh9RtQAAAACORJAFAACAIxFkAQAA4EgEWQAAADgSQRYAAACORJAFAACAIxFkAQAA4EgEWQAAADgSQRYAAACORJAFAACAIxFkAQAA4Eghvu4AkBPFLdiZ6W20zpKeAADgXIzIAgAAwJEIsgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJFCfN0BwIniFuz0dRcAAMjxGJEFAACAIxFkAQAA4EgEWQAAADgSQRYAAACO5BdBdtKkSVKmTBnJmzevNGjQQDZs2JBm+/nz50vFihVN+2rVqsmSJUs81rtcLhk8eLBERUVJvnz5JDY2Vvbt2+fR5vTp09K+fXspWLCghIeHS+fOneXChQsebXbs2CEPPvig2U+pUqVk1KhRGe4LAAAAAjTIzp07V/r06SNDhgyRLVu2SI0aNaR58+Zy4sQJr+3Xrl0r7dq1M8Fz69at0rp1a3PbtWuX3UYD54QJE2Ty5Mmyfv16yZ8/v9nmlStX7DYaYn/++WdZtmyZLFq0SH788Ufp1q2bvf7cuXPy6KOPSunSpWXz5s0yevRoeeedd2TKlCkZ6gsAAACyR5BLhy99SEdg69WrJxMnTjSPk5OTzejna6+9Jm+99VaK9m3atJGLFy+a8Gm5//77pWbNmia46pcTHR0tb7zxhvTt29esP3v2rERERMiMGTOkbdu2smfPHqlcubJs3LhR6tata9osXbpUHn/8cTly5Ih5/scffyxvv/22JCQkSO7cuU0b7c/ChQvll19+SVdfbkXDcqFChUz/dGQYd0aglM5qfSTlEYKMWFiyvwTC1wFktaz43vCHzzVfh3/9vPSHPmRW/DPV5E7ISD7y6YjstWvXzGinHvq3O5Qrl3m8bt06r8/R5e7tlY62Wu0PHDhgwqd7G30xNDBbbfRepxNYIVZpe923juBabR566CE7xFr72bt3r5w5cyZdfQEAAECAXhDh1KlTkpSUZEZL3elja9TzZhpSvbXX5dZ6a1labYoXL+6xPiQkRIoUKeLRJiYmJsU2rHWFCxe+ZV9udvXqVXOz6F8a1l8eSL93vv6Zl0tELl65lqnX4eolzznhTv06gKyWFd8b/vC55uvwr5+X/tCHzLpTecXaT3omDXBlrzsoPj5ehg4dmmK5TqUAMurDTL9k8wLk6wCy2rwA+VzzdfjTz0t/6IPTPtfnz583R9X9NsgWLVpUgoOD5fjx4x7L9XFkZKTX5+jytNpb97pMqxa4t9G5q1abm08mu3Hjhqlk4L4db/tx38et+nKzuLg4c2KbRecD6z7vvvtuCQoKkuz+60YD8+HDh5mPm0PxGQCfA/AZgBN+J+hIrIZYPWfpVnwaZHX+aZ06dWT58uXmbH8r3Onjnj17en1Ow4YNzfrevXvby7TygC5XOh1Ag6S2sYKrvlk697V79+72NhITE838XN2/WrFihdm3zqW12ujJXtevX5fQ0FB7PxUqVDDTCtLTl5vlyZPH3NzpXN07ST+s/vaBxZ3FZwB8DsBnAP7+O+FWI7E2l4/NmTPHlSdPHteMGTNcu3fvdnXr1s0VHh7uSkhIMOtfeOEF11tvvWW3X7NmjSskJMT1wQcfuPbs2eMaMmSIKzQ01LVz5067zXvvvWe28dVXX7l27NjhatWqlSsmJsZ1+fJlu02LFi1ctWrVcq1fv97173//21W+fHlXu3bt7PWJiYmuiIgIs/9du3aZfoaFhbk++eSTDPXFX5w9e1Ynmph75Ex8BsDnAHwGEGi/E3weZNVHH33kuueee1y5c+d21a9f3/XTTz/Z65o0aeLq2LGjR/t58+a57rvvPtO+SpUqrsWLF3usT05Odg0aNMgEUQ3JzZo1c+3du9ejzX//+18TXAsUKOAqWLCgq1OnTq7z5897tNm+fburcePGZhslSpQwAflmt+qLvwiUDyxuH58B8DkAnwEE2u8En9eRxZ2h1RL0ZDOdp3vz9AbkDHwGwOcAfAYQaL8TCLIAAABwJJ9fohYAAAC4HQRZAAAAOBJBFgAAAI5EkM0BJk2aJGXKlJG8efOaOrkbNmzwdZeQRd555x1zMQ33W8WKFe31V65ckR49epiLbhQoUECeffbZFBfxOHTokLRs2VLCwsLMpZv79etnLhAC//Xjjz/Kk08+aYqF63u+cOFCj/V6Du/gwYPNRWHy5csnsbGxsm/fPo82ejGW9u3bm/qRWs+6c+fOcuGC5yUwd+zYIQ8++KD52aGF00eNGnVHvj5k/jPw4osvpvjZ0KJFC482fAacLz4+XurVqyd33XWX+fmtNfn37t3r0Sarfg+sWrVKateubU4MK1eunMyYMUP8AUE2wM2dO9dcTWzIkCGyZcsWqVGjhjRv3jzFlc3gXFWqVJFjx47Zt3//+9/2utdff12++eYbmT9/vqxevVqOHj0qzzzzjL0+KSnJ/PC6du2arF27VmbOnGl+OGkIgv+6ePGi+V7WP1K90cA5YcIEmTx5srkYTP78+c33vf5Cs2iI/fnnn81FXBYtWmSCUbdu3ez1eiGZRx99VEqXLm0uHjN69Gjzh9OUKVPuyNeIzH0GlAZX958NX3zxhcd6PgPOt3r1ahNSf/rpJ/O9rBdx0u9b/Xxk5e+BAwcOmDZNmzaVbdu2mQtBdenSRb777jvxOV/X/0L20rq8PXr0sB8nJSW5oqOjXfHx8bz0AUAvwlGjRg2v6/SiHnqBjvnz59vL9MId+m2/bt0683jJkiWuXLly2RcgUR9//LGprXz16tU78BUgs/T9/PLLLz3qaEdGRrpGjx7t8VnQethffPGFeawXn9Hnbdy40W7z7bffuoKCglx//PGHefy///u/rsKFC3t8Dt58801XhQoVeNP8/DOgtP66XgwoNXwGAtOJEyfM52H16tVZ+nugf//+pla+uzZt2riaN2/u8jVGZAOY/nWlIyl6WNGSK1cu83jdunU+7Ruyjh4y1sOL9957rxlh0UNESt97/evc/f3XaQf33HOP/f7rfbVq1SQiIsJuoyN3Ohqno3VwHh05SUhI8Hjf9VKPOq3I/X3X6QR169a122h7/fmgI7hWm4ceeshcStz9s6GHLc+cOXNHvybcHj0UrIeJ9dLqeon2//73v/Y6PgOB6ezZs+a+SJEiWfp7QNu4b8Nq4w9ZgiAbwE6dOmUOGbh/OJU+1l90cD4NJ3oIaOnSpfLxxx+bEKNzGs+fP2/eYw0hGlhSe//13tvnw1oH57Het7S+7/VeA467kJAQ88uPz0Zg0GkFs2bNkuXLl8v7779vDik/9thj5neC4jMQeJKTk80h/wceeECqVq1qlmXV74HU2mjYvXz5svhSiE/3DiBT9BeTpXr16ibY6pzGefPmmZN8AORMbdu2tf+vo23686Fs2bJmlLZZs2Y+7RuyR48ePWTXrl0e50nkBIzIBrCiRYtKcHBwirMT9XFkZKTP+oXso39133fffbJ//37zHuv0ksTExFTff7339vmw1sF5rPctre97vb/5hE89Q1nPYuezEZh06pH+TtCfDYrPQGDp2bOnOWlz5cqVUrJkSXt5Vv0eSK2NVj3x9aAJQTaA6eGEOnXqmENL7oce9HHDhg192jdkDy2f9Ouvv5qyS/reh4aGerz/Or9R59Ba77/e79y50yPU6Jmv+sOpcuXKvE0OFBMTY37puL/vevhP5766v+/6i03nz1lWrFhhfj7oqL7VRisZ6Pw698+GzrcsXLjwHf2akHlHjhwxc2T1Z4PiMxAYXC6XCbFffvml+R7W7393WfV7QNu4b8Nq4xdZwtdnmyF7zZkzx5ytPGPGDHOWardu3Vzh4eEeZyfCud544w3XqlWrXAcOHHCtWbPGFRsb6ypatKg5c1W98sorrnvuuce1YsUK16ZNm1wNGzY0N8uNGzdcVatWdT366KOubdu2uZYuXeoqVqyYKy4uzodfFW7l/Pnzrq1bt5qb/hgfO3as+f/BgwfN+vfee898n3/11VeuHTt2mLPXY2JiXJcvX7a30aJFC1etWrVc69evd/373/92lS9f3tWuXTt7vZ7tHBER4XrhhRdcu3btMj9LwsLCXJ988glvkJ9/BnRd3759zVnp+rPhhx9+cNWuXdu8x1euXLG3wWfA+bp37+4qVKiQ+T1w7Ngx+3bp0iW7TVb8Hvjtt9/M93+/fv1M1YNJkya5goODTVtfI8jmAB999JH5EOfOnduU4/rpp5983SVkES1/EhUVZd7bEiVKmMf79++312twefXVV00ZJf0h9PTTT5sfcu5+//1312OPPebKly+fCcEajq9fv8575MdWrlxpwsvNNy25ZJXgGjRokAmi+odss2bNXHv37vXYxn//+18TXAsUKGDK7HTq1MkEIHfbt293NW7c2GxDP18akOH/nwENMRpKNIxo6aXSpUu7unbtmmIAg8+A84mXz4Depk+fnuW/B/QzV7NmTfP75t577/XYhy8F6T++HhUGAAAAMoo5sgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJEIsgAAAHAkgiwAAAAciSALAAAARyLIAgAAwJEIsgDgh37//XcJCgqSbdu23dH9lilTRsaNGyf+YMaMGRIeHu7rbgDwYwRZAMgmGkTTur3zzju89n4YoAE4R4ivOwAAgerYsWP2/+fOnSuDBw+WvXv32ssKFCjgo54BQGBgRBYAsklkZKR9K1SokBmFtR4XL15cxo4dKyVLlpQ8efJIzZo1ZenSpaluKykpSV566SWpWLGiHDp0yCz76quvpHbt2pI3b1659957ZejQoXLjxg37Obq/qVOnytNPPy1hYWFSvnx5+frrrzP0NSQmJkqXLl2kWLFiUrBgQXnkkUdk+/bt9nodVda+f/rpp2ZUVb/Otm3byvnz5+02+v/27dtL/vz5JSoqSj788EN5+OGHpXfv3ma9/v/gwYPy+uuv26PV7r777jupVKmSCf4tWrTw+AMBQM5GkAUAHxg/fryMGTNGPvjgA9mxY4c0b95cnnrqKdm3b1+KtlevXpXnn3/ezJf917/+Jffcc4+579Chg/Tq1Ut2794tn3zyiZlTOnLkSI/narj985//bPbx+OOPm0B5+vTpdPdT93vixAn59ttvZfPmzSY4N2vWzGMbv/76qyxcuFAWLVpkbqtXr5b33nvPXt+nTx9Zs2aNCdHLli0zfd+yZYu9fsGCBSbQDxs2zIRU96B66dIl8xppUP7xxx9NiO/bt2+GXmsAAcwFAMh206dPdxUqVMh+HB0d7Ro5cqRHm3r16rleffVV8/8DBw649Ef0v/71L1ezZs1cjRs3diUmJtptddm7777r8fxPP/3UFRUVZT/W5w8cONB+fOHCBbPs22+/TbWfpUuXdn344Yfm/7rvggULuq5cueLRpmzZsq5PPvnE/H/IkCGusLAw17lz5+z1/fr1czVo0MD8X5eHhoa65s+fb6/Xr0Of06tXL6/7dX/NtL/79++3l02aNMkVERGRav8B5CzMkQWAO+zcuXNy9OhReeCBBzyW62P3w/aqXbt2ZrRyxYoVki9fPnu5ttNRTvcRWJ1+cOXKFTOKqVMJVPXq1e31emhfpwfoCGt66D4uXLggd999t8fyy5cvm1FYi04puOuuu+zHOn3A2sdvv/0m169fl/r169vrdfpBhQoV0tUH/TrKli3rddsAQJAFAD+m0wE+++wzWbdunZmfatGAqdMGnnnmmRTP0TmzltDQUI91Ov80OTk5XfvWfWhwXLVqVYp17mWxMrOPW/G27f8bbAYAqhYAwB2no6LR0dFmRLVJkyb2cn3sPnKpunfvLlWrVjXzZxcvXmy317mqWgGhXLly2dZP3UdCQoKEhISYUdfboSehaRjduHGjmdurzp49K//5z3/koYcestvlzp3bjCgDQEYwIgsAPtCvXz8ZMmSIOWyuZ/1Pnz7dnMz1+eefp2j72muvmZD3xBNPmJOuGjdubEp56WMNh88995zkypXLTAXYtWuXjBgxIkv6GBsbKw0bNpTWrVvLqFGj5L777jNTIjRQayWEunXr3nIbOuWgY8eO5ustUqSIqdagX7f21706gQZlPZlLKx5oFYeiRYtmydcAILARZAHAB/7617+akck33njDzPmsXLmyOatfS2R5o6Wq9HC9TjXQMl1a5UArBOiZ/u+//74Z9dTSXFoqK6to0FyyZIm8/fbb0qlTJzl58qQpHaYjqREREenejpYZe+WVV0zw1tHo/v37y+HDhz2mQOjX8fLLL5tgr1UamD4AID2C9IyvdLUEACALXLx4UUqUKGHKj3Xu3JnXFMBtY0QWAJCttm7dKr/88ouZ/6uj0Dr6qlq1asUrDyBTCLIAgGynFzXQk9P0pK46deqYiyIwDxZAZjG1AAAAAI7EJWoBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAADgSARZAAAAOBJBFgAAAI5EkAUAAIAjEWQBAAAgTvT/AEa8lYxCeSbTAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 700x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "train_lengths = [len(ex[\"token_ids\"]) for ex in train_examples]\n",
    "val_lengths = [len(ex[\"token_ids\"]) for ex in val_examples]\n",
    "\n",
    "# Normalize counts because the validation split is much smaller\n",
    "bins = range(0, max(train_lengths + val_lengths) + 64, 64)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 4))\n",
    "ax.hist(train_lengths, bins=bins, density=True, alpha=0.6, label=\"Train\")\n",
    "ax.hist(val_lengths, bins=bins, density=True, alpha=0.6, label=\"Validation\")\n",
    "ax.set_xlabel(\"Token length\")\n",
    "ax.set_ylabel(\"Density\")\n",
    "ax.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"1.pdf\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1c5687d-149d-4a90-88a5-70125350e74e",
   "metadata": {},
   "source": [
    "- There are much fewer validation samples, which is why the validation histogram seems a bit jagged, but as we can see, it has a good distribution coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c136a47-269f-4406-96aa-426afbcc6756",
   "metadata": {},
   "source": [
    "&nbsp;\n",
    "## Exercise 8.2: Distilling without `<think>` tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6306a090-c0cb-40ac-ab0e-11157d69ab4c",
   "metadata": {},
   "source": [
    "- To replicate the runs without `<think></think>` tokens in the script execution command:\n",
    "\n",
    "```bash\n",
    "uv run distill.py \\\n",
    "--data_path deepseek-r1-math-train.json \\\n",
    "--validation_size 25 \\\n",
    "--epochs 3 \\\n",
    "--lr 1e-5 \\\n",
    "--max_seq_len 2048 \\\n",
    "--grad_clip 1.0\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e72f0ce6-b7a3-4f31-b37c-dc8cf6707b80",
   "metadata": {},
   "source": [
    "- Then, for the evaluation, we use the `base` instead of `reasoning` model:\n",
    "\n",
    "```bash\n",
    "uv run evaluate_math500.py \\\n",
    "--dataset_size 500 \\\n",
    "--which_model base \\\n",
    "--max_new_tokens 4096 \\\n",
    "--checkpoint_path run_11/checkpoints/distill/qwen3-0.6B-distill-step05746-epoch1.pth\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98f0d928-2e5e-4a10-ad49-eb66b8f43fae",
   "metadata": {},
   "source": [
    "- The new results (without think tokens) are shown first, with corresponding think-token results in parentheses\n",
    "\n",
    "| Teacher data                      | Epoch | MATH-500 Acc  | Final val loss       |\n",
    "|-----------------------------------|------:|--------------:|---------------------:|\n",
    "| Base (chapter 3)                  |     - | 15.2%         |              -       |\n",
    "| Reasoning (chapter 3)             |     - | 48.2%         |              -       |\n",
    "| DeepSeek R1 distillation data     |     1 | 31.8% (30.6%) | 0.5436   (0.5404)    |\n",
    "| DeepSeek R1 distillation data     |     2 | 31.8% (32.4%) | 0.5349   (0.5339)    |\n",
    "| DeepSeek R1 distillation data     |     3 | 30.2% (33.6%) | 0.5343   (0.5306)    |\n",
    "| Qwen3 235B A22B distillation data |     1 | 44.8% (45.0%) | 0.4043   (0.3130)    |\n",
    "| Qwen3 235B A22B distillation data |     2 | 39.4% (43.8%) | 0.3963   (0.3087)    |\n",
    "| Qwen3 235B A22B distillation data |     3 | 39.8% (44.2%) | 0.3948   (0.3078)    |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8c03372-cba2-408f-bdbc-41caf8b1c0e0",
   "metadata": {},
   "source": [
    "- Interestingly, the Qwen3 model has a lower validation loss when `<think></think>` tokens are omitted, but this doesn't translate into better modeling performance\n",
    "- As we can see, the omission of `<think></think>` makes the results slightly worse in almost all cases"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}