{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b1739f6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch       : 1.12.1\n",
      "lightning   : 2022.9.8\n",
      "torchmetrics: 0.9.3\n",
      "matplotlib  : 3.5.2\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,lightning,torchmetrics,matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d198b6a3",
   "metadata": {},
   "source": [
    "# Class distance weighted cross-entropy loss for ordinal regression and deep learning -- cement strength dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ffe6ee",
   "metadata": {},
   "source": [
    "Implementation of a method for ordinal regression by Polat et al 2022 [1].\n",
    "\n",
    "**Paper reference:**\n",
    "\n",
    "- [1] G Polat, I Ergenc, HT Kani, YO Alahdab, O Atug, A Temizel. \"[Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation](https://arxiv.org/abs/2202.05167).\" arXiv preprint arXiv:1612.00775 (2022)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "654568ec",
   "metadata": {},
   "source": [
    "**Note:**\n",
    "    \n",
    "To keep the notation lean and minimal, this notebook only contains \"Class Distance Weighted Cross-Entropy Loss \"-specific comments. For more comments on the PyTorch Lightning use, please see the cross-entropy baseline notebook [baseline-light_cement.ipynb](./baseline-light_cement.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13449160",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1acc2499",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "NUM_EPOCHS = 200\n",
    "LEARNING_RATE = 0.005\n",
    "NUM_WORKERS = 0\n",
    "\n",
    "DATA_BASEPATH = \".\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa5d9470-1ffa-499e-8ef4-ba4c97bb8f3c",
   "metadata": {},
   "source": [
    "## 2 - Implementing the class distance weighted cross-entropy loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6179955-6513-4be7-9384-87a8b056b858",
   "metadata": {},
   "source": [
    "According to the paper, the loss is described as follows:\n",
    "\n",
    "$$\\mathbf{C D WC E}=-\\sum_{i=0}^{N-1} \\log (1-\\hat{y}) \\times|i-c|^{\\text {power }}$$\n",
    "\n",
    "where\n",
    "\n",
    "- $N$: the number of class labels\n",
    "- $\\hat{y}$: the predicted scores\n",
    "- $c$: ground-truth class\n",
    "- power: a hyperparameter term that determines the strength of the cost coefficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "55306c8a-d3e1-4555-bd48-47f85fd384c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3792, 0.3104, 0.3104],\n",
       "        [0.3072, 0.4147, 0.2780],\n",
       "        [0.4263, 0.2248, 0.3490],\n",
       "        [0.2668, 0.2978, 0.4354]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "targets = torch.tensor([0, 2, 1, 2])\n",
    "\n",
    "logits = torch.tensor( [[-0.3,  -0.5, -0.5], # each row is 1 training example\n",
    "                        [-0.4,  -0.1, -0.5],\n",
    "                        [-0.3,  -0.94, -0.5],\n",
    "                        [-0.99, -0.88, -0.5]])\n",
    "\n",
    "probas = F.softmax(logits, dim=1)\n",
    "probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a8628712-7e04-46b9-a06e-64221e7fe313",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([12.2654, 12.2824,  0.9848, 10.2828])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cdw_ce_loss_naive1(probas, targets, power=5):\n",
    "    \n",
    "    loss = torch.zeros(probas.shape[0])\n",
    "    for example in range(probas.shape[0]):\n",
    "        for i in range(probas.shape[1]):\n",
    "            loss[example] += -torch.log(1-probas[example, i]) * torch.abs(i - targets[example])**power\n",
    "    \n",
    "    return loss\n",
    "        \n",
    "cdw_ce_loss_naive1(probas, targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4dfa0d27-972d-4239-8a88-3864858131f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "147 µs ± 2.64 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit cdw_ce_loss_naive1(probas, targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2ed813da-4787-4d8e-898a-e12e832a25f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([12.2654, 12.2824,  0.9848, 10.2828])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cdw_ce_loss_naive2(probas, targets, power=5):\n",
    "    \n",
    "    loss = 0.\n",
    "    for i in range(probas.shape[1]):\n",
    "        loss += (-torch.log(1-probas[:, i]) * torch.abs(i - targets)**power)\n",
    "        \n",
    "    return loss\n",
    "        \n",
    "cdw_ce_loss_naive2(probas, targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "893da285-61ed-4be9-9abd-f78abb37cfcc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "33.6 µs ± 82.6 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit cdw_ce_loss_naive2(probas, targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "217c1285-8535-402c-aa75-1cc91abe758c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([12.2654, 12.2824,  0.9848, 10.2828])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cdw_ce_loss_naive3(probas, targets, power=5):\n",
    "    \n",
    "    labels = torch.arange(probas.shape[1]).repeat(probas.shape[0], 1)\n",
    "    loss = (-torch.log(1-probas) * torch.abs(labels - targets.reshape(probas.shape[0], 1))**power).sum(dim=1)\n",
    "        \n",
    "    return loss\n",
    "        \n",
    "cdw_ce_loss_naive3(probas, targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "218b4093-e98d-4b2a-990d-992a9af5a9ef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.6 µs ± 61.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit cdw_ce_loss_naive3(probas, targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3da4f855-1463-4953-9a7c-e77ee6cc78df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(8.9538)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cdw_ce_loss(logits, targets, power=5, reduction=\"mean\"):\n",
    "    \n",
    "    probas = torch.softmax(logits, dim=1)\n",
    "    labels = torch.arange(probas.shape[1]).repeat(probas.shape[0], 1).type_as(logits)\n",
    "    loss = (-torch.log(1-probas) * torch.abs(labels - targets.type_as(labels).\\\n",
    "                                             reshape(probas.shape[0], 1))**power).sum(dim=1)\n",
    "        \n",
    "    if reduction == \"none\":\n",
    "        return loss\n",
    "    elif reduction == \"sum\":\n",
    "        return loss.sum()\n",
    "    elif reduction == \"mean\":\n",
    "        return loss.mean()    \n",
    "    else:\n",
    "        raise ValueError(\"reduction must be 'none', 'sum', or 'mean'\")    \n",
    "\n",
    "cdw_ce_loss(logits, targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "debfdfd1",
   "metadata": {},
   "source": [
    "## Implementing a `MultiLayerPerceptron` using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1922e731",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "class MultiLayerPerceptron(torch.nn.Module):\n",
    "    def __init__(self, input_size, hidden_units, num_classes):\n",
    "        super().__init__()\n",
    "\n",
    "        self.num_classes = num_classes\n",
    "        \n",
    "        all_layers = []\n",
    "        for hidden_unit in hidden_units:\n",
    "            layer = torch.nn.Linear(input_size, hidden_unit)\n",
    "            all_layers.append(layer)\n",
    "            all_layers.append(torch.nn.ReLU())\n",
    "            input_size = hidden_unit\n",
    "\n",
    "        output_layer = torch.nn.Linear(hidden_units[-1], num_classes)\n",
    "        \n",
    "        all_layers.append(output_layer)\n",
    "        self.model = torch.nn.Sequential(*all_layers) \n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.model(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6562578a",
   "metadata": {},
   "source": [
    "And then, we will use them below in the `LightningModule`. They are only used in the `_shared_step` method as indicated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1381f777",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightning as L\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "class LightningMLP(L.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        self.model = model\n",
    "\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        self.train_mae = torchmetrics.MeanAbsoluteError()\n",
    "        self.valid_mae = torchmetrics.MeanAbsoluteError()\n",
    "        self.test_mae = torchmetrics.MeanAbsoluteError()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "\n",
    "        loss = cdw_ce_loss(logits, true_labels)\n",
    "        predicted_labels = torch.argmax(logits, dim=1)\n",
    "        \n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        self.train_mae(predicted_labels.float(), true_labels.float())\n",
    "        self.log(\"train_mae\", self.train_mae, on_epoch=True, on_step=False)\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_mae(predicted_labels.float(), true_labels.float())\n",
    "        self.log(\"valid_mae\", self.valid_mae,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_mae(predicted_labels.float(), true_labels.float())\n",
    "        self.log(\"test_mae\", self.test_mae, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae78a8e",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Note: There Are No Changes Compared To The Baseline Below\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39ac693d",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b070e95b",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a26a6486",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>response</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>540.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1040.0</td>\n",
       "      <td>676.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>540.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1055.0</td>\n",
       "      <td>676.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>332.5</td>\n",
       "      <td>142.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>228.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>932.0</td>\n",
       "      <td>594.0</td>\n",
       "      <td>270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>332.5</td>\n",
       "      <td>142.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>228.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>932.0</td>\n",
       "      <td>594.0</td>\n",
       "      <td>365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>198.6</td>\n",
       "      <td>132.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>192.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978.4</td>\n",
       "      <td>825.5</td>\n",
       "      <td>360</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   response     V1     V2   V3     V4   V5      V6     V7   V8\n",
       "0         4  540.0    0.0  0.0  162.0  2.5  1040.0  676.0   28\n",
       "1         4  540.0    0.0  0.0  162.0  2.5  1055.0  676.0   28\n",
       "2         2  332.5  142.5  0.0  228.0  0.0   932.0  594.0  270\n",
       "3         2  332.5  142.5  0.0  228.0  0.0   932.0  594.0  365\n",
       "4         2  198.6  132.4  0.0  192.0  0.0   978.4  825.5  360"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "data_df = pd.read_csv(\"https://raw.githubusercontent.com/gagolews/\"\n",
    "                      \"ordinal_regression_data/master/cement_strength.csv\")\n",
    "data_df[\"response\"] = data_df[\"response\"]-1\n",
    "\n",
    "data_labels = data_df[\"response\"]\n",
    "data_features = data_df.loc[:, [\n",
    "    \"V1\", \"V2\", \"V3\", \"V4\", \"V5\", \"V6\", \"V7\", \"V8\"]]\n",
    "\n",
    "data_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b7044584",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 8\n",
      "Number of examples: 998\n",
      "Labels: [0 1 2 3 4]\n",
      "Label distribution: [196 310 244 152  96]\n"
     ]
    }
   ],
   "source": [
    "print('Number of features:', data_features.shape[1])\n",
    "print('Number of examples:', data_features.shape[0])\n",
    "print('Labels:', np.unique(data_labels.values))\n",
    "print('Label distribution:', np.bincount(data_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60df81f",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "952a6c2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline MAE: 1.03\n"
     ]
    }
   ],
   "source": [
    "avg_prediction = np.median(data_labels.values)  # median minimizes MAE\n",
    "baseline_mae = np.mean(np.abs(data_labels.values - avg_prediction))\n",
    "print(f'Baseline MAE: {baseline_mae:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14f82826",
   "metadata": {},
   "source": [
    "### Creating a `Dataset` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aac85ec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class MyDataset(Dataset):\n",
    "\n",
    "    def __init__(self, feature_array, label_array, dtype=np.float32):\n",
    "        self.features = feature_array.astype(dtype)\n",
    "        self.labels = label_array\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        inputs = self.features[index]\n",
    "        label = self.labels[index]\n",
    "        return inputs, label\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.features.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "998d6e7a",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "64efaa4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sebastianraschka/miniforge3/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.1\n",
      "  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "class DataModule(L.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        data_df = pd.read_csv(\n",
    "            'https://raw.githubusercontent.com/gagolews/'\n",
    "            'ordinal_regression_data/master/cement_strength.csv')\n",
    "        data_df.to_csv(\n",
    "            os.path.join(self.data_path, 'cement_strength.csv'), index=None)\n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        data_df = pd.read_csv(\n",
    "            os.path.join(self.data_path, 'cement_strength.csv'))\n",
    "        data_df[\"response\"] = data_df[\"response\"]-1  # labels should start at 0\n",
    "        self.data_labels = data_df[\"response\"]\n",
    "        self.data_features = data_df.loc[:, [\n",
    "            \"V1\", \"V2\", \"V3\", \"V4\", \"V5\", \"V6\", \"V7\", \"V8\"]]\n",
    "        \n",
    "        # Split into\n",
    "        # 70% train, 10% validation, 20% testing\n",
    "        \n",
    "        X_temp, X_test, y_temp, y_test = train_test_split(\n",
    "            self.data_features.values,\n",
    "            self.data_labels.values,\n",
    "            test_size=0.2,\n",
    "            random_state=1,\n",
    "            stratify=self.data_labels.values)\n",
    "\n",
    "        X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "            X_temp,\n",
    "            y_temp,\n",
    "            test_size=0.1,\n",
    "            random_state=1,\n",
    "            stratify=y_temp)\n",
    "        \n",
    "        # Standardize features\n",
    "        sc = StandardScaler()\n",
    "        X_train_std = sc.fit_transform(X_train)\n",
    "        X_valid_std = sc.transform(X_valid)\n",
    "        X_test_std = sc.transform(X_test)\n",
    "\n",
    "        self.train = MyDataset(X_train_std, y_train)\n",
    "        self.valid = MyDataset(X_valid_std, y_valid)\n",
    "        self.test = MyDataset(X_test_std, y_test)\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.train, batch_size=BATCH_SIZE,\n",
    "                          num_workers=NUM_WORKERS,\n",
    "                          drop_last=True)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.valid, batch_size=BATCH_SIZE,\n",
    "                          num_workers=NUM_WORKERS)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return DataLoader(self.test, batch_size=BATCH_SIZE,\n",
    "                          num_workers=NUM_WORKERS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d909e6cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path=DATA_BASEPATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a448bc5",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8ee17411",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = MultiLayerPerceptron(\n",
    "    input_size=data_features.shape[1],\n",
    "    hidden_units=(40, 20),\n",
    "    num_classes=np.bincount(data_labels).shape[0])\n",
    "\n",
    "lightning_model = LightningMLP(\n",
    "    model=pytorch_model,\n",
    "    learning_rate=LEARNING_RATE)\n",
    "\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode=\"min\", monitor=\"valid_mae\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"mlp-beckham-cement\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9e21b802",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (mps), used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "/Users/sebastianraschka/miniforge3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1791: UserWarning: MPS available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='mps', devices=1)`.\n",
      "  rank_zero_warn(\n",
      "\n",
      "  | Name      | Type                 | Params\n",
      "---------------------------------------------------\n",
      "0 | model     | MultiLayerPerceptron | 1.3 K \n",
      "1 | train_mae | MeanAbsoluteError    | 0     \n",
      "2 | valid_mae | MeanAbsoluteError    | 0     \n",
      "3 | test_mae  | MeanAbsoluteError    | 0     \n",
      "---------------------------------------------------\n",
      "1.3 K     Trainable params\n",
      "0         Non-trainable params\n",
      "1.3 K     Total params\n",
      "0.005     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sebastianraschka/miniforge3/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:219: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 10 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "/Users/sebastianraschka/miniforge3/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:219: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 10 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "/Users/sebastianraschka/miniforge3/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1894: PossibleUserWarning: The number of training batches (5) is smaller than the logging interval Trainer(log_every_n_steps=10). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "025ef56594a54bc7a45de8b2cd99b869",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=200` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 0.13 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = L.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    accelerator=\"cpu\",  # Uses Apple Silicon, GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    deterministic=True,\n",
    "    log_every_n_steps=10)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73a8e075",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d23d4a7d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='MAE'>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAibUlEQVR4nO3de3RV9Z338fc3FxIghGuNAVToACoKgsFLS70gteKlYhUVV23RRxdPbZ3qOLjEdvrYcdr12JnWtk4Vn3aqUkdFpaI4tbaWJqW2iiWKlKuIxRJAboIQIJCE7/PH3jme3HOSnLMP7M9rrb323r+zzz4fdsL55rev5u6IiIgA5EQdQEREsoeKgoiIJKgoiIhIgoqCiIgkqCiIiEhCXtQBumLQoEE+bNiwTr9/37599O7du/sCdRPlSo1ypS5bsylXajqbq7Kycoe7f6LFF939iB3Kysq8K8rLy7v0/nRRrtQoV+qyNZtypaazuYCl3sr3qnYfiYhIgoqCiIgkqCiIiEjCEX2gWUSOPrW1tVRVVVFTUxN1lIS+ffuyevXqqGM0016uwsJChg4dSn5+fofXqaIgIlmlqqqKPn36MGzYMMws6jgA7N27lz59+kQdo5m2crk7O3fupKqqiuHDh3d4nWnbfWRmj5jZNjNbkdQ2wMxeMbN14bh/0mt3m9m7ZrbWzC5KVy4RyW41NTUMHDgwawrCkcrMGDhwYMo9rnQeU3gMmNKkbTawyN1HAovCecxsNDAdOCV8z0NmlpvGbCKSxVQQukdntmPaioK7LwY+bNI8FZgbTs8Frkhqn+fuB939b8C7wJnpysZHm+D336Xn/k1p+wgRkSNRpo8plLj7FgB332Jmx4TtQ4DXk5arCtuaMbOZwEyAkpISKioqUg5RtPddJlT+Ozkj7ujU+9OturpauVKgXKnL1mzV1dX07duXvXv3Rh2lkfr6+qzLBB3LVVNTk9rPurWr2rpjAIYBK5Lmdzd5fVc4fhC4Pqn958BV7a2/01c0b13lfk+xr3j63zr3/jQ72q6eTDflSl22ZisvL/dVq1ZFmmHXrl3+4IMPNmrbs2dPu++7+OKLfdeuXSl/3owZM/zZZ59N+X3uHcvV0vYki65o3mpmpQDheFvYXgUcl7TcUGBz2lLkFQKQc/hQ2j5CRI5Mu3fv5qGHHmrWXl9f3+b7XnrpJfr165emVJmT6d1HC4EZwH3h+IWk9ifN7H5gMDASeCNtKVQURI4I//riSlZt3tOt6xw9uJh7Pn9Kq6/Pnj2b9evXM27cOPLz8ykqKmLQoEGsXLmSVatWccUVV7Bx40Zqamq47bbbmDlzJgDDhg1j6dKlVFdXc/HFF/OZz3yGP//5zwwZMoQXXniBnj17tptt0aJFzJo1i7q6Os444wzmzJlDQUEBs2fPZuHCheTl5fG5z32O73//+zz77LPcc8895Ofn07dvXxYvXtwt2ydtRcHMngLOBwaZWRVwD0ExeMbMbgL+DlwN4O4rzewZYBVQB3zN3dsuy12RHxSF3HoVBRFp7L777mPFihUsW7aMiooKLr30Ul5//XXGjBkDwCOPPMKAAQM4cOAAZ5xxBldddRUDBw5stI5169bx1FNP8bOf/YxrrrmGX/7yl1x//fVtfm5NTQ033HADixYtYtSoUXz5y19mzpw5fPnLX2bBggWsWbMGM2P37t0A3HvvvSxYsIATTzwx0dYd0lYU3P26Vl6a3Mry3wW+m648jainIHJEaOsv+kw588wzSb5F/wMPPMCCBQsA2LhxI+vWrWtWFIYPH864ceMAKCsrY8OGDe1+ztq1axk+fDijRo0CYMaMGTz44IPceuutFBYWcvPNN3PppZdy2WWXATBx4kRuueUWrrvuOq688squ/0ND8bz3kYqCiHRQ8vMKKioq+N3vfsdrr73G22+/zfjx41u8OKygoCAxnZubS11dXbufExz/bS4vL4833niDq666iueff54pU4LLvx5++GH+5V/+hY0bNzJu3Dh27tyZ6j+t5c/rlrUcacwgt4Ccw7VRJxGRLNOnT59WT/P86KOP6N+/P7169WLNmjW8/vrrLS7XGSeddBIbNmzg3XffZcSIETz++OOcd955VFdXs3//fi655BLOPvtsRowYAcD69es544wzuOCCC3jxxRfZuHFjsx5LZ8SzKADkF5Jz+GDUKUQkywwcOJCJEydy6qmn0rNnT0pKShKvTZkyhYcffpixY8dy4okncvbZZ3fb5xYWFvLoo49y9dVXJw40f+UrX+HDDz9k6tSp1NTU4O788Ic/BODOO+9k7dq1mBmTJ0/mtNNO65Yc8S0KeYXqKYhIi5588slG8w09h4KCAn7961+3+J6G4waDBg1ixYrELd+YNWtWm5/12GOPJaYnT57MW2+91ej10tJS3nij+cmYzz33XFpu1BfPYwoQFgUdUxARSRbznoKKgohkxte+9jX+9Kc/NWq77bbbuPHGGyNK1LL4FoX8QnIPqiiISGY8+OCDUUfoEO0+EhGRBBUFERFJiHlR0NlHIiLJ4lsUdJ2CiEgz8S0KeT21+0hEuqyoqAiAzZs3M23atBaXOf/881m6dGmr6xg2bBg7duxIS75Uxbgo6DYXItJ9Bg8ezPz586OO0WUxPiW1p26dLZLtfj0bPvhr967z2DFw8X2tvnzXXXdxwgkn8NWvfhWAb3/72xw6dIglS5awa9cuamtr+c53vsPUqVMbvW/Dhg1cdtllrFixggMHDnDjjTeyatUqTj75ZA4cONDhePfffz+PPPIIADfffDO33347+/bt45prrqGqqor6+nq+9a1vce2113LPPffw8ssvN3rOQlfFtyjkFWj3kYg0M336dG6//fZEUXjmmWeYP38+s2fPpri4mB07dnD22Wdz+eWXY2YtrmPOnDn06tWL5cuXs3z5ck4//fQOfXZlZSWPPvooS5Yswd0566yzOO+883jvvfcYPHgwv/rVr4DgxnwffvghL774Iu+8806j5yx0VYyLQk9yvBYOH4ac+O5FE8lqbfxFny7jx49n27ZtbN68me3bt9O/f3+OPfZYvvGNb7B48WJycnLYtGkTW7du5dhjj21xHYsXL+brX/86AGPHjmXs2LEd+uxXX32VL3zhC4nbdV955ZX88Y9/ZMqUKcyaNYu77rqLyy67jHPOOYe6uroWn7PQVfH9NswL73derzOQRKSxadOmMX/+fJ5++mmmT5/OM888w/bt26msrGTZsmWUlJS0+ByFZK31ItrS2jMVRo0aRWVlJWPGjOHuu+/m3nvvJS8vj/Ly8mbPWeiq+BaF/PB5qbUd39cnIvEwffp05s2bx/z585k2bRofffQRxxxzDPn5+ZSXl/P++++3+f5zzz2XJ554AoAVK1awfPnyDn3uueeey/PPP8/+/fvZt28fCxYs4JxzzmHz5s306tWL66+/nlmzZvHmm29SXV3Nnj17uOSSS/jRj37EsmXLuvrPBmK9+yjsKdSppyAijZ1yyins3buXIUOGUFpayrXXXst1113HhAkTGDduHCeddFKb77/lllu48cYbGTt2LOPGjePMM8/s0Oeefvrp3HDDDYnlb775ZsaPH89vfvMb7rzzTnJycsjPz2fOnDns3buXq6++mtra2kbPWeiqGBeFsKdQp56CiDT3179+fNbTwIEDee2111pcrrq6GgiuNWh4jkLPnj2ZN29ehz8r+RnOd9xxB3fccUej1y+66CIuuuiiZu+rqKjQ8xS6jXoKIiLNxLenoGMKIpJhZ511FgcPNv5D9PHHH2fMmDERJWouvkVBPQWRrOXunTp7J9stWbIko5/X2tlMbYnx7iMdUxDJRoWFhezcubNTX2jyMXdn586dFBYWpvQ+9RTUUxDJKkOHDqWqqort27dHHSWhpqYm5S/XTGgvV2FhIUOHDk1pnfEtCjqmIJKV8vPzGT58eNQxGqmoqGD8+PFRx2gmHblivPtIPQURkaZiXBR0TEFEpKkYFwX1FEREmopvUdAxBRGRZuJbFPLCI/Z1bd/pUEQkTiIpCmb2T2a20sxWmNlTZlZoZgPM7BUzWxeO+6c5BIctX0VBRCRJxouCmQ0Bvg5McPdTgVxgOjAbWOTuI4FF4Xxa1ef2gFoVBRGRBlHtPsoDeppZHtAL2AxMBeaGr88Frkh3iMM5PdRTEBFJYlFcSm5mtwHfBQ4Av3X3L5rZbnfvl7TMLndvtgvJzGYCMwFKSkrKUrk9bVNn/Plm9vY/hTUn/1On15EO1dXVFBUVRR2jGeVKTbbmguzNplyp6WyuSZMmVbr7hBZfdPeMDkB/4PfAJ4B84HngemB3k+V2tbeusrIy74rq7412f/pLXVpHOpSXl0cdoUXKlZpszeWevdmUKzWdzQUs9Va+V6PYffRZ4G/uvt3da4HngE8DW82sFCAcb0t3kMM5BTqmICKSJIqi8HfgbDPrZcG9cScDq4GFwIxwmRnAC+kOcjhHZx+JiCTL+A3x3H2Jmc0H3gTqgLeAnwJFwDNmdhNB4bg63Vl0oFlEpLFI7pLq7vcA9zRpPkjQa8gYFQURkcbie0Uzuk5BRKSpWBcF9RRERBpTUVBREBFJiHlR0NlHIiLJYl4UdJ2CiEiymBeFfKg/CBHc6kNEJBvFvCj0CCa0C0lEBIh5UajPbXgkp4qCiAjEvCgczskPJnRcQUQEiH1RaNh9pOc0i4iAikIwUXcw2iAiIllCRQGgVj0FERFQUQgm1FMQEQFUFIIJHVMQEQFiXxTCs4/UUxARAWJeFBLXKeiYgogIEPOioJ6CiEhjMS8KOqYgIpJMRQHUUxARCakogI4piIiEYl4UdExBRCRZrIsClgO5BTqmICISindRAMgrVE9BRCSkopBfqGMKIiIhFYW8AvUURERCKgp5PXVMQUQkpKKQX6gnr4mIhFQU8gr1jGYRkZCKgoqCiEiCioKKgohIgoqCjimIiCSoKKinICKSEElRMLN+ZjbfzNaY2Woz+5SZDTCzV8xsXTjun5EwKgoiIglR9RR+DLzs7icBpwGrgdnAIncfCSwK59NPRUFEJCHjRcHMioFzgZ8DuPshd98NTAXmhovNBa7ISCAdUxARSTB3z+wHmo0DfgqsIuglVAK3AZvcvV/ScrvcvdkuJDObCcwEKCkpKZs3b16ns1RXV3Pq9oUMe/9pKs57Hsw6va7uVF1dTVFRUdQxmlGu1GRrLsjebMqVms7mmjRpUqW7T2jxRXfP6ABMAOqAs8L5HwP/Buxustyu9tZVVlbmXVFeXu6++Afu9xS7H9rfpXV1p/Ly8qgjtEi5UpOtudyzN5typaazuYCl3sr3ahTHFKqAKndfEs7PB04HtppZKUA43paRNHmFwVjHFUREMl8U3P0DYKOZnRg2TSbYlbQQmBG2zQBeyEig/LAo6LiCiAh5EX3uPwJPmFkP4D3gRoIC9YyZ3QT8Hbg6I0nUUxARSYikKLj7MoJjC01NznAUFQURkSS6ollFQUQkQUVBxxRERBJUFNRTEBFJUFFQURARSehQUTCz3maWE06PMrPLzSw/vdEyJL9nMK7Vc5pFRDraU1gMFJrZEIKb1d0IPJauUBnV0FNQURAR6XBRMHffD1wJ/Ke7fwEYnb5YGdSjdzCu3R9tDhGRLNDhomBmnwK+CPwqbIvqwrfu1VAUDu2LNoeISBboaFG4HbgbWODuK83sk0B52lJlUl7DMQX1FEREOvTXvrv/AfgDQHjAeYe7fz2dwTImJwfye6mnICJCx88+etLMis2sN8HN69aa2Z3pjZZBKgoiIkDHdx+Ndvc9BE9Dewk4HvhSukJlXI/e2n0kIkLHi0J+eF3CFcAL7l4LZPaRbenUo7d6CiIidLwo/D9gA9AbWGxmJwB70hUq4/J7qacgIkLHDzQ/ADyQ1PS+mU1KT6QI9NAxBRER6PiB5r5mdr+ZLQ2HHxD0Go4OPYrgkHoKIiId3X30CLAXuCYc9gCPpitUxuX3glr1FEREOnpV8j+4+1VJ8/9qZsvSkCcaPXqppyAiQsd7CgfM7DMNM2Y2ETh67iCXr1NSRUSg4z2FrwC/MLO+4fwuYEZ6IkWgRy84VA3uYBZ1GhGRyHT07KO3gdPMrDic32NmtwPL05gtc3r0Bj8MdQc/fjyniEgMpfTkNXffE17ZDHBHGvJEI1+3zxYRga49jvPo2c/So1cw1rUKIhJzXSkKR89tLvLDoqCegojEXJvHFMxsLy1/+RvQMy2JotCjKBgfqo42h4hIxNosCu7eJ1NBIpXYfaSegojEW1d2Hx09dKBZRARQUQjoQLOICKCiEOgR9hRUFEQk5lQUQLuPRERCKgqg3UciIqHIioKZ5ZrZW2b2P+H8ADN7xczWheP+GQuTVwiYegoiEntR9hRuA1Ynzc8GFrn7SGBROJ8ZZnrQjogIERUFMxsKXAr8V1LzVGBuOD0XuCKjoRrulCoiEmPmnvm7VZjZfOD/An2AWe5+mZntdvd+Scvscvdmu5DMbCYwE6CkpKRs3rx5nc5RXV1NUVFwNfNZr/9v9hSPYvXof+70+rpLcq5solypydZckL3ZlCs1nc01adKkSnef0OKL7p7RAbgMeCicPh/4n3B6d5PldrW3rrKyMu+K8vLyj2ce+rT7k9d1aX3dpVGuLKJcqcnWXO7Zm025UtPZXMBSb+V7taMP2elOE4HLzewSoBAoNrP/BraaWam7bzGzUmBbRlP16K3nNItI7GX8mIK73+3uQ919GDAd+L27Xw8s5OOnuc0AXshosPxeOiVVRGIvm65TuA+40MzWAReG85nTo7fOPhKR2Iti91GCu1cAFeH0TmByZGHye2n3kYjEXjb1FKJV0AcO7o06hYhIpFQUGhT2hZqPIIJTdEVEsoWKQoPCvnC4Tre6EJFYU1FoUNg3GNd8FG0OEZEIqSg0SBSFPdHmEBGJkIpCA/UURERUFBIK+wVjFQURiTEVhQbqKYiIqCgkFBYH45rdkcYQEYmSikKDgoaioJ6CiMSXikKD/MLgsZwqCiISYyoKyRquahYRiSkVhWQqCiIScyoKyVQURCTmVBSSFfaFg7qiWUTiS0UhWUGxegoiEmsqCsm0+0hEYk5FIZmeqSAiMaeikKywL9QfgrqaqJOIiERCRSGZ7n8kIjGnopBMRUFEYk5FIZluny0iMaeikEw9BRGJORWFZIW6U6qIxJuKQrKG3UcHdkUaQ0QkKioKyXoPAsuFvR9EnUREJBIqCslycqHPsbB3S9RJREQioaLQVJ9S2LMp6hQiIpFQUWiqeDDs2Rx1ChGRSKgoNFU8GPZo95GIxJOKQlPFg+HQXqjRcxVEJH4yXhTM7DgzKzez1Wa20sxuC9sHmNkrZrYuHPfPdDYA+gwOxjrYLCIxFEVPoQ74Z3c/GTgb+JqZjQZmA4vcfSSwKJzPvOKwKOhgs4jEUMaLgrtvcfc3w+m9wGpgCDAVmBsuNhe4ItPZACguDcY6riAiMRTpMQUzGwaMB5YAJe6+BYLCARwTSaiG3Uc6A0lEYsg8oqeMmVkR8Afgu+7+nJntdvd+Sa/vcvdmxxXMbCYwE6CkpKRs3rx5nc5QXV1NUVFRs/aJr17PtmMmsm7ULZ1ed1e0litqypWabM0F2ZtNuVLT2VyTJk2qdPcJLb7o7hkfgHzgN8AdSW1rgdJwuhRY2956ysrKvCvKy8tbfuGhie5PXNOldXdFq7kiplypydZc7tmbTblS09lcwFJv5Xs1irOPDPg5sNrd7096aSEwI5yeAbyQ6WwJxaXafSQisRTFMYWJwJeAC8xsWThcAtwHXGhm64ALw/loFA+GjzZCRLvWRESikpfpD3T3VwFr5eXJmczSqqFnQuVjsPlNGFIWdRoRkYzRFc0tOfFiyMmDVQujTiIiklEqCi3pNQCGnQOrF2oXkojEiopCa0ZfDh++B1tXRp1ERCRjVBRac9LnwXLgr89EnUREJGNUFFpT9Ak4+fNQORcO7Ys6jYhIRqgotOXsr0LNbnj7qaiTiIhkhIpCW447CwafDq8/rAPOIhILKgptMYMzboad66DqL1GnERFJOxWF9pz8ecgrhOU64CwiRz8VhfYUFgcXs618Dupro04jIpJWKgodMeYa2L8T1pdHnUREJK1UFDpixGehZ39dsyAiRz0VhY7I6wGjr4A1v4KD1VGnERFJGxWFjhp7LdTuh7UvRZ1ERCRtVBQ66rizoO/xOgtJRI5qKgodlZMDY6bB+t9D9fao04iIpIWKQirGXgNeH5yeKiJyFFJRSMUxJ0PJGO1CEpGjlopCqsZeDZuWws71UScREel2KgqpOnUaYMEznEVEjjIqCqnqOyQ4PfW1n8B7f4g6jYhIt1JR6IxLfwADR8KzN8Drc+DArqgTiYh0CxWFzigogulPwqCR8PJs+I+R8N/T4M1fwL6dUacTEem0vKgDHLEGjYCbfgubl8GKX8LqhbDwH8Fuh+HnwMmXw8jPQd+hwXMZRESOACoKXTV4XDBceC98sBxWvRAMv7ojeL2oJHh6W+nYYJfToBEwcAQU9IkytYhIi1QUuosZlJ4WDBd8C7athg2vwqbKYHjnZSDpkZ59SoPiMHBEsBuq3/FQPCToWfjhyP4ZIhJvKgrpYAYlo4OhQW0N7Pob7FgXPN5z5/pgetXzzQ5Un2t5sHwoFA8NznbqO/TjglE8JGgr7KfdUiLS7VQUMiW/MLgi+piTm7+2/0PY/XfYswk+2kTVitc4vm9OMP/+a7B3Mxyua7K+3kFxaCgSjQpION2jd2b+bSJy1FBRyAa9BgTD4HEAvHdgFMeff/7Hrx+uh+ptYdHYCB9tCqergvG61VC9lUa7pwB69IGe/YJeRc9+jaebjfuHy/QPikluD/VERGJIReFIkJMLxaXBMHRCy8vUHYK9WxK9DfZUBYXkwO5g91TNbtjxbjA+sBvqDrTzoQZ5hZBXAPk9Ia+AMw4ehrUDgvb8wmCck9fCkBsOyfNNX2/yHjOwnOBzLSccWmuzRm2Dtq+ENfvaXa79z7AUlrOPpxu2FzSaL6jZHhTuRq81X67ldTRdjg4u17H12eHapGeOt7EO/WEQOyoKR4u8HtD/hGDoiLqDQXFoKBINhePAbqjdFxwDqUsaamvYv2UjvYv6BPOH9ge7vQ7XB7u2EkOTeT/cfL6bnQqwsttX22WfAng96hQtOw9gcarv6mgBarJ8Cu3n1NfDn3I7sZ62lu1oW1J7k7ZPHToIlYXtLpdSxo5suzb+6GDkhVBwId1NRSGu8gqgT0kwdNDKigrOT96t1RmHDwe3H08uFPUNxcKDsTeMm7Z5i8st/csbTJhQ1u5yLbd5B5cLi1lyW8PuOm/YbZc876xZu5aTRo36+LVWlmu8jtbX1+y1Liz33t/e45PDhyftceymz2r2b0itffPGKo47bmjn1tOVtkbtzdt2btnM4NLSNpZLNWNHtl077ykeAgfpdllXFMxsCvBjIBf4L3e/L+JI0p1ycoAcyM3vtlVW99kRnAqcZT7YU8FJZedHHaNFfz9cwSfPPT/qGM2sr6jguK7+4ZEG71RUMDgLc1FR0e2rzKrbXJhZLvAgcDEwGrjOzEa3/S4REekuWVUUgDOBd939PXc/BMwDpkacSUQkNswb7feKlplNA6a4+83h/JeAs9z91qRlZgIzAUpKSsrmzZvX6c+rrq6mqKioa6HTQLlSo1ypy9ZsypWazuaaNGlSpbu3fCqju2fNAFxNcByhYf5LwH+2tnxZWZl3RXl5eZfeny7KlRrlSl22ZlOu1HQ2F7DUW/lezbbdR1XAcUnzQ4HNEWUREYmdbCsKfwFGmtlwM+sBTAcWRpxJRCQ2suqUVHevM7Nbgd8QnJL6iLtn4WVJIiJHp6wqCgDu/hLwUtQ5RETiKKvOPkqVmW0H3u/CKgYBO7opTndSrtQoV+qyNZtypaazuU5w90+09MIRXRS6ysyWemunZUVIuVKjXKnL1mzKlZp05Mq2A80iIhIhFQUREUmIe1H4adQBWqFcqVGu1GVrNuVKTbfnivUxBRERaSzuPQUREUmioiAiIgmxLApmNsXM1prZu2Y2O8Icx5lZuZmtNrOVZnZb2P5tM9tkZsvC4ZIIsm0ws7+Gn780bBtgZq+Y2bpw3D+CXCcmbZdlZrbHzG6PYpuZ2SNmts3MViS1tbqNzOzu8HdurZldlOFc/2Fma8xsuZktMLN+YfswMzuQtN0eTleuNrK1+rOLeJs9nZRpg5ktC9szts3a+I5I3+9Za3fKO1oHgttnrAc+CfQA3gZGR5SlFDg9nO4DvEPwcKFvA7Mi3k4bgEFN2v4dmB1Ozwa+lwU/yw+AE6LYZsC5wOnAiva2UfhzfRsoAIaHv4O5Gcz1OSAvnP5eUq5hyctFtM1a/NlFvc2avP4D4P9kepu18R2Rtt+zOPYUsuZBPu6+xd3fDKf3AquBIVFk6aCpwNxwei5wRXRRAJgMrHf3rlzV3mnuvhj4sElza9toKjDP3Q+6+9+Adwl+FzOSy91/6+514ezrBHcgzrhWtllrIt1mDczMgGuAp9Lx2W1p4zsibb9ncSwKQ4CNSfNVZMEXsZkNA8YDS8KmW8Ou/iNR7KYheEr4b82sMnywEUCJu2+B4JcVOCaCXMmm0/g/atTbDFrfRtn0e/e/gF8nzQ83s7fM7A9mdk5EmVr62WXLNjsH2Oru65LaMr7NmnxHpO33LI5FwVpoi/S8XDMrAn4J3O7ue4A5wD8A44AtBF3XTJvo7qcTPC/7a2Z2bgQZWmXBrdUvB54Nm7Jhm7UlK37vzOybQB3wRNi0BTje3ccDdwBPmllxhmO19rPLim0GXEfjPz4yvs1a+I5oddEW2lLaZnEsCln1IB8zyyf4YT/h7s8BuPtWd69398PAz0hTl7kt7r45HG8DFoQZtppZaZi7FNiW6VxJLgbedPetkB3bLNTaNor8987MZgCXAV/0cAd0uJthZzhdSbAPelQmc7Xxs8uGbZYHXAk83dCW6W3W0ncEafw9i2NRyJoH+YT7Kn8OrHb3+5PaS5MW+wKwoul705yrt5n1aZgmOEi5gmA7zQgXmwG8kMlcTTT66y3qbZaktW20EJhuZgVmNhwYCbyRqVBmNgW4C7jc3fcntX/CzHLD6U+Gud7LVK7wc1v72UW6zUKfBda4e1VDQya3WWvfEaTz9ywTR9CzbQAuITiKvx74ZoQ5PkPQtVsOLAuHS4DHgb+G7QuB0gzn+iTBGQxvAysbthEwEFgErAvHAyLabr2AnUDfpLaMbzOCorQFqCX4C+2mtrYR8M3wd24tcHGGc71LsK+54ffs4XDZq8Kf8dvAm8DnI9hmrf7sotxmYftjwFeaLJuxbdbGd0Tafs90mwsREUmI4+4jERFphYqCiIgkqCiIiEiCioKIiCSoKIiISIKKgkg7zKzeGt+ZtdvurBvecTOqaypEmsmLOoDIEeCAu4+LOoRIJqinINJJ4T32v2dmb4TDiLD9BDNbFN7gbZGZHR+2l1jwLIO3w+HT4apyzexn4f3yf2tmPSP7R0nsqSiItK9nk91H1ya9tsfdzwR+AvwobPsJ8At3H0tw47kHwvYHgD+4+2kE9+5fGbaPBB5091OA3QRXzIpEQlc0i7TDzKrdvaiF9g3ABe7+XnjTsg/cfaCZ7SC4VUNt2L7F3QeZ2XZgqLsfTFrHMOAVdx8Zzt8F5Lv7dzLwTxNpRj0Fka7xVqZbW6YlB5Om69GxPomQioJI11ybNH4tnP4zwd13Ab4IvBpOLwJuATCz3AieWyDSLv1FItK+nhY+tD30srs3nJZaYGZLCP7Aui5s+zrwiJndCWwHbgzbbwN+amY3EfQIbiG4M6dI1tAxBZFOCo8pTHD3HVFnEeku2n0kIiIJ6imIiEiCegoiIpKgoiAiIgkqCiIikqCiICIiCSoKIiKS8P8BGunuOkUhF8AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzN0lEQVR4nO3deXxU9b3/8ddnJpNMkslGAkkgQNiVsBoE9xsUFaxKrVjB6m2xXurWWm9ttdfW5ZZqrT+1tW7VllKtAq2KKyrCJVIVVIIsYZMdwpYFsu/J9/fHmWxkkjAhs8B8no/HPDJzzplz3pwM88n3e875HjHGoJRSKnTZAh1AKaVUYGkhUEqpEKeFQCmlQpwWAqWUCnFaCJRSKsSFBTqAt5KSkkx6enq33ltRUUF0dHTPBuohwZpNc3knWHNB8GbTXN7pbq6cnJxCY0xvjzONMafUIzMz03TXihUruv1eXwvWbJrLO8Gay5jgzaa5vNPdXMAa08H3qnYNKaVUiNNCoJRSIc5nhUBE5olIvojkdrHc2SLSICIzfJVFKaVUx3x5sHg+8AzwckcLiIgdeAz4yIc5lFJBqK6ujry8POLi4tiyZUug47RzquZyOp2kpaXhcDhOeJ0+KwTGmJUikt7FYj8G3gDO9lUOpVRwysvLIyYmhsTERGJjYwMdp52ysjJiYmICHaOdznIZYygqKiIvL49Bgwad8DrF+HDQOXcheM8YM8rDvH7Aa8DFwF/dy73ewXrmAHMAkpOTMxcuXNitPOXl5bhcrm6919eCNZvm8k6w5oLgyxYXF8eQIUNobGzEbrcHOk47DQ0Np2QuYww7d+6kpKSkzfTJkyfnGGMmdPgmXz2AdCC3g3n/As5xP58PzDiRderpo/6lubwTrLmMCb5smzdvNsYYU1paGuAknp3KuZr2bWt0cvpoIC8omwAsFBGAJOAKEak3xrzli41tO1zGG9trGT2hhkRXhC82oZRSp6SAnT5qjBlkjEk3xqQDrwO3+6oIAOwsKOfdnXUUltf6ahNKKXVK8uXpowuAVcAIEckTkR+KyK0icquvttmZMJsAUNfQGIjNK6WCTHFxMc8995zX77viiisoLi7u+UAB5MuzhmZ5sewPfJWjiSPMqnm1WgiUUrQUgttvv73N9K4Oxi5ZssTX0fzulBt0rrvC7VYhqG/QW3MqFWwefncTmw+W9ug6R/aN5cGrMjqcf99997Fz507GjRuHw+HA5XKRmprKunXr2Lx5M7NmzeLQoUNUV1dz1113MWfOHADS09NZs2YN5eXlTJs2jQsuuIDPP/+cfv368fbbbxMZGelxe1lZWYwfP56cnBwKCgp4+eWXefTRR9m4cSPXX389c+fOBeDb3/42+/fvb7fdpUuX8uCDD1JVVcWwYcP429/+1mNngYXMEBPaNaSUau13v/sdQ4YMYd26dTz++ON8+eWX/Pa3v2Xz5s0APPvss+Tk5LBmzRqefvppioqK2q1j+/bt3HHHHWzatIn4+HjeeOONTrcZHh7OypUrufXWW5k+fTrPPvssubm5zJ8/v3n98+bNa7fdwsJC5s6dy7Jly/j3v//NhAkTePLJJ3tsX4RMi6Cpa0gLgVLBp7O/3P1l4sSJbS7CeuGFF5q7gfbv38/27dtJTExs855BgwYxbtw4ADIzM9mzZ0+n27j66qsBGD16NBkZGaSmpgIwePBg9u/fT2JiIk8//TSLFy9us93CwkI2b97M+eefT2NjI/X19Zx77rk98c8GQqkQ2JoKgXYNKaXaaz3Gf3Z2NtnZ2axatYqoqCiysrKorq5u956IiJZT0e12O1VVVZ1uo2l5m83W5r02m436+nqys7NZtmxZu+0aY7j00ktZsGCBT654DpmuIUeY1TVUry0CpRQQExNDWVmZx3klJSXEx8cTFRXF1q1bWb16tV8ylZSUkJCQ0G6755xzDp999hk7duwAoLKykm+++abHthsyLYIwm541pJRqkZiYyPnnn8+oUaOIjIwkOTm5ed7UqVN55plnGDNmDCNGjOCcc87xS6apU6fywgsvtNtu7969mT9/PrNmzaKqqgqbzcbcuXMZPnx4j2w3ZApB01lD2jWklGry2muveZweERHBm2++6bELpuk4QFJSErm5LaPs33PPPZ1uKzs7u/l5VlYWWVlZHud98MEHHt9/8cUX89VXX2nX0MnQriGllPIsZFoEYTY9a0gp5Xt33HEHn332WZtpd911F7Nnzw5Qoq6FTCHQriGllD88++yzgY7gtZDpGgqz6wVlSinlScgUAkfTEBON2iJQSqnWQqgQWC2C2nptESilVGshUwhEBLto15BSSh0vZAoBgN2mXUNKqe5pGunz4MGDzJgxw+MyWVlZrFmzxp+xekRoFQLRriGl1Mnp27cvr7/+eqBj9KiQOX0UIMwG9Y1aCJQKOh/cB4c39uw6U0bDtN91OPvee+9l4MCBzTemeeihhxARVq5cybFjx6ipqeGRRx5h+vTpbd63Z88errzySnJzc6mqqmL27Nls3ryZM888s8tB51wuF3fccQfLli0jISGBRx55hF/84hfs27ePP/zhD1x99dXs2bOHm266iYqKCgCeeeYZzjvvPAAef/xxFixYQH19Pddccw0PP/zwyeyhZiHWIhDq6rVrSCkFM2fOZNGiRc2v//nPfzJ79mwWL17M2rVref/99/nZz36GMR1/Zzz//PNERUWxYcMG7r//fnJycjrdZkVFBVlZWeTk5BATE8OvfvUrPv74YxYvXswDDzwAQJ8+ffj4449Zu3YtixYt4ic/+Qlg3Zhm+/btZGdns27dOnJycli5cmUP7IkQbBHUaYtAqeDTyV/uvjJ+/Hjy8/M5ePAgBQUFJCQkkJqayt133938BXvgwAGOHDlCSkqKx3WsXLmy+Yt6zJgxjBkzptNthoeHM3XqVMC6J0FERAQOh4PRo0c3j2FUV1fHnXfeybp167Db7c2jjC5dupSlS5dywQUXYLPZKC8vZ/v27Vx00UUnvS9CqxCIXlmslGoxY8YMXn/9dQ4fPszMmTN59dVXKSgoICcnh+rqakaPHu3xPgSticgJb8/hcDQv3/qeBE33IwB46qmnSE5OZv369TQ2NuJ0OgEwxvDLX/6SG264QQedOxl2mw46p5RqMXPmTBYuXMjrr7/OjBkzKCkpoU+fPjgcDlauXMnevXs7ff9FF13Eq6++CkBubi4bNmw46UwlJSWkpqZis9l45ZVXaGhoAODyyy9n3rx5lJeXA1ZrJT8//6S3B6HWIrCJXkeglGqWkZFBWVkZ/fr1IzU1le9973tcddVVTJgwgYyMDM4444xO33/bbbcxe/ZsxowZw7hx45g4ceJJZ7r99tu59tpr+de//sXkyZOb75x22WWXsWXLFqZMmYLNZsPlcvGPf/yDPn36nPQ2Q6oQ2AVqtWtIKdXKxo0tZyslJSWxatUqgHbj/jf9JZ6ent58H4LIyEgWLlx4wttqWgdYZyl5mjds2LA2LYtHH320+fldd93FzTffrF1DJyNMu4aUUqqdkGsRaNeQUsrXJk2aRE1NTZtpr7zyCqNHjw5Qos6FViGwiZ41pFQQ6ewc/VPZF198EbBtd2efhlbXkLYIlAoaTqeToqKi07YYBIIxhqKiouZTTk9UiLUIoEpbBEoFhbS0NPLy8iguLvb6i8sfqqurT8lcTqeTtLQ0r9YZUoUgTKBOB51TKig4HA4GDRpEdnY248ePD3ScdkIpV0h1DdltQq12DSmlVBshVQis00e1a0gppVoLqUKgp48qpVR7IVUIwmxaCJRS6nghVQjsotcRKKXU8XxWCERknojki0huB/O/JyIb3I/PRWSsr7I00TuUKaVUe75sEcwHpnYyfzfwH8aYMcBvgBd9mAVoOkZg9AIWpZRqxWeFwBizEjjayfzPjTHH3C9XA95dAeGtssOMrllDFNXaPaSUUq2IL/86FpF04D1jzKgulrsHOMMYc0sH8+cAcwCSk5MzvRn2tUnv/M/I2Px7Lqt5jJ9dMoKIsBO/q5A/lJeX43K5Ah2jHc3lnWDNBcGbTXN5p7u5Jk+enGOMmeBxpjHGZw8gHcjtYpnJwBYg8UTWmZmZabpl+8fGPBhrrrnvSVNcUdu9dfjQihUrAh3BI83lnWDNZUzwZtNc3uluLmCN6eB7NaBDTIjIGOAvwDRjTJFPNxYRB0CMVOkN7JVSqpWAnT4qIgOAN4GbjDHf+HyDEdYdfWKo1GsJlFKqFZ+1CERkAZAFJIlIHvAg4AAwxrwAPAAkAs+JCEC96aj/qie4C4FLqnSYCaWUasVnhcAYM6uL+bcAHg8O+4QzFgAXVTrwnFJKtRI6VxY7ojEIMaJdQ0op1VroFAKbjVpbJDFo15BSSrUWOoUAqLVHadeQUkodJ7QKgS2SGKnUFoFSSrUSUoWgzt0i0GMESinVIuQKQYxoIVBKqdZCqhDU2yPdF5Rp15BSSjUJrUIQFo1LWwRKKdVGaBUCexQxeoxAKaXaCKlC0BAWSZTUUF9XF+goSikVNEKsEERZT2rLAhtEKaWCSEgVgsawaACkpjTASZRSKniEWCGIBEBqygOcRCmlgkdoFQKH1SKwadeQUko1C6lCYNwtAi0ESinVIrQKgcM6WGyv064hpZRqElKFoMF9sNhepy0CpZRqElKFoN59+qijXlsESinVJKQKQaMtgnpshGnXkFJKNQupQoAIFUThqK8IdBKllAoaoVUIgEoiCdeuIaWUahZ6hUCitBAopVQrIVcIyiQGZ4OeNaSUUk1CrhCU22KI1EKglFLNQq4QVNldRDXooHNKKdUk5ApBXXg80doiUEqpZiFXCExkAhHUQF1VoKMopVRQCLlCYItKAMBUHQtwEqWUCg4hVwjCXYkAlBcXBjiJUkoFh5ArBBGxViEoPZYf4CRKKRUcQq4QRMf1BqDiWEGAkyilVHAIuUIQ08sqBFWlRQFOopRSwSHkCkFCYjIAdeVaCJRSCkKwEMTHJVBn7DRUHA10FKWUCgohVwjsdhul4gI9fVQppQAfFgIRmSci+SKS28F8EZGnRWSHiGwQkbN8leV4FbYYbDXF/tqcUkoFNV+2COYDUzuZPw0Y5n7MAZ73YZY2qsNiCa8t8dfmlFIqqPmsEBhjVgKddcRPB142ltVAvIik+ipPa3XhcTjrdeA5pZQCEGOM71Yukg68Z4wZ5WHee8DvjDGful8vB+41xqzxsOwcrFYDycnJmQsXLuxWnvLyclwuFxGrnyCtags7sl5CRLq1rp7WlC3YaC7vBGsuCN5smss73c01efLkHGPMBI8zjTE+ewDpQG4H894HLmj1ejmQ2dU6MzMzTXetWLHCGGPM+pduNSUPJJuy6rpur6unNWULNprLO8Gay5jgzaa5vNPdXMAa08H3aiDPGsoD+rd6nQYc9MeG7dG9iJUqCov1lpVKKRXIQvAO8J/us4fOAUqMMYf8seHIuCQAvtmX54/NKaVUUPPl6aMLgFXACBHJE5EfisitInKre5ElwC5gB/AScLuvshxvYL9+APzz3xtpbPTdMRKllDoVhPlqxcaYWV3MN8Advtp+Z+wua7yhysJ9LN18hKmjUgIRQymlgkLIXVkMQL+zMLYwpkVt4x+r9wY6jVJKBVSnhUBEYjuZN6Dn4/iJMw7pP4ks+wbyjlUGOo1SSgVUVy2C7KYn7vP8W3urp8P41dBL6F+znYbSI02nryqlVEjqqhC0vtqqVyfzTj1DLgHg7IavKaupD3AYpZQKnK4KgenguafXp5aUMVRHJPIf9g3kl1YHOo1SSgVMV2cN9RGR/8b667/pOe7XvX2azNdsNqoTMxhYdYDDJTUM7RMT6ERKKRUQXbUIXgJiAFer502v/+LbaL4XFp1ADJUc0RaBUiqEddoiMMY83NE8ETm75+P4V4QrgVip4EiZFgKlVOjy6oIyERkJzARmASWA55HsThGOqHjiqCS/RAuBUip0dVkIRGQg1hf/LKAeGAhMMMbs8W00P3DGES71HC3RexMopUJXVxeUfY41JpADmGGMyQTKTosiAOCMA6CipCjAQZRSKnC6OlhcgHVwOJmWs4RO7dNGW3MXgqqyzm6kppRSp7dOC4ExZjowGlgLPCwiu4EEEZnoj3A+54wHoK7ymF5drJQKWV0OOmeMKTHGzDPGXAqcAzwI/EFE9vs8na+5WwTRjRUcragNcBillAoMr0YfNcYcMcY8bYw5D7jAR5n8x10IYqnkSGlNgMMopVRgdHrWkIi808X7r+7BLP7XVAikgvyyakbS4WCrSil12urq9NFzgf3AAuALTvWB5o7XqkVQUlUX4DBKKRUYXRWCFOBSrGsIbgDeBxYYYzb5OphfOJwYewSx9VoIlFKhq6uzhhqMMR8aY76PdaB4B5AtIj/2Szp/cMYRSwUllVoIlFKh6USuLI4AvoXVKkgHngbe9G0s/xFnHAllVezSFoFSKkR1dbD478Ao4APgYWNMrl9S+ZMzjgR7JaXVWgiUUqGpqxbBTUAFMBz4iUjzsWIBjDHm1D/NxhlHvG2/HiNQSoWsroah9uo6g1OSM45YtmohUEqFrNP/i74rzjhcpoKSKr1vsVIqNGkhcMYRaSop1RaBUipEaSFwxuEwtVRVVQQ6iVJKBYQWAvfVxfaaUuobGgMcRiml/E8LQavxhkqr9TiBUir0aCFw35MgFj1OoJQKTVoImlsEOt6QUio0aSFoHoG0QguBUiokaSHQFoFSKsRpIdB7EiilQpwWAocTYw8nVrRrSCkVmrQQYA1FHW+r0rOGlFIhyaeFQESmisg2EdkhIvd5mB8nIu+KyHoR2SQis32Zp0POOBLtVdoiUEqFJJ8VAhGxA88C04CRwCwRGXncYncAm40xY4Es4AkRCfdVpg4540iwtRwjWLrpMH/+ZKffYyilVCD4skUwEdhhjNlljKkFFgLTj1vGADFi3ejABRwF/H95rzOOOLFuTlNb38iv387lj8u3Y4zxexSllPI38dWXnYjMAKYaY25xv74JmGSMubPVMjHAO8AZQAxwvTHmfQ/rmgPMAUhOTs5cuHBhtzKVl5fjcrnaTR+56ffUFu3m6sYnuGKQg39sqQXgqaxIEpz+OYzSUbZA01zeCdZcELzZNJd3uptr8uTJOcaYCZ7mdXnP4pMgHqYdX3UuB9YBFwNDgI9F5N/GmNI2bzLmReBFgAkTJpisrKxuBcrOzsbje8sWU1u2jdJSWLv1G5xh6VTXQ/KwMZw3JKlb2+qxbAGmubwTrLkgeLNpLu/4Ipcv/9zNA/q3ep0GHDxumdnAm8ayA9iN1TrwL2cc4XVlPDs1jiUR/8OTYw8AsLtQh6ZWSp3+fFkIvgKGicgg9wHgmVjdQK3tAy4BEJFkYASwy4eZPHPGQUMNl/c+BsDUtHqcDhu7C7QQKKVOfz7rGjLG1IvIncBHgB2YZ4zZJCK3uue/APwGmC8iG7G6ku41xhT6KlOH3FcXk78ZAFvVUdITR2mLQCkVEnx5jABjzBJgyXHTXmj1/CBwmS8znBD3UNQc2WT9rChkcO9othwqC1gkpZTyF72yGNq1CKgsYnCSi31HK6nTu5YppU5zWgigpRAUuS8iqyxiUFI0DY2G/UcrA5dLKaX8QAsBtBQC02D9rCxiUO9oAH6y8GvuXrROLy5TSp22tBBASyFoUlHI8OQY+sVHcqS0hsVfH2DrYT1eoJQ6PWkhgLaFwBENVUdxOWx8dt/FfHjXhdhtwrvrj78EQimlTg9aCADCnGB3j3WXMhpMI1QXA5DoiuD8oUm8u+Ggdg8ppU5LWggARFpaBaljrJ+VRdbP0kP8IO0Q+49WsW5/cUDiKaWUL2khaNJUCFJaFQJj4F/fZ/JXt2Kngc93FgUun1JK+YgWgibOOBA7JLtvmVBRCJvfgv1fIHWVZITnc7SiNqARlVLKF7QQNHHGQUwquJKt1+WH4eMHIcoafTQzIo9jWgiUUqchLQRNRl0LE34AUYnW661LoHgvTHsM7OGMCtvH0UotBEqp049Pxxo6pYy/seW5Ixp2rQBbGAy7DHqfwfDiPdoiUEqdlrRF4ElUonUKaf9zwBkLKWNIr9tJUXlNoJMppVSP00LgSbS7e2joJdbPlNHENBRjr8wPXCallPIRLQSeNB0nGDrF+pkyGoBB9buormtou2zZYair8mM4pZTqWVoIPIkfCLFpkDzKep0yCoMwRnZRXFnXslxDPbxwIbw5JzA5lVKqB2gh8GTKQ3DLMrC5d48zjtKEDC60b2h7LcGBHKjIhy3vwJ7PAhJVKaVOlhYCT5yxEJvaZlJ5WhZnyXZKjxW0TNy5HMQGrhR4/79hxSNQsM3PYZVS6uRoIThBjUMuxi6GsL0rWybuWAb9JsC3noBje+GTx+DV66Bezy5SSp06tBCcoKjB51Bqoog98Ik1oaIIDqy1DiifeSX86jDctNi6CO2LPwc2rFJKeUEvKDtBcdGRfNw4iksOfQh/mQI1ZYBpObMIYMjF1gVo2Y9axw3G3wiZPwhUZKWUOiHaIjhBYXYbi8KuZG/0WIiIgdi+MP4m6Duu7YJXPG4Vh2N74au/BCSrUkp5Q1sEXtgXPZY/pl7IMzec1fFCCelw/SvWgHWrnoX6WggL91tGpZTylrYIvJAQHc6xEx14LmU0NNZBoZ5FpJQKbloIvJAQFU5+aQ2PfbiVL3cf7XzhphvcHN7o+2BKKXUStBB4ITE6nO355TyfvZM5r6zhYHEnQ0skDoGwSC0ESqmgp4XAC71cVl//jecMoK6+kTtfW0tlbb3nhW12SM7QQqCUCnpaCLxww8QBPHHdWH4zfRS/nzGWdfuLmfXi6o6Hp04ZbRUCY/wbVCmlvKCFwAv9e0VxbWYaIsK3xqTy55smsPVwGb99f4vnN6SMgupieOt2ePcu+PgBqDrm18xKKdUVLQQn4dKRyczITGNJ7iHKquvaLzB4snU66c7lsO0D+PxPsPx//Z5TKaU6o4XgJM3ITKO6rpElGw+1n5k4BO5aD/d8Yz3O/i/ImQ/5HbQglFIqALQQnKRx/eMZ0juaBV/uJ/dACe+sP8jz2TvZV1TZfuGs+6yrkuddDs+dC3k5PRPi6G7462VQtLNn1qeUCilaCE6SiHDdhP6s21/MlX/6lJ8s+JrHPtxK1v9bwR+XbW+7cFQvuO7vMOIKqCiE9++GxsaTD5H7Buz/Apb++uTXpZQKOTrERA+YfX46w5Nd1DUY+sVHkhAdztz3NvP0/23nW2NSGNonpmXhIZOtx/pFsHgObPwnjJ3Z9UZqK6zbYjZJGNRy45wd7vsibHsfdv8bBl3Ys/9ApdRpzactAhGZKiLbRGSHiNzXwTJZIrJORDaJyCe+zOMrEWF2Lj4jmcszUhjVL45+8ZHM/fYoosLt/Pb9LRhPp4+Ovg76jodlD0Oth26k1qpL4JmJ8KezWh5v3NwyL+9LmHSbdXvNpff3TCtDKRUyfFYIRMQOPAtMA0YCs0Rk5HHLxAPPAVcbYzKA63yVx98SXRHcOXkoK7YVMGHuMn79Vm7bM4tsNrj8ESg7aA1O15l/PwGlB2Da4/CdlyBzNmxaDLtXWo/GejjjWzDlQTi0HjYs8u0/Til1WvFl19BEYIcxZheAiCwEpgObWy1zA/CmMWYfgDEm34d5/O6WCwfTKzqcT3cU8uoXe1m+5Qjzb57I8GR3V9HA8+DMq+DTp+g3cBZ8ub39ShrqYPULVvfRpDnWtDOvtu6O9sG9EN0bwmOg/0QQO6x+HpY/DLXl7dcVFgGjroXwaN/9o5VSpxzx2G3REysWmQFMNcbc4n59EzDJGHNnq2X+ADiADCAG+KMx5mUP65oDzAFITk7OXLhwYbcylZeX43K5uvXek7WzuIE/fV2DAL86x0lipNUYi6w8xFlrf46jvqzD99Y64sjJfJIaZ1LztKSCVYzc/Dg208Dh5Cy2nnk3ALElWxi7/kHsjZ6vdj6UMoVtZ/z4hHMHcp91RnN5L1izaS7vdDfX5MmTc4wxEzzONMb45IHVzfOXVq9vAv503DLPAKuBaCAJ2A4M72y9mZmZprtWrFjR7ff2hM0HS8yoBz40I361xFz3/OfmkSWbzdq9R42prTSffvSWMeUFnh+1VZ5XWFVizW9oaDu9uszzet7/uTEPxhlzaMMJZw70PuuI5vJesGbTXN7pbi5gjenge9WXXUN5QP9Wr9OAgx6WKTTGVAAVIrISGAt848NcAXNmaiyLfnQu/8rZz9p9xcz7dLf1+MHZNITHQXRS1ytpzRnreXqEy3ocb/IvrbOUFt0Efc5smS42uOBuSPP8x4JPZD8Gh9ZBXBpc+r/W6bRf/hku+kXbf1fVMeu02MqiNm8fVVgIh9z3hu57Flx0D4hY4zqteASO5EKvwTDlIbA7updx5/9B/lY49/aul/30Kejnx/2nVA/yZSH4ChgmIoOAA8BMrGMCrb0NPCMiYUA4MAl4yoeZAm5k31ge7JsBQEllHde/uIpbX8nh55nd/LLyRmQCXP0nWPk4lOxvmV6SB0c2wR1f+uduauUFkP0IxPaDbUsgJhXy1linv9rCrC/vJisehXWvWiO5thJRUw4lVVBXZa0jdQwMv9w6iL7y91YR2LbEGuJj4n95n7GqGF7/IVQdtW5HOvC8jpfdsQyWPQSuFOzj/uD9tpQKMJ+dNWSMqQfuBD4CtgD/NMZsEpFbReRW9zJbgA+BDcCXWF1Jub7KFGziohzMnz2R+Khwnsyp9nw1ck878yr40Uq49dOWx3f+Asd2w1cv+X77ALtWWD+vf8W6uO6Tx6wiEN0bVj1n3e8ZoHA7rPkrZP6gbd5bPyVnwh+s57evhsShsPRX1rUWyx6C5FFw5xoYeAFkP2qdYuutf/8/qzUSlQgf/U/Hp+Q2NlgtlqgkKD9M//2Lu7FDlAosn15QZoxZAiw5btoLx71+HHjclzmCWUqck7/ffDbT/7SSWS+t5onvjuWcwYn+DTFsCgy5BD66v92geBc2NsK6FLjpLeuv5IU3WCOqAkTEwszXoP/Z3m1vx3LrCzZ1vNUttH0pxPWH/3wbnj/fuk7CFmadFhsWCVn/0/G67A649DewcBb8bqB1e9Cb3rLuB3H5b+HFLPj9EOu1N+qrYdyN1sV5i38Ev02xup6OZxqhoRa++zJseov0TYtg7tvebaunTLgZpj4amG2rU5peWRwEhvaJ4Z4JTv62TZj54mrunjKcn1wyFPH0xeMr334OvvorNLQ92+jAvn0MOPKxNYR22WHAwKQfWTPXL4IPfg63/F/LVc5daWy0RmMdPNl6T9IwmLkAYvtag/TdsMia32T4VHD17nydI6ZZXV5FO6DPSOvKbbC6dL77MhxYc2LZWgt3WV1KEXHWqbjF+zpettdg65TegRewpzyc9LRk77d3svK3wurnIOM73hdmFfK0EASJQXF2PvzpBfzqrVyeWvYN6/OKSUuIZGBiNOcPTeSMlA4ODPeUmBS4+P52k3dlZzNgWAb831xrwvRnYfyN1vM+I62/lj//I/SfdGLbKd4HFQUwdErLtOGXtTwf/B/WwxsicNZ/ep438mrrcTLOvuXElotOZM+gWaRnZZ3c9rqjptxqSX30S6uVlTyq45MJlDqOFoIgEhUexhPXjWVAryheWbWXNXuOUlpdjwg8eOVIfnD+oMAEO/dOyHkZohJg7KyW6aO/C1/82eqX94YtDIZc3KMRQ16ECy55AN6+A/42DZJGwG2fdf+MKRVStBAEGRHhp1OG89MpwwE4XFLNA2/n8tC7m6lvNNxy4WD/h3JEwo8+AXt42752mw1+8B7kfeXd+lzJEBOA7pPT3fgbrZbA/i/gg1/Amnkt3XhKdUILQZBLiXPy/I2Z3PaPHH7/4TayRvQh1hnGy6v2smpXEQ9fnUFG31hW7Sri5c/30mgMN0waQGpcJAAOuzAoKfrkjzdE9fI8PTwaBmed3LpVz+k7DlLHwtb3rTOmKgqs6Y4o7PUjAhpNBS8tBKcAu02Ye80oLn1yJTf99QsKy2uobzTEOh1c/+dVDEiMZsuhUhKjwxGBpZuPtHn/RcN784vLRxDjDKN/QhQ2mx8PQiv/E7HOHnrlGmvAQgDTyJDUS2HKtwKbTQUlLQSniD4xTv53egYPvL2JGyYOYPb5g4gMt3PL39dQU9/AY9eOZvq4fgB8vrOQ6jrrvPd9Ryv547LtXPmnTwEY0juaGyYNZNKgXoxMjW1XFCpq6gmzCxFhXp5uqYJLcoZ1e9QmH91P6qpn4fBGSBkduFwqKGkhOIVMH9ev+cu+yTt3nt+u2+fiM9r2v181tm/zgedFX+3jN+9ZA8AO7h3NjZMGMnFQLwrKa/go9zBvfn0ADExIT+Cxa8f49h+k/Oeie6j/6u84/jHDOk338t9a98NQCi0Ep7wT6fvvFx9JP3cBuXHSAA4UV7FqZxEvr9rL/77XMip4RJiNa89KI8YZxsIv9/Gd5z9nSj9DUU4eV45N1VbCqSwygS1n3s2YqlVwcC2882OY84n3F9qp05IWghAjIqQlRHHdhChmZKax/2gV6/OKSXSFMzYtnugI6yMxIzON2X/7igVbq1iwdT1/XrmTm85Nxy6CwbD/aBU7C8qZcmYfLs9Iae5iOlJSzcYDJVTXNVJQVsPmQyWM65/AtWf1wxne9ksnJiLshApZQ6PBrsc1TtrRxEzI+pl1j+vXb4a1f7fuTxFg9vqK7g0D4mNtcoW7vC+ajQ3WxYj2CHA4rWl1VdaV6N0OFdH993ZCC0EIExEGJEYxIDGq3bzhyTFk/zyLD5d/QlT/kdy/OJdfv9UyDFSYTegdE8HHm49w7xsbO1i/1Rr5aNMRHvtwa7v58VEOxveP56wBCaTEOdvMq6prYP3+Er7ed4zdRRVMHtGHWy4YxLlDEv17xfXpKOM71g2M3rvbegTYhQCfBjpFe21ypYyBW5ZZN3c6EQ11MG+qdVV7RJx1mnV1Mbx2PdSdxJhi5/8UHJO7//4OaCFQHXLYbcSEC1lnJnPhsN4cq2z5SyYu0kFEmI1Vu4rYfLC0zfRx/eOJjXQQHRGGKyKMTQdLWL3raJt7NxsDO/LLWbvvGCu2FXjcfmJ0OOMHJHDR8N68t+EgN/zlC0Ykx9C/VySmooZ+Z5YxuLeLgrIa1u47xtq9xygsr2FUvziSXNZ/2EZj2JFfTn5ZDdeM78d5nRSSxkZDU8La+kZyD5ZQUVPP2DTr32OTE+uKC3oicP2rsOlNa6ykANuxYwdDhw4NdIx2mnNVFFjDjH/5Epx3Z9dvBMiZbxWBc++Eda9ZAxdWHrWGmp90a/dD9R0Pu0+iRdEBLQTqhISH2UiOdbabft6QJM4b0vl9FDL6xpHRN67D+aXVdZRU1rWZ5rDbSI6NaP7ivW/aGbyz7iCv5+RxsLia7UfqWf7UynYZE6IcvLWu7W0vwmxCZLid13Py6B0TweCkaLYeLsNhF0b1i8MZZmf/sUq2Hi6jobHjO/ZFhdsZkxZHQlQ4RRW1bMwroVd0OGcNTGB8/3iKq+pYvq6KH2d/RFpCFNeM78u+o5X0djm5NrMfB4urKSirwW4TzkiJYWBiVOAKS0wynHNbYLZ9nLyabIaemxXoGO20yXV4I3zye2solq66iEyjdQ1H+oVw2VxrKPQl91jzrpsPGdecXLDd2Sf3fg+0EKiAi3U6iHV2PhSC02Hnu2f357tnW/c6enfpCgqj0ymrrifGGcb4AQmMTI0lPMxGflk1FTUNze9NiXUiAu9tOMSn2wvYXVTJtFEp1DUYthwqpb6xkSRXBP914WCi3Mcx7DZheHIMrogwcg+UUFXXQGF5DRvySthZUE50RBjXTUijqLyWr3Yf5d31B7EJ9HPZ+NboVL7eV8wjS7biigijvKaep5a1v9dSYnQ4Y/vH44po+9/Q5QxjXFo88VHWPmk0hm2HyzlcWkVG3zj6xHTcPeGw2zgzNbZdV5s6SZfNhRcnwxs/PLHl7RHWmVki1jDqOfOt+4GM/LYPQ3afFgJ1SooJF67qYOylPjFO6w7Yx5mRmcaMzDSvt3XukK6HBT9SWk1UuJ2c1Z+RlTUGYwyHSqpJiXWyu6iC5VuOMDjJRf9eUdTUN5B7oJScvcfYdLCEmvq23TOF5TW89kXb0U5FrIPrC77cz4kIt9sIswsjUmIYlBgNAkcO1/BO/jpsIgzt4+KsAQmMSYvD6dAzh7rU50y4O7flSu2uRCWCq4/13O6AW5ZbLYkg7VrUQqBUDzi+20xE6BtvDfMxpLeLIb3b3jp0TFo8N0wa4HFdjY2GvUcrqaipb542IDGKmIgw8o5VUVJV5/F9YB1k35BXQkFZDdV1DWw6WMKXe44CUF3dwN6qo9Q1NPJ6Th5gtXwi3YVAgImDenHdhP7tWil94509M1SJB8U1jazdd4wRyTFU1zWw7UhZu/sAJcWEM6xPTGDPHotO8v52sk0cwd1C00KgVJCx2azxoTzp3yuqzY3APTk73fO4UNnZ2WS5h8guKq/h633FbMgrpqLW6karrmvgg9zDLN+a7/H90eF2wsOs+05EhNm5elxfhvVxsflQKfUNHR9bAahvNGw9XEpBWQ2j+sZRWF7DrsIKGhqNVdhWfI5NoJNDNLgiwhjXP570pCiE9gWhoKyGjQdKqKytJyo8jNH9WrbTdKJCRJidUf3iSI1z4rDbyOgbS+bAhMAerwkCWgiUCkGJrgimjExmysi2V6H/+sqRbDpY0uYL2RjYXVjOlkNlNLq/UI+UVvPXT3fT0GiIdNiJDO+8e0mwrmQfmxbPpoPWQfbLM1Jw2IXao4fIOns0Ww6VEhluZ1TfOCIcLTc6MgYOFFeydm8xOXuPsflQqcdtxDjDOGtgAglRDo5W1LLxQAmJrbYDUFZdz/q8YtbuO0ZVbQNVdVYRTHJF8N0JaVxyZp/mgrCjuAHnriJyD5RwsLgaEZq71Ib1cZ1WY3ZpIVBKNXM67GQObN+imDio/bTDJdWUVdcxpPfJfSlmZxeSNSqFqaNSOlmqF9eM9/74TmcaGg3b88tYu7eY7G35PP/JTp7L3tl2odWrAas11GBM8xherogw4iIdREfYGd0vniRXePNb0npFkdE3lnC7jZQ4Z/OpzMFMC4FSqltS4pyn9NlJ1mm8sZyREssNkwaw/2glOwvKm+dv2LCB8ePGMjw5huRYJ8YY9hRVkrP3GBvyiqmsbeBYRS2ffJNPuft4TqOxrkFpLTXOSYS7Sy3MfVbXgF6Rbbq37DbhsozkTk+z9iUtBEophfv4S69WV9kfCuPCYS33yxaxjt0MSoru8OwzYwx5x6r45oh1Tcqeogq2Hiqjwd2lVlXbwJo9R1myse29wRsaDX9cvp1BSdGEddK6uv7s/vji0jstBEop1UNEpH1BOQElVXUs+mof6/YXd7pckisCfDAskxYCpZQKsLhIB3MuGnJCy2Znb+/x7du6XkQppdTpTAuBUkqFOC0ESikV4rQQKKVUiNNCoJRSIU4LgVJKhTgtBEopFeK0ECilVIiT1veRPRWISAGwt5tvTwIKezBOTwrWbJrLO8GaC4I3m+byTndzDTTG9PY045QrBCdDRNYYYyYEOocnwZpNc3knWHNB8GbTXN7xRS7tGlJKqRCnhUAppUJcqBWCFwMdoBPBmk1zeSdYc0HwZtNc3unxXCF1jEAppVR7odYiUEopdRwtBEopFeJCphCIyFQR2SYiO0TkvgDm6C8iK0Rki4hsEpG73NMfEpEDIrLO/bgiANn2iMhG9/bXuKf1EpGPRWS7+2dCAHKNaLVf1olIqYj8NBD7TETmiUi+iOS2mtbhPhKRX7o/c9tE5HI/53pcRLaKyAYRWSwi8e7p6SJS1Wq/veDnXB3+3vy1vzrJtqhVrj0iss493S/7rJPvB99+xowxp/0DsAM7gcFAOLAeGBmgLKnAWe7nMcA3wEjgIeCeAO+nPUDScdN+D9znfn4f8FgQ/C4PAwMDsc+Ai4CzgNyu9pH797oeiAAGuT+Ddj/mugwIcz9/rFWu9NbLBWB/efy9+XN/dZTtuPlPAA/4c5918v3g089YqLQIJgI7jDG7jDG1wEJgeiCCGGMOGWPWup+XAVuAfoHIcoKmA393P/878O3ARQHgEmCnMaa7V5efFGPMSuDocZM72kfTgYXGmBpjzG5gB9Zn0S+5jDFLjTH17perAc93XPehDvZXR/y2v7rKJiICfBdY4Kvtd5Cpo+8Hn37GQqUQ9AP2t3qdRxB8+YpIOjAe+MI96U53M35eILpgAAMsFZEcEZnjnpZsjDkE1ocU6BOAXK3NpO1/zkDvM+h4HwXT5+5m4INWrweJyNci8omIXBiAPJ5+b8G0vy4EjhhjWt8g2K/77LjvB59+xkKlEIiHaQE9b1ZEXMAbwE+NMaXA88AQYBxwCKtZ6m/nG2POAqYBd4jIRQHI0CERCQeuBv7lnhQM+6wzQfG5E5H7gXrgVfekQ8AAY8x44L+B10Qk1o+ROvq9BcX+cptF2z84/LrPPHw/dLioh2le77NQKQR5QP9Wr9OAgwHKgog4sH7Jrxpj3gQwxhwxxjQYYxqBl/Bhk7gjxpiD7p/5wGJ3hiMikurOnQrk+ztXK9OAtcaYIxAc+8yto30U8M+diHwfuBL4nnF3Kru7EYrcz3Ow+pWH+ytTJ7+3gO8vABEJA74DLGqa5s995un7AR9/xkKlEHwFDBORQe6/KmcC7wQiiLvv8a/AFmPMk62mp7Za7Bog9/j3+jhXtIjEND3HOtCYi7Wfvu9e7PvA2/7MdZw2f6UFep+10tE+egeYKSIRIjIIGAZ86a9QIjIVuBe42hhT2Wp6bxGxu58Pdufa5cdcHf3eArq/WpkCbDXG5DVN8Nc+6+j7AV9/xnx9FDxYHsAVWEfgdwL3BzDHBVhNtw3AOvfjCuAVYKN7+jtAqp9zDcY6+2A9sKlpHwGJwHJgu/tnrwDttyigCIhrNc3v+wyrEB0C6rD+GvthZ/sIuN/9mdsGTPNzrh1Y/cdNn7MX3Mte6/4drwfWAlf5OVeHvzd/7a+OsrmnzwduPW5Zv+yzTr4ffPoZ0yEmlFIqxIVK15BSSqkOaCFQSqkQp4VAKaVCnBYCpZQKcVoIlFIqxGkhUOo4ItIgbUc77bHRat2jWAbqegelPAoLdAClglCVMWZcoEMo5S/aIlDqBLnHp39MRL50P4a6pw8UkeXuQdSWi8gA9/Rkse4DsN79OM+9KruIvOQeb36piEQG7B+lFFoIlPIk8riuoetbzSs1xkwEngH+4J72DPCyMWYM1sBuT7unPw18YowZizXu/Sb39GHAs8aYDKAY66pVpQJGryxW6jgiUm6McXmYvge42Bizyz0w2GFjTKKIFGINk1Dnnn7IGJMkIgVAmjGmptU60oGPjTHD3K/vBRzGmLl++Kcp5ZG2CJTyjungeUfLeFLT6nkDeqxOBZgWAqW8c32rn6vczz/HGtEW4HvAp+7ny4HbAETE7ucx/5U6YfqXiFLtRYr7puVuHxpjmk4hjRCRL7D+iJrlnvYTYJ6I/BwoAGa7p98FvCgiP8T6y/82rNEulQoqeoxAqRPkPkYwwRhTGOgsSvUk7RpSSqkQpy0CpZQKcdoiUEqpEKeFQCmlQpwWAqWUCnFaCJRSKsRpIVBKqRD3/wE4cc2WfQ2BQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_mae\", \"valid_mae\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='MAE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "911969d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/mlp-beckham-cement/version_16/checkpoints/epoch=178-step=895.ckpt\n",
      "Loaded model weights from checkpoint at logs/mlp-beckham-cement/version_16/checkpoints/epoch=178-step=895.ckpt\n",
      "/Users/sebastianraschka/miniforge3/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:219: PossibleUserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 10 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8de3728789984b59be6db2942c0e39f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_mae          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.5199999809265137     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_mae         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.5199999809265137    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_mae': 0.5199999809265137}]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}