{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright (c) Recommenders contributors.\n", "\n", "Licensed under the MIT License." ] }, { "cell_type": "markdown", "metadata": { "inputHidden": false, "outputHidden": false }, "source": [ "# Movie recommender with multinomial RBM (PyTorch, GPU)\n", "\n", "A Restricted Boltzmann Machine (RBM) is a generative neural network model typically used to perform unsupervised learning. The main task of an RBM is to learn the joint probability distribution $P(v,h)$, where $v$ are the visible units and $h$ the hidden ones. The hidden units represent latent variables while the visible units are clamped on the input data. Once the joint distribution is learnt, new examples are generated by sampling from it.\n", "\n", "In this notebook, we provide an example of how to utilize the RBM to perform user/item recommendations. In particular, we use as a case study the [movielens dataset](https://movielens.org), comprising user's ranking of movies on a scale of 1 to 5.\n", "\n", "This notebook provides a quick start, showing the basic steps needed to use and evaluate the algorithm. A detailed discussion of the RBM model together with a deeper analysis of the recommendation task is provided in the [RBM Deep Dive section](../02_model_collaborative_filtering/rbm_deep_dive.ipynb). The RBM implementation presented here is based on the article by Ruslan Salakhutdinov, Andriy Mnih and Geoffrey Hinton [Restricted Boltzmann Machines for Collaborative Filtering](https://www.cs.toronto.edu/~rsalakhu/papers/rbmcf.pdf).\n", "\n", "Following the paper, each visible unit is a **one-hot softmax unit** over the rating scale: an item rated $l$ is encoded as a vector of $r$ binary units with a single 1 in position $l$, and every rating value of every item has its own weights and bias. This is what allows the model to represent an arbitrary rating distribution per item. A single scalar unit per item, with $p(v=l) \\propto e^{l\\,\\phi}$, can only represent monotone distributions and therefore cannot express something as ordinary as *\"most users rate this movie a 3\"*.\n", "\n", "### Advantages of RBM:\n", "\n", "The model generates ratings for a user/movie pair using a collaborative filtering based approach. While matrix factorization methods learn how to reproduce an instance of the user/item affinity matrix, the RBM learns the underlying probability distribution. This has several advantages:\n", "\n", "- Generalizability : the model generalize well to new examples.\n", "- Stability in time: if the recommendation task is time-stationary, the model does not need to be trained often to accomodate new ratings/users.\n", "- The PyTorch implementation presented here allows fast training on GPU " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 0 Global Settings and Import" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:18.244173Z", "iopub.status.busy": "2026-08-07T06:48:18.243926Z", "iopub.status.idle": "2026-08-07T06:48:23.301304Z", "shell.execute_reply": "2026-08-07T06:48:23.300383Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "System version: 3.11.0 (main, Mar 16 2025, 13:39:19) [GCC 11.4.0]\n", "Pandas version: 2.3.3\n", "PyTorch version: 2.5.1+cu121\n", "CUDA available: True\n" ] } ], "source": [ "import sys\n", "import numpy as np\n", "import pandas as pd\n", "import torch\n", "\n", "from recommenders.models.rbm.rbm import RBM\n", "from recommenders.datasets.python_splitters import numpy_stratified_split\n", "from recommenders.datasets.sparse import AffinityMatrix\n", "from recommenders.datasets import movielens\n", "from recommenders.evaluation.python_evaluation import (\n", " exp_var,\n", " mae,\n", " map_at_k,\n", " ndcg_at_k,\n", " precision_at_k,\n", " recall_at_k,\n", " rmse,\n", " rsquared,\n", ")\n", "from recommenders.utils.timer import Timer\n", "from recommenders.utils.plot import line_graph\n", "from recommenders.utils.notebook_utils import store_metadata\n", "\n", "# For interactive mode only\n", "%load_ext autoreload\n", "%autoreload 2\n", "%matplotlib inline\n", "\n", "print(f\"System version: {sys.version}\")\n", "print(f\"Pandas version: {pd.__version__}\")\n", "print(f\"PyTorch version: {torch.__version__}\")\n", "print(f\"CUDA available: {torch.cuda.is_available()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1 Load Data \n", "\n", "Here we select the size of the movielens dataset. In this example we consider the 100k ratings datasets, provided by 943 users on 1682 movies. The data are imported in a pandas dataframe including the user ID, the item ID, the ratings and a timestamp denoting when a particular user rated a particular item. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:23.369492Z", "iopub.status.busy": "2026-08-07T06:48:23.368908Z", "iopub.status.idle": "2026-08-07T06:48:23.415919Z", "shell.execute_reply": "2026-08-07T06:48:23.414954Z" }, "tags": [ "parameters" ] }, "outputs": [], "source": [ "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n", "MOVIELENS_DATA_SIZE = \"100k\"" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:23.418612Z", "iopub.status.busy": "2026-08-07T06:48:23.418409Z", "iopub.status.idle": "2026-08-07T06:48:27.729292Z", "shell.execute_reply": "2026-08-07T06:48:27.727881Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", " 0%| | 0.00/4.81k [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIDmovieIDratingtimestamp
01962423.0881250949
11863023.0891717742
2223771.0878887116
3244512.0880606923
41663461.0886397596
\n", "" ], "text/plain": [ " userID movieID rating timestamp\n", "0 196 242 3.0 881250949\n", "1 186 302 3.0 891717742\n", "2 22 377 1.0 878887116\n", "3 244 51 2.0 880606923\n", "4 166 346 1.0 886397596" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = movielens.load_pandas_df(\n", " size=MOVIELENS_DATA_SIZE, header=[\"userID\", \"movieID\", \"rating\", \"timestamp\"]\n", ")\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Split the data using the stratified splitter \n", "\n", "As a second step we generate the user/item affiity matrix and then split the data into train and test set. If you are familiar with training supervised learning model, here you will notice the first difference. In the former case, we cut off a certain proportion of training examples from dataset (e.g. images), here corresponding to users (or items), ending up with two matrices (train and test) having different row dimensions. Here we need to mantain the same matrix size for the train and test set, but the two will contain different amounts of ratings, see the [deep dive notebook](../02_model/rbm_deep_dive.ipynb) for more details. The affinity matrix reads " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:27.732300Z", "iopub.status.busy": "2026-08-07T06:48:27.732115Z", "iopub.status.idle": "2026-08-07T06:48:27.792619Z", "shell.execute_reply": "2026-08-07T06:48:27.791444Z" }, "inputHidden": false, "outputHidden": false, "tags": [ "sparse_matrix" ] }, "outputs": [], "source": [ "# to use standard names across the analysis\n", "header = {\n", " \"col_user\": \"userID\",\n", " \"col_item\": \"movieID\",\n", " \"col_rating\": \"rating\",\n", "}\n", "\n", "# instantiate the sparse matrix generation\n", "am = AffinityMatrix(df=data, **header)\n", "\n", "# obtain the sparse matrix\n", "X, _, _ = am.gen_affinity_matrix()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The method also returns informations on the sparsness of the dataset and the size of the user/affinity matrix. The former is given by the ratio between the unrated elements and the total number of matrix elements. This is what makes a recommendation task hard: we try to predict 93% of the missing data with only 7% of information!\n", "\n", "We split the matrix using the default ration of 0.75, i.e. 75% of the ratings will constitute the train set." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:27.795836Z", "iopub.status.busy": "2026-08-07T06:48:27.795653Z", "iopub.status.idle": "2026-08-07T06:48:27.899014Z", "shell.execute_reply": "2026-08-07T06:48:27.897809Z" }, "tags": [ "split" ] }, "outputs": [], "source": [ "Xtr, Xtst = numpy_stratified_split(X)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The splitter returns:\n", "\n", "- Xtr: a matrix containing the train set ratings \n", "- Xtst: a matrix containing the test elements \n", "\n", "Note that the train/test matrices have exactly the same dimension, but different entries as it can be explicitly verified:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:27.902195Z", "iopub.status.busy": "2026-08-07T06:48:27.902007Z", "iopub.status.idle": "2026-08-07T06:48:27.930258Z", "shell.execute_reply": "2026-08-07T06:48:27.928997Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "train matrix size (943, 1682)\n", "test matrix size (943, 1682)\n" ] } ], "source": [ "print(\"train matrix size\", Xtr.shape)\n", "print(\"test matrix size\", Xtst.shape)" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "model", "train" ] }, "source": [ "## 2 Train the RBM model\n", "\n", "The model has been implemented as a PyTorch `nn.Module`. The training loop (Gibbs sampling followed by the contrastive-divergence parameter update) is hidden inside the `fit()` method, so no explicit call is needed. The algorithm operates in three different steps:\n", "\n", "- Model initialization: This is where we build the model, i.e. the weight tensor and the two biases. The constructor only takes the **model properties**: the number of visible units (items), the number of hidden units and the weight initialization. Since every rating value of every item has its own parameters, the weights are a rank 3 tensor of shape (items, ratings, hidden units) and the visible bias a matrix of shape (items, ratings). If a GPU is available the model is moved to it automatically.\n", "\n", "- Model fit: This is where we train the model on the data. Every **training hyperparameter** (number of epochs, minibatch size, learning rate, weight decay, dropout keep probability, Gibbs sampling protocol, metrics) is passed to `fit()` together with the training set matrix. Note that the model is trained **only** on the training set; the test set is used to display the generalization accuracy of the trained model, useful to have an idea of how to fix the hyper parameters.\n", "\n", "- Model prediction: This is where we generate ratings for the unseen items. Once the model has been trained, the rating of an item is inferred as its *expected* value under the learned distribution, $\\sum_l l \\, P(v=l|h)$. From these we extract the top_k (e.g. 10) most relevant recommendations. The prediction is then returned in a dataframe format ready to be analysed and deployed. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:27.933293Z", "iopub.status.busy": "2026-08-07T06:48:27.933113Z", "iopub.status.idle": "2026-08-07T06:48:28.143241Z", "shell.execute_reply": "2026-08-07T06:48:28.142086Z" }, "inputHidden": false, "outputHidden": false, "tags": [ "initialization" ] }, "outputs": [], "source": [ "# First we initialize the model class.\n", "# The constructor only takes the model properties (architecture + initialization).\n", "model = RBM(\n", " possible_ratings=np.setdiff1d(np.unique(Xtr), np.array([0])),\n", " visible_units=Xtr.shape[1],\n", " hidden_units=200,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the first time the fit method is called it may take longer to return the result. This is due to the fact that PyTorch needs to initialize the CUDA context on the GPU. You will notice that this is not the case when training the algorithm the second or more times. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:28.146568Z", "iopub.status.busy": "2026-08-07T06:48:28.146376Z", "iopub.status.idle": "2026-08-07T06:48:34.652126Z", "shell.execute_reply": "2026-08-07T06:48:34.651333Z" }, "inputHidden": false, "outputHidden": false, "tags": [ "training" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Took 6.34 seconds for training.\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAHACAYAAAD5vIKYAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAATalJREFUeJzt3XlYVPXiBvB3ZmBm2AZkG1bZXFADXEFEc6NwybSs3DXTTNPKKEtzayfrauaSdk1TK5MWW1wiDfcNFNxFREVBZVcYFlnn/P7w59wmUZEBzjC8n+eZ55Fzzhzec5+8r9+zfI9EEAQBREREVGtSsQMQERE1dixTIiIiA7FMiYiIDMQyJSIiMhDLlIiIyEAsUyIiIgOxTImIiAzEMiUiIjKQmdgBjJFWq8X169dhY2MDiUQidhwiIhKJIAgoLCyEm5sbpNJ7jz9ZptW4fv06PD09xY5BRERGIj09HR4eHvdczzKtho2NDYDb/+OpVCqR0xARkVg0Gg08PT11vXAvLNNq3Dm1q1KpWKZERPTAS368AYmIiMhALFMiIiIDsUyJiIgMxDIlIiIyEMuUiIjIQCxTIiIiA7FMiYiIDMQyJSIiMhDLlIiIyEAsUyIiIgOxTImIiAzEMiUiIjIQy5SIiMhALNN6tGZ/KhKu3BA7BhER1TOWaT36au9F7DmfK3YMIiKqZyzTeuRso0ROYZnYMYiIqJ6xTOuRk40COYWlYscgIqJ6xjKtR07WCo5MiYiaAJZpPXJWsUyJiJoClmk9crJRIKeoDIIgiB2FiIjqEcu0HjnbKFBRJSC/pELsKEREVI9YpvXIyUYBAMjmqV4iIpPGMq1HTtZKAOB1UyIiE8cyrUd3RqY5RXw8hojIlLFM65GFXAYbhRmyNRyZEhGZMpZpPbs9cQPLlIjIlLFM65mTjYI3IBERmTiWaT3jyJSIyPSxTOuZs40SOUUsUyIiU8YyrWdONgpka3g3LxGRKTOKMl2+fDm8vb2hVCoREhKC+Pj4e25bUVGB999/H35+flAqlQgKCkJMTIzeNu+++y4kEonex9/fv74Po1pONgpoSitRWlElyu8nIqL6J3qZRkdHIzIyEvPnz0diYiKCgoIQERGB7OzsarefM2cOvvrqKyxduhRnz57F5MmT8dRTT+HYsWN627Vr1w4ZGRm6z/79+xvicO7i/P/PmubyVC8RkckSvUwXLVqEF198EePHj0fbtm2xcuVKWFpaYs2aNdVu/+233+Kdd97BgAED4OvriylTpmDAgAFYuHCh3nZmZmZwcXHRfRwdHRvicO7CKQWJiEyfqGVaXl6OhIQEhIeH65ZJpVKEh4fj0KFD1X6nrKwMSqVSb5mFhcVdI8+UlBS4ubnB19cXo0aNQlpaWt0fQA3oZkFimRIRmSxRyzQ3NxdVVVVQq9V6y9VqNTIzM6v9TkREBBYtWoSUlBRotVrs2LEDmzZtQkZGhm6bkJAQrF27FjExMVixYgVSU1PRo0cPFBYWVrvPsrIyaDQavU9dsbeUQyaVcGRKRGTCRD/N+7C++OILtGzZEv7+/pDL5Zg2bRrGjx8PqfR/h9K/f388++yzCAwMREREBLZt24b8/Hz8+OOP1e4zKioKtra2uo+np2ed5ZVKJXC0lnNkSkRkwkQtU0dHR8hkMmRlZektz8rKgouLS7XfcXJywm+//Ybi4mJcuXIF586dg7W1NXx9fe/5e+zs7NCqVStcuHCh2vWzZs1CQUGB7pOenl77g6ouMyduICIyaaKWqVwuR6dOnRAbG6tbptVqERsbi9DQ0Pt+V6lUwt3dHZWVlfjll18wePDge25bVFSEixcvwtXVtdr1CoUCKpVK71OXnG2UyCnks6ZERKZK9NO8kZGRWLVqFdatW4ekpCRMmTIFxcXFGD9+PABg7NixmDVrlm77uLg4bNq0CZcuXcK+ffvQr18/aLVavPXWW7pt3nzzTezZsweXL1/GwYMH8dRTT0Emk2HEiBENfnwA4GTN+XmJiEyZmdgBhg0bhpycHMybNw+ZmZlo3749YmJidDclpaWl6V0PLS0txZw5c3Dp0iVYW1tjwIAB+Pbbb2FnZ6fb5urVqxgxYgTy8vLg5OSE7t274/Dhw3BycmrowwMAeDSzQMyZTAiCAIlEIkoGIiKqPxJBEASxQxgbjUYDW1tbFBQU1Mkp3x1ns/Di+qM4PKsvXGyVD/4CEREZhZr2geineZsCfxcbAEBSZt09ckNERMaDZdoAPJpZwFphhnMZ1T/nSkREjRvLtAFIJBK0drHBOY5MiYhMEsu0gfi72CA5kyNTIiJTxDJtIP6uKlzILkJ5pVbsKEREVMdYpg3E38UGlVoBF3OKxI5CRER1jGXaQFr//x29vG5KRGR6WKYNRKU0h7udBc7xuikRkclhmTagNq42fDyGiMgEsUwbkL+Liqd5iYhMEMu0AbV2sUGWpgw3i8vFjkJERHWIZdqAAj1sAQCHLuWJnISIiOoSy7QBeTlYIcjDFpsSr4odhYiI6hDLtIEN7eSB3ck5yC363/tNr+QV4/fj1/D5jvPIL+EpYCKixkb095k2NYMC3fDBlrP44/h1vNDdBwtizmHF7osAAIkEKKvUYmZ/f5FTEhHRw+DItIE1s5Kjj78zfkm8ipjTGVix+yKmh7dE4tzHMKmHL74/fAWFpRVixyQioofAMhXB0I4eOHNdg+nRxzEgwAWv9W0Jeys5xof5oLSyChvj08WOSERED4FlKoJerZ1hbyWHm60FFgwNhEQiAQC42CoxpL07Vu9P5YT4RESNCMtUBHIzKaIndUX0S6GwUZrrrZv0qC8yNaXYfOK6SOmIiOhhsUxF0lJtAycbRbXLw1o4YPNJlikRUWPBMjVCQR52nMOXiKgRYZkaIX9XFTI1pXzmlIiokWCZGqE2unefcnRKRNQYsEyNkLejFeQyKc5l8A0zRESNAcvUCJnLpGjhbI3kLI5MiYgaA5apkfJ3tUHSP25C2nziOm7w1W1EREaJZWqk/F1scD6rEFqtgAvZhXjlh2P4795LYsciIqJqsEyNlL+LCiXlVUi/WYKfE64BALacvA5BEERORkRE/8YyNVL+rrfv6D1zXYNfj11FG1cVrt68hZNXC0RORkRE/8YyNVJO1grYW8nx9b5LyNKU4aOnHoGjtRxbODMSEZHRYZkaKYlEAn8XGySm5aOlszU6eNqh3yMu2HoyA1otT/USERkTlqkR83dRAQCe6eQBiUSCJwLdcL2gFMfSb4qcjIiI/ollasSCPG0hl0nxVAd3AEAXb3s42ygQfSSdNyIRERkRM7ED0L09Eeh2u0BVSgCATCrB+DAfLIg5h4yCUnz8VAA87S1FTklERByZGjGZVAI3Owu9ZVN6+eGb57vgYnYRIhbvxZHLN0RKR0REd7BMG6He/s7YHtkTQR52eOGbIzh9jY/LEBGJiWXaSFkrzLBqXGf4Oltj7Jp4pOYWix2JiKjJYpk2YtYKM6wb3wUKMymnGiQiEhHLtJGzs5Qjop0L9iRn8w5fIiKRsExNQG9/Z1wvKEVKdpHYUYiImiSWqQkI8bGH0lyKXeeyxY5CRNQksUxNgNJchm5+jtiVzDIlIhIDy9RE9G7thKOXb6KwtELsKERETQ7L1ET0au2MSq2AAxdyxY5CRNTkcDpBE+Fpbwk/Jyv8ePQqisuqUCUIGNrRAzKpROxoREQmj2VqQvo94oLluy5i5//fiORoLUcff7XIqYiITB9P85qQNx5rjaNzwnHug35wUSlx6GKe2JGIiJoEjkxNiFQqgaO1AgAQ6ueAQ5dYpkREDYEjUxMV6uuAM9c1KCjh3b1ERPWNZWqiuvo6QBCAuFSOTomI6hvL1ER52lvA3c4Chy/xfadERPWNZWqiJBIJuvryuikRUUMwijJdvnw5vL29oVQqERISgvj4+HtuW1FRgffffx9+fn5QKpUICgpCTEyMQfs0VaF+DkjK0OBmcbnYUYiITJroZRodHY3IyEjMnz8fiYmJCAoKQkREBLKzq59nds6cOfjqq6+wdOlSnD17FpMnT8ZTTz2FY8eO1XqfpirUzwEAr5sSEdU3iSDySzBDQkLQpUsXLFu2DACg1Wrh6emJV155BTNnzrxrezc3N8yePRtTp07VLRs6dCgsLCzw3Xff1Wqf/6bRaGBra4uCggKoVKq6OEzR9PxsF0J9HfDJ0ECxoxARNTo17QNRR6bl5eVISEhAeHi4bplUKkV4eDgOHTpU7XfKysqgVCr1lllYWGD//v0G7VOj0eh9TMXg9u7YfOI6isoqxY5CRGSyRC3T3NxcVFVVQa3Wn/JOrVYjMzOz2u9ERERg0aJFSElJgVarxY4dO7Bp0yZkZGTUep9RUVGwtbXVfTw9Pevg6IzD8C6euFVRhd+PX9MtE/lkBBGRyRH9munD+uKLL9CyZUv4+/tDLpdj2rRpGD9+PKTS2h/KrFmzUFBQoPukp6fXYWJxudlZoI+/Gt8dToMgCDh5NR8hH8fih/g0saMREZkMUcvU0dERMpkMWVlZesuzsrLg4uJS7XecnJzw22+/obi4GFeuXMG5c+dgbW0NX1/fWu9ToVBApVLpfUzJqK7NkZShwdZTGZiw7ijKKrV459dT2HziutjRiIhMgqhlKpfL0alTJ8TGxuqWabVaxMbGIjQ09L7fVSqVcHd3R2VlJX755RcMHjzY4H2aqkdbOsGjmQWmbTgGhZkUOyIfxVPt3fF69HHsS8kROx4RUaMn+mneyMhIrFq1CuvWrUNSUhKmTJmC4uJijB8/HgAwduxYzJo1S7d9XFwcNm3ahEuXLmHfvn3o168ftFot3nrrrRrvs6mRSSWY0N0HzSzNsXZ8FzjbKPHpM4EI8LDFf/deEjseEVGjJ/pbY4YNG4acnBzMmzcPmZmZaN++PWJiYnQ3EKWlpeldDy0tLcWcOXNw6dIlWFtbY8CAAfj2229hZ2dX4302RePDfDAqxAtys9v/W5rJpAjxceCpXiKiOiD6c6bGyJSeM72fn46mY8bPJ5H0fj9YyGVixyEiMjqN4jlTEpefszUA4FJukchJiIgaN5ZpE+bneLtML+YUi5yEiKhxY5k2YbaW5nC0VuBiNkemRESGYJk2cX5OVriYwzIlIjIEy7SJ83O25mleIiIDsUybOD8na1zKKYJWy5u6iYhqi2XaxPk5WaGsUotr+bfEjkJE1GixTJs4P6c7d/QWoUor4J1fT+H0tQKRUxERNS6iz4BE4nK3s4DCTIqLOcXILizDhrg0ZGvK8PW4zmJHIyJqNFimTZxUKoGPoxXOXtfg0MVcOForEHsuC2l5JWjuYCl2PCKiRoGneQl+ztb47fg1ZBWW4dsJwbC1MMf6Q5fFjkVE1GiwTAl+Ttao0goY1sUTbVxVGBHcHNFH01FcVil2NCKiRoFlSujY3A7NLM0xvW9LAMDorl4oKa/CpsSrIicjImocWKaEXq2dcXTOY3BWKQHcvikpop0aaw9e5vOnREQ1wDIlALdfIP5Pz3fzwcWcYuy/kCtSIiKixoNlStXq4t0M7dxUWHvwsthRiIiMHsuUqiWRSPB8N2/sPJeN1FzO3UtEdD8sU7qnQUFusLeSYx1Hp0RE98UypXtSmsswMrg5fk64iryiMrHjEBEZLZYp3de4bt5QmEkR+eMJ3tlLRHQPLFO6LycbBRYNa48953Pw1d5LYschIjJKLFN6oJ6tnPByLz/8Z3syEtNuih2HiMjosEypRiIfawUvB0v8EJcmdhQiIqPDMqUaMZNJ0bu1Mw5cyIUg8NopEdE/sUypxrq3cMT1glJczisROwoRkVFhmVKNBfvYw0wq4RSDRET/wjKlGrNSmKFDczscSGGZEhH9E8uUHkpYC0ccvJiLKj5zSkSkwzKlh9K9hSM0pZU4fa1A7ChEREaDZUoPJcjTDlZyGQ5c5KleIqI7WKb0UMxlUoT4OmDv+RyxoxARGQ2WKT20JwJdcfjSDRzjbEhERABYplQLg9u7w9/FBlHbznECByIisEypFmRSCWYNaIP4yzew42yW2HGIiETHMqVa6dnKCT1aOuKTmHOoqNKKHYeISFQsU6q1t/v541JOMXYn82YkImraWKZUa4+428LPyQo7zmaKHYWISFQsUzLI4+1c8HdSNmdEIqImjWVKBnmsrRo3isuRcIWPyRBR08UyJYO097CDk42Cp3qJqEljmZJBpFIJHmurxvazWXzmlIiaLJYpGezxtmpcySvB+awiveXZmlKREhERNSyWKRks1M8B1gozRB9J1y37/fg1BH8ci5jTPP1LRKbPTOwA1PgpzGR4ubcfPo1Jho3SDGEtHDHjp5MAgL+TstDvEReRExIR1S+WKdWJl3u1AAB8GpOMFbsvopNXM7R2scGfpzMgCAIkEonICYmI6g/LlOrMy71awEZpju1nMrFsZEecvJqPtQcv43xWEVq72Igdj4io3rBMqU6N6eqFMV29AABdvO2hMJNi7/kclikRmTTegET1Rmkuu/0i8RTO3UtEpo1lSvXq0ZaOiE+9gdKKKrGjEBHVG5Yp1atHWzmhrFKL+NQbYkchIqo3LFOqVy2dreGiUmLPeZ7qJSLTxTKleiWRSBDe1hlbT2agki8RJyITZRRlunz5cnh7e0OpVCIkJATx8fH33X7x4sVo3bo1LCws4Onpiddffx2lpf+buu7dd9+FRCLR+/j7+9f3YdA9DO/SHJmaUo5Oichk1apMs7KyMGbMGLi5ucHMzAwymUzv8zCio6MRGRmJ+fPnIzExEUFBQYiIiEB2dna122/YsAEzZ87E/PnzkZSUhNWrVyM6OhrvvPOO3nbt2rVDRkaG7rN///7aHCrVgUfcbdHOTYWN/5hukIjIlNTqOdPnn38eaWlpmDt3LlxdXQ2a3WbRokV48cUXMX78eADAypUrsXXrVqxZswYzZ868a/uDBw8iLCwMI0eOBAB4e3tjxIgRiIuL09vOzMwMLi6cxs5YDO/iiXc3n0W2phTOKqXYcYiI6lStynT//v3Yt28f2rdvb9AvLy8vR0JCAmbNmqVbJpVKER4ejkOHDlX7nW7duuG7775DfHw8goODcenSJWzbtg1jxozR2y4lJQVubm5QKpUIDQ1FVFQUmjdvXu0+y8rKUFZWpvtZo9EYdFx0tyfbu+OjbUn4KeEqpvZuIXYcIqI6Vasy9fT0rJN3V+bm5qKqqgpqtVpvuVqtxrlz56r9zsiRI5Gbm4vu3btDEARUVlZi8uTJeqd5Q0JCsHbtWrRu3RoZGRl477330KNHD5w+fRo2NnfPxBMVFYX33nvP4OOhe7O1MMeAAFesPXgZaXklUJhL8VJPP7jbWYgdjYjIYLW6Zrp48WLMnDkTly9fruM4D7Z79258/PHH+PLLL5GYmIhNmzZh69at+OCDD3Tb9O/fH88++ywCAwMRERGBbdu2IT8/Hz/++GO1+5w1axYKCgp0n/R0XturDy/28IWPgxWSswrx+/HrePePM2JHIiKqE7UamQ4bNgwlJSXw8/ODpaUlzM3N9dbfuFGzB/QdHR0hk8mQlZWltzwrK+ue1zvnzp2LMWPGYOLEiQCAgIAAFBcXY9KkSZg9ezak0rv/fWBnZ4dWrVrhwoUL1e5ToVBAoVDUKDPVXhtXFX6cHAoA2JR4FZE/nsDpawV4xN1W5GRERIapVZkuXry4Tn65XC5Hp06dEBsbiyFDhgAAtFotYmNjMW3atGq/U1JScldh3rmD+F6nnouKinDx4sW7rquSeJ4McsOynRew+O/z+HpcF7HjEBEZpFZlOm7cuDoLEBkZiXHjxqFz584IDg7G4sWLUVxcrLu7d+zYsXB3d0dUVBQAYNCgQVi0aBE6dOiAkJAQXLhwAXPnzsWgQYN0pfrmm29i0KBB8PLywvXr1zF//nzIZDKMGDGiznKTYcxkUrzatyWmRx/Hyav5CPSwEzsSEVGt1bhMNRoNVCqV7s/3c2e7mhg2bBhycnIwb948ZGZmon379oiJidHdlJSWlqY3Ep0zZw4kEgnmzJmDa9euwcnJCYMGDcJHH32k2+bq1asYMWIE8vLy4OTkhO7du+Pw4cNwcnKqcS6qf4OC3LBkZwo+2HIW304IgdL84Z5RJiIyFhKhhrflymQyZGRkwNnZGVKptNpnSwVBgEQiQVVV435DiEajga2tLQoKCh7qHwb08OJTb2DM6jj0bOWEL0d1hJnMKCblIiICUPM+qPHIdOfOnbC3twcA7Nq1y/CERACCfeyxYnRHTFqfgBk/n8Si54IMmgSEiEgMNR6ZNiUcmTa8O3f3Rk/qihBfB7HjEBEBqIeRaXVKSkqQlpaG8vJyveWBgYGG7JaaoCHt3fGfv5Kx5WQGy5SIGp1alWlOTg7Gjx+PP//8s9r1jf2aKTU8qVSCgYGu2JR4DfMHteW1UyJqVGr1/1jTp09Hfn4+4uLiYGFhgZiYGKxbtw4tW7bEH3/8UdcZqYkYFOSGvOJyHLqUJ3YUIqKHUquR6c6dO/H777+jc+fOkEql8PLywmOPPQaVSoWoqCgMHDiwrnNSExDgbgsvB0tsPnEd3Vs44sOtSThzvQDfTQjhSJWIjFqtyrS4uBjOzs4AgGbNmiEnJwetWrVCQEAAEhMT6zQgNR0SiQSDAt2w/tBlWCvMseZAKgDgl8SrGNal+jf+EBEZg1r9c79169ZITk4GAAQFBeGrr77CtWvXsHLlSri6utZpQGpanghyhaa0EmsOpOLdQW0xKMgNi3acx61yXocnIuNVq5Hpa6+9hoyMDADA/Pnz0a9fP3z//feQy+VYu3ZtXeajJqa12gZPBrkhyNMOz4f5oI9/Cfou2o01B1L5HlQiMlp18pxpSUkJzp07h+bNm8PR0bEucomKz5kal/c3n8WPR9Ox963esLeSix2HiJqQmvbBQ5/mraiogJ+fH5KSknTLLC0t0bFjR5MoUjI+0/rcHpF+tfeiyEmIiKr30GVqbm6O0tLS+shCVC17KznGh3lj/cEryCksEzsOEdFdanUD0tSpU7FgwQJUVlbWdR6iak3s7gszqQQr93B0SkTGp1Y3IB05cgSxsbHYvn07AgICYGVlpbd+06ZNdRKO6A5bS3NM6OGDFbsvYtKjvlCrlGJHIiLSqVWZ2tnZYejQoXWdhei+XujugzX7U7FmfypmDWgjdhwiIp1alek333xT1zmIHkilNMfAQDfEnMnEzP7+fFUbERmNWl0z7dOnD/Lz8+9artFo0KdPH0MzEd1TX39nXMkrwaXcYrGjEBHp1KpMd+/efddr1wCgtLQU+/btMzgU0b2EtXCEwkyKnUnZYkchItJ5qNO8J0+e1P357NmzyMzM1P1cVVWFmJgYuLu71106on+xkMvQzc8BO89l48VHfcWOQ0QE4CHLtH379pBIJJBIJNWezrWwsMDSpUvrLBxRdfq0UeO9P86g4FYFbC3MxY5DRPRwZZqamgpBEODr64v4+Hg4OTnp1snlcjg7O0Mmk9V5SKJ/6uPvjLm/nca+lBw4WSsw7/czmPSoL4Z28hA7GhE1UQ9Vpl5eXgAArVZbo+0HDhyIr7/+mm+SoTrlbmcBfxcbLNpxHml5JXCyUeCNn07gZkk5JvbgqV8ianj1+sblvXv34tatW/X5K6iJCm+jxqWcYowJ9cLet3pjSi8/fLg1CXN/O42Ckgqx4xFRE1Or50yJxPZybz/09ndCJy97AMDb/fzholJiQcw5bD55HTMiWmNUiJfIKYmoqajXkSlRfbGUm+mK9I5x3byx+81eCG+jxuxfT2N/Sq5I6YioqWGZkklxVinx2TOB6OLdDB9uPYsqrcGv6yUieiCWKZkciUSCOQPb4lxmIX48mi52HCJqAlimZJKCPO3wVAd3LNyejMJS3pBERPWrXsv0nXfegb29/YM3JKoHb/VrjeKyKoxeHY/0GyVixyEiE1brMv32228RFhYGNzc3XLlyBQCwePFi/P7777ptZs2aBTs7O4NDEtWGq60FNk7qiryiMgxcsg87z2WJHYmITFStynTFihWIjIzEgAEDkJ+fj6qqKgC333O6ePHiusxHZJAgTztsfbUHgn0cMGl9AvaczxE7EhGZoFqV6dKlS7Fq1SrMnj1bb/rAzp0749SpU3UWjqgu2FqYY8Xojni0lROmfJeAY2k3xY5ERCamVmWampqKDh063LVcoVCguJjvmSTjYy6TYvnIjmjrqsL4tUeQW1QmdiQiMiG1KlMfHx8cP378ruUxMTFo06aNoZmI6oWFXIavxnTCrfIqRB/hIzNEVHdqNZ1gZGQkpk6ditLSUgiCgPj4ePzwww+IiorC119/XdcZieqMg7UCg9u74bvDV/DSo74wk/HpMCIyXK3KdOLEibCwsMCcOXNQUlKCkSNHws3NDV988QWGDx9e1xmJ6tTYUG/8ePQq/k7KRr9HXMSOQ0QmQCIIgkHzrZWUlKCoqAjOzs51lUl0Go0Gtra2KCgogEqlEjsO1YOnvzwApbkMG17sKnYUIjJiNe2DWp3junXrFkpKbj8Eb2lpiVu3bmHx4sXYvn177dISNbCxod44eDEP57MKxY5CRCagVmU6ePBgrF+/HgCQn5+P4OBgLFy4EIMHD8aKFSvqNCBRfegf4AIXlRJPLtuPyB+PI+5SHgw8SUNETVityjQxMRE9evQAAPz8889wcXHBlStXsH79eixZsqROAxLVB4WZDH9MC8O03i1w9PJNDPvvYfT+z26s3HMRFVVaseMRUSNTqxuQSkpKYGNjAwDYvn07nn76aUilUnTt2lU3tSCRsXNWKTGtT0u83KsF4lJv4Kej6fg05hzMpBJM7OErdjwiakRqNTJt0aIFfvvtN6Snp+Ovv/7C448/DgDIzs7mDTvU6EilEoT6OWDRsPYYHtwcS3deQEEJ3zRDRDVXqzKdN28e3nzzTXh7eyMkJAShoaEAbo9Sq5sZiaixmB7eEhVVWizblSJ2FCJqRGr9aExmZiYyMjIQFBQEqfR2J8fHx0OlUsHf379OQzY0PhrTtH3xdwqW77qA2Dd6wtPeUuw4RCSimvaBwc+ZmiKWadNWUl6JXp/tRmsXG6wdHwyZVCJ2JCISSU37oFY3IJWWlmLp0qXYtWsXsrOzodXq3/2YmJhYm90SGQVLuRkWPdceY9fE4T/bk/F2v8Z9poWI6l+tynTChAnYvn07nnnmGQQHB0Mi4b/cybR0b+mImf398fG2c3jEzRYDA13FjkRERqxWZbplyxZs27YNYWFhdZ2HyGi82MMXp65pEPnjcVjIpejjrxY7EhEZqVqVqbu7u+45UyJTJZFI8NkzgSirqMKL6xMQ9VQAFOZS/HH8OnwcrTCzvz/fOkNEAGr5aMzChQvx9ttvc4IGMnlKcxm+HNURQzu6461fTuK1jceRW1yObw5exqsbj6G8krMlEVEtR6adO3dGaWkpfH19YWlpCXNzc731N27cqJNwRMbATCbFgqGBGBDgCh9HK3g5WGH7mUxM23AME9Ydwbwn2qKlmmdqiJqyWj0aEx4ejrS0NEyYMAFqtfquG5DGjRtXZwHFwEdjqCb2p+TijZ+OI0tThl6tnfDek+3g5WAldiwiqkM17gOhFiwsLITjx4/X5qvVWrZsmeDl5SUoFAohODhYiIuLu+/2n3/+udCqVStBqVQKHh4ewvTp04Vbt24ZtM9/KigoEAAIBQUFtToeajrKKqqEXxLShbBPYoXe/9klFNwqFzsSEdWhmvZBra6Z+vv749atW7Wr+X+Jjo5GZGQk5s+fj8TERAQFBSEiIgLZ2dnVbr9hwwbMnDkT8+fPR1JSElavXo3o6Gi88847td4nUW3JzaR4uqMHvp0QgtzCMkzfeBxVWs6DQtTk1Kap//rrL6Fbt27Crl27hNzcXKGgoEDv8zCCg4OFqVOn6n6uqqoS3NzchKioqGq3nzp1qtCnTx+9ZZGRkUJYWFit9/lvHJlSbexOzhZ8Zm4RFvyZJHYUIqoj9Toy7devHw4dOoS+ffvC2dkZzZo1Q7NmzWBnZ4dmzZrVeD/l5eVISEhAeHi4bplUKkV4eDgOHTpU7Xe6deuGhIQExMfHAwAuXbqEbdu2YcCAAbXeZ1lZGTQajd6H6GH1bOWEmf398eXui/jjxHXd8vJKLbQcrRKZtIe+m7ei4varqVauXInWrVsb9Mtzc3NRVVUFtVr/YXi1Wo1z585V+52RI0ciNzcX3bt3hyAIqKysxOTJk3WneWuzz6ioKLz33nsGHQsRcHuih7PXNXjr5xPwbGaBo5dvYklsCp4P88Ybjxv294WIjNdDl6m5uTkcHBzQu3dvtGzZsj4y3dfu3bvx8ccf48svv0RISAguXLiA1157DR988AHmzp1bq33OmjULkZGRup81Gg08PT3rKjI1IRKJBJ8MDcTFnGI89eVBSCVAaxcVVu9PxQthPmhmJRc7IhHVg1qd5h09ejRWr15t8C93dHSETCZDVlaW3vKsrCy4uLhU+525c+dizJgxmDhxIgICAvDUU0/h448/RlRUFLRaba32qVAooFKp9D5EtaU0l+G/YzthXKgXtr7aA99NCIYgAN8cSBU7GhHVk1pN2lBZWYk1a9bg77//RqdOnWBlpf9s3aJFi2q0H7lcjk6dOiE2NhZDhgwBAGi1WsTGxmLatGnVfqekpET3/tQ7ZDIZAEAQhFrtk6iuudpa4L3Bj+h+HhnSHN8cvIyJj/pCpTS/zzeJqDGqVZmePn0aHTt2BACcP39eb93DvkEmMjIS48aNQ+fOnREcHIzFixejuLgY48ePBwCMHTsW7u7uiIqKAgAMGjQIixYtQocOHXSneefOnYtBgwbpSvVB+yRqaJMe9cW3h67g20NXMLV3C7HjEFEdq1WZ7tq1q84CDBs2DDk5OZg3bx4yMzPRvn17xMTE6G4gSktL0xuJzpkzBxKJBHPmzMG1a9fg5OSEQYMG4aOPPqrxPokamlqlxHNdPLBq3yWMDvGCrSVHp0SmpFbTCZo6TidI9SFbU4pe/9mNUSHNMXtgW7HjEFEN1LQP+P4oogbirFJiSk8/rD14GVfyisWOQ0R1iGVK1IAm9vCFo7UCn/yp/8zztfxb+CXhqkipiMhQLFOiBmQhl+Gtfq3x5+lM7Dh7+/Gt8kotJq0/ijd+OoHdyZw/mqgxYpkSNbDBQe6IaKfG9I3HcC5Tg8//Po/kzEL4u9jg/c1n+cJxokaIZUrUwKRSCT4f1h5eDlYYszoeK/dcxOuPtcLi4e1xOa8Yaw9ycgeixqZWj8YQkWEs5Wb4elxnDF5+AF287DG5px9kUgnGdPXCF3+noLC0EoIA9GjpiBBfB7HjEtED8NGYavDRGGooBSUVUJhLoTSX6X6esO4IMgpKUVhaAVtLc+x5szek0oebDIWI6kZN+4AjUyIR/XvyBltLc/w8pRsAIOHKTQxdcRD7L+Ti0VZOYsQjohriNVMiI9WxuR1aq23wQ3yabhlPJBEZJ5YpkZGSSCQYEeyJHWezkF1YipSsQvT4dBfW8u0zREaHZUpkxJ7q6AGZVILPd5zHyK/jcKO4HB9vO4ekDI3Y0YjoH1imREbM1sIcTwS64Yf4dNhamOPvyJ7wdbLC9I3HUVpRJXY8Ivp/LFMiIzelly8GBrhiw8QQuNlZYPHw9kjNK8anMcliRyOi/8cyJTJyLZxtsHxURzirlAAAfxcVZvX3x5oDqYg5nSlyOiICWKZEjdLz3bzR/xEXzPjpBN9AQ2QEWKZEjZBEIsGCZwLhYC3H5O8ScSG7UOxIRE0ay5SokVIpzbFidCfkFZUhfNFePP9NPFJzOUolEgPLlKgRa+Oqwr63e2Phs0FIzizEh1vOih2JqEnidIJEjZzCTIahnTxQXqXFO7+ewtWbJfBoZil2LKImhSNTIhMxuL0brOVmetMPElHDYJkSmQhLuRme7uiO6CPpfME4UQNjmRKZkFFdvZBbVI7oI2n4YMtZBL77Fw5ezBU7FpHJY5kSmZBWahsE+9hj7u9nEH0kHfZWcsz59TTKKjn1IFF9YpkSmZjZA9rglT4tsGdGL/x3bGek3SjBqr2X9La5UVyOKd8l4OTVfHFCEpkY3s1LZGKCPO0Q5GkHAHCwVmBCDx8s3XkBTwa5o7mDJcortZjyXQLiUm/gfFYhtr7aA0pzmbihiRo5jkyJTNxrfVvC0VqBp1ccxOr9qZj722kkpt3Ex08FIO1GCZbuTBE7IlGjx5EpkYmzlJsh+qWuWBKbgo+3JaFKK+CzZwLxbGdP5BSWYcnOFAwIcEU7N1uxoxI1WhJBEASxQxgbjUYDW1tbFBQUQKVSiR2HqM5czi3Gpdwi9PFXAwDKK7V4ctl+3KqoQvSkULjYKkVOSGRcatoHPM1L1IR4O1rpihQA5GZSrBrbGRWVWoxcdRjZhaUipiNqvFimRE2cp70lfpjUFSXlVRjx38O6yfK1WgFf77uEMavjMGn9Ucz+9RRKK/iIDVF1WKZEBC8HK/wwqSu0AjBo6X78dDQdL6w7gg+3JsFMKkFJeRW+j0tDwpWbYkclMkosUyICAPg4WuGPaWHo2coJM34+iZNXC7B2fBd8Mz4Y618IhrXCDCf4XCpRtXg3LxHp2CjNsWxkBwxJckeghy3Uqts3JEmlEjzirsKpqwUiJyQyThyZEpEeiUSCx9qqdUV6R5CHHU6yTImqxTIlohoJ8LDFtfxbyC0qEzsKkdFhmRJRjQR52AEAT/USVYNlSkQ14tHMAnaW5rwJiagaLFMiqhGJRIIAd1uOTImqwTIlohoL8rDDiasF4CykRPpYpkRUYwEetsgtKkOmhtMOEv0TnzMlohq7cxPSyt0XdYW6fGRHmMlu/7v86s0SWMnN0MxKLlZEIlFwZEpENaZWKeBmq8S6Q1eQUVCKHWezsHp/KgAg/UYJBi7Zj5e+S9D7Tm5RGU8Lk8ljmRJRjUkkEkS/FIpDs/rgj2nd8UKYDxbtOI/kzEK8/H0itFoB8ak3EHcpDwBw5noBQqNisflkhsjJieoXy5SIHoqnvSVcbS0AAG883houtkoMWX4AyVmF2PBiV/i72GDZrguo0gqYtekUKqoE/HUmU+TURPWLZUpEtWYhl+GTpwNRqdXi3UHtEOBhi2l9WmBfSi5m/HwCp64VoI+/M/aez0FllVbsuET1hmVKRAYJ9XPAsXmPY2RIcwBA/0dc4etkhU2J1zCmqxde69sShaWVfH0bmTSWKREZzFrxvwcDZFIJ3unfBp29mmFGRGsEuNvC0VqOXck5d32vvFLLm5PIJLBMiajOhbdV4+cp3WCjNIdUKkHPVs7YdS5bb5uiskqELdiJ7w5fESklUd1hmRJRvevj74zkrEJcy7+lW7Yh7gpyCsuw/tAVjk6p0WOZElG969HKETKpRDc6Lauswtf7UtHS2Rop2UU4lp4vbkAiA7FMiajeqZTm6OLdDGsOpOLqzRL8knANOUVlWDG6I9ztLPDT0XSxIxIZhGVKRA3iwyGPoKJKi8HLDmDpzhT0a+eCFs42eKaTBzafyEBJeaXYEYlqzSjKdPny5fD29oZSqURISAji4+PvuW2vXr0gkUju+gwcOFC3zfPPP3/X+n79+jXEoRDRPbRwtsHvU7vDz8kaGQWleLlXCwDAM508UFxeia2cJYkaMdEnuo+OjkZkZCRWrlyJkJAQLF68GBEREUhOToazs/Nd22/atAnl5eW6n/Py8hAUFIRnn31Wb7t+/frhm2++0f2sUCjq7yCIqEbsreT4bmIILucVo5XaBsDtGZXC/Byxat8lPN7OBbYW5iKnJHp4oo9MFy1ahBdffBHjx49H27ZtsXLlSlhaWmLNmjXVbm9vbw8XFxfdZ8eOHbC0tLyrTBUKhd52zZo1a4jDIaIHkJtJdUV6x+yBbZBdWIYxq+NQUFIhUjKi2hO1TMvLy5GQkIDw8HDdMqlUivDwcBw6dKhG+1i9ejWGDx8OKysrveW7d++Gs7MzWrdujSlTpiAvL++e+ygrK4NGo9H7EFHDaeOqwvcTQ5B2owRPrziAYV8dQtgnO7EhLk3saEQ1ImqZ5ubmoqqqCmq1Wm+5Wq1GZuaDJ8aOj4/H6dOnMXHiRL3l/fr1w/r16xEbG4sFCxZgz5496N+/P6qqqqrdT1RUFGxtbXUfT0/P2h8UEdVKOzdbfD8xBB7NLOGsUsLBWo6v91/iM6jUKIh+zdQQq1evRkBAAIKDg/WWDx8+XPfngIAABAYGws/PD7t370bfvn3v2s+sWbMQGRmp+1mj0bBQiUTQzs0W6164/fd5f0ouRq+Ow/H0fHRozss0ZNxEHZk6OjpCJpMhKytLb3lWVhZcXFzu+93i4mJs3LgREyZMeODv8fX1haOjIy5cuFDteoVCAZVKpfchInGF+jnARaXEpsRrYkcheiBRy1Qul6NTp06IjY3VLdNqtYiNjUVoaOh9v/vTTz+hrKwMo0ePfuDvuXr1KvLy8uDq6mpwZiJqGDKpBEM6uGPzyesoq6z+Eg2RsRD9bt7IyEisWrUK69atQ1JSEqZMmYLi4mKMHz8eADB27FjMmjXrru+tXr0aQ4YMgYODg97yoqIizJgxA4cPH8bly5cRGxuLwYMHo0WLFoiIiGiQYyKiujG0ozvySyrumiSfyNiIfs102LBhyMnJwbx585CZmYn27dsjJiZGd1NSWloapFL9zk9OTsb+/fuxffv2u/Ynk8lw8uRJrFu3Dvn5+XBzc8Pjjz+ODz74gM+aEjUyLdU2CPSwxc8J19DvEZ5ZIuMlEXir3F00Gg1sbW1RUFDA66dEItsQl4bZv53Ctld7oI0r/z5Sw6ppH4h+mpeI6H6e7ewBHwcrfLwt6a51F7ILsXzXBWi1HBOQuFimRGTUzGVSzOzvj30pudhzPke3vLC0AhPWHcVnfyVjQzwndyBxsUyJyOg91laNYB97fLw1CaUVVRAEATM3ncKNonI81laNT/48h4yCWw/eEVE9YZkSkdGTSCSYM7ANLuYUoeMHOzDsv4ex9WQGFjwTiP88GwQrhQyzfz2tmy1JqxWw/tBljFx1GIWlnOuX6h9vQKoGb0AiMk6pucXYdioD289kolsLR7zdzx8AsONsFl5cfxSdvZrh8XZq7EvJxb6UXEgkt9+jOirES+Tk1FjVtA9YptVgmRI1PptPXMfvx69hb0ou7C3l+PSZQKw/dBmZmlJseaWH2PGokappH4j+nCkRUV0YFOSGQUFuKK2ogkwqgblMiooqLSasO4pTVwsQ4GGr21arFbDvQi6SMzVIzS3GoCA3dPNzFDE9NXYsUyIyKUpzme7PPVs5wdVWiQ3xaYjyCAAAZBaUIvLH4zh4MQ+Wchks5WbYcTYL21/vCXsruVixqZHjDUhEZLLMZFIM6+KJP45fw/H0fPx370X0/2IvLuYU4dsJwTjzXgS2vdYdlVoB834/LXZcasRYpkRk0p7r7IlbFVUYsvwAFm4/jx4tnfDna4+iR0snSCQSONso8d6T7bDlZAa2nswQOy41UjzNS0Qmzc3OAquf7wJzqRSdvZvpnQa+48kgN/x5KhNv/HQcaTdKMKG7D+RmHGtQzfFu3mrwbl6ipqe4rBKf7ziPbw5ehq+jFb6bGAK1Sil2LBIZ5+YlInoIVgozzHmiLba80h25RWVYvuuC2JGoEWGZEhH9QxtXFV4I88HGI+nILiy9a/3Z6xoU3OKsSqSPZUpE9C9ju3lDIZNi9b5UveUZBbfwxNJ96PpxLGb/egrpN0pESkjGhmVKRPQvthbmGBPqhe8OX0F+Sblu+fYzWZBKJJjYwwd/ncnE5O8SRExJxoRlSkRUjRe6+6BKEPDNgcu6ZX+dyUSonwPeeLw1FgwNxJnrGpy+ViBeSDIaLFMiomo4WiswvEtzrD14GUVllbhZXI641BuIaOcC4PbsSk42Cvx0NF3kpGQMWKZERPfwUk9flJRX4rvDVxB7LhtaQcDjbdUAbs+u9HRHd/x2/DpKK6pETkpiY5kSEd2Dq60Fhnb0wNf7UvHHievo4GkH5388e/psJ08U3KrA30lZIqYkY8AyJSK6j8k9/XCjuAx7z+foTvHe0cLZGh2b2+HHo1dFSkfGgmVKRHQf3o5WeCLQDQDuKlMAGN6lOfaez0HnD//G2DXxOHAht6EjkhHg3LxERA8wa4A/Qnzt4e1odde6Zzt7wN5KjpNX87E3JRdjVsdhzsC2GB/mDYlEgooqLcxlHLeYOs7NWw3OzUtEtVGlFfDJn0lYtS8VLZytkVNYhuKySvw4ORQdmzcTOx7VQk37gGVaDZYpERliy8nr2J+SC097S91r3f6YFgYzjlAbnZr2AU/zEhHVsScC3XTXWbu3cMSQLw/gu8NX8HyYj8jJqL7wn0lERPUoyNMOI4KbY+H288jS/G/i/PQbJVix+yLyispETEd1had5q8HTvERUl/JLytF34R6UlFeht78TlGYy/HHiOiq1Ah5xV+GHF7vCRmkudkyqBt9nSkRkJOws5fjjle54pW8LXL15C/GXb2DWgDb4ZUo3pOWVYMK6o5xFqZHjyLQaHJkSUUNJuHIDo76OQ2sXFRYMDYC/S83/P+dSThFyCssQ4utQjwmbNo5MiYgagU5e9tjwYlcUl1XiiSX7sWjHedRkjJNTWIYRqw5j0rcJqKzSNkBSuh+WKRGRyDo2b4atr3bH1N4tsCQ2BRvi0+67fWWVFq/+cAyFpZUouFWBo1duNlBSuheWKRGREVCYyfD6Y60wumtzvL/5LM5lagAABbcqUFhaobftoh3nEZeah6/HdYaTjQKxnGhfdCxTIiIjMmdgW/g4WmHq94mYuiERXT78G09/eVB3g9KJ9Hys2HMRbzzeGt38HNHX3xmxSdkipyaWKRGREVGay7BsZAdkacqQklWIyb38cCWvBF/EpqCySotZm06hrasKLz3qCwDo20aNS7nFuJhTJHLypo0zIBERGZkWzjY4OiccCjMpJBIJFGZSLNyejGxNGc5lavDb1P9NTdi9hSMUZlLEJmXBz8la5ORNF0emRERGSGkug0QiAQC89Kgv2rnZ4pfEqxgb6o1ADzvddhZyGbq3cMTfSdmorNLi9LUC3CguFyl108WRKRGRkTOTSfH5sCB8tecS3ni81V3r+7ZRY85vp9Dhgx0oLK2EpVyG8WHeeLGHL+ws5SIkbno4aUM1OGkDETUm+SXlmPPbabR0tkGwjz32puRg7YHLqNRq0d7TDt38HPFCmA9sLW9PWXj6WgH+PJ2Bab1bwkIuEzm9ceMr2AzAMiWixi6nsAzbTmXg8KU87EvJhYO1HKvGdsbVmyWY+v0x3KqoQofmdlgzrguaWXH0ei8sUwOwTInIlKTllWDSt0dxJa8EZZVV6NtGjYndffDy94mwtTTHDy92hVqlFDumUeJ0gkREBABo7mCJTS93w6AgV7z4qC9Wju6EEF8H/DKlGzS3KrAg5pzYERs9likRURNgKTfDp88EYVb/NpBJb98l7O1ohdf6tsSvx67hfFahyAkbN5YpEVETNqxLc7jbWWDR9vNiR2nUWKZERE2Y3EyK18NbIeZMJk6k54sdp9FimRIRNXFDOrijhbM1Rq46jGdWHMSHW85C86/J9en+WKZERE2cTCrBN893wcu9W8DNzgLRR9Ix9MuDSMsrETtao8FHY6rBR2OIqCm7kF2EieuOoOBWBb6dEIJH3G3FjiQaPhpDRES10sLZGr9NDYOLrQVm/3oKWi3HXA/CMiUiorvYWcrx7qC2OHG1AJtPXtdbV1Jeif/uvYg953N0y1JzizF1QyIu5xY3dFSjwInuiYioWiG+DnisrRqf/ZWMfo+4QAIJfjt+DQu3JyNLUwYAeLmXH0L9HDBtwzEU3Lp909LykR3FjC0KoxiZLl++HN7e3lAqlQgJCUF8fPw9t+3VqxckEsldn4EDB+q2EQQB8+bNg6urKywsLBAeHo6UlJSGOBQiIpMys78/MgpKMXHdUXT7JBZv/XwSnb3tsWdGL7zdzx9f7b2EMavjEehhi3cG+GPryQycy9SIHbvBiV6m0dHRiIyMxPz585GYmIigoCBEREQgOzu72u03bdqEjIwM3ef06dOQyWR49tlnddt8+umnWLJkCVauXIm4uDhYWVkhIiICpaWlDXVYREQmwc/JGs9388bJqwV4ItANf01/FMtHdoSXgxWm9PLDxkldMau/P755vgvGh/nA094Ci3fcHrxczCnCr8euoinc5yr63bwhISHo0qULli1bBgDQarXw9PTEK6+8gpkzZz7w+4sXL8a8efOQkZEBKysrCIIANzc3vPHGG3jzzTcBAAUFBVCr1Vi7di2GDx/+wH3ybl4iov+5UxN3XlZ+Pz8eTcdbP5/Ec5098Ouxa6ioEvD12M4Ib6uu75j1olHczVteXo6EhASEh4frlkmlUoSHh+PQoUM12sfq1asxfPhwWFlZAQBSU1ORmZmpt09bW1uEhITUeJ9ERPQ/dy6n1cTTHdzh7WCJP05cxyt9WqJ7C0e8v+UsSiuq6jmluES9ASk3NxdVVVVQq/X/xaJWq3Hu3IPfYhAfH4/Tp09j9erVumWZmZm6ffx7n3fW/VtZWRnKysp0P2s0Te98PxFRXTCTSRH9UigkAJxVSlzILkS/xfuwen8qpvZuIXa8eiP6NVNDrF69GgEBAQgODjZoP1FRUbC1tdV9PD096yghEVHTo1Yp4fz/70dt4WyD57t5Y9nOC8gouCVysvojapk6OjpCJpMhKytLb3lWVhZcXFzu+93i4mJs3LgREyZM0Ft+53sPs89Zs2ahoKBA90lPT3/YQyEiont4LbwllOZSfLXnkthR6o2oZSqXy9GpUyfExsbqlmm1WsTGxiI0NPS+3/3pp59QVlaG0aNH6y338fGBi4uL3j41Gg3i4uLuuU+FQgGVSqX3ISKiumGjNMeYrl748Wg6Ckr0J9DPLCjFu3+cQcKVGyKlqxuin+aNjIzEqlWrsG7dOiQlJWHKlCkoLi7G+PHjAQBjx47FrFmz7vre6tWrMWTIEDg4OOgtl0gkmD59Oj788EP88ccfOHXqFMaOHQs3NzcMGTKkIQ6JiIj+ZUyoNyq1AjbEpwEAKqq0+HL3BfRZuBtrD17Gqz8cR1FZJQCgskqLHWezEJ96A7lFZffbrdEQfQakYcOGIScnB/PmzUNmZibat2+PmJgY3Q1EaWlpkEr1Oz85ORn79+/H9u3bq93nW2+9heLiYkyaNAn5+fno3r07YmJioFQq6/14iIjobk42CjzV3h1rD6ZiZHBzvBZ9DPtScjEu1BtPd3THsysP4T9/JWP2wDaYvvE4tp7K0H332U4eWDA0EFJpze4oFoPoz5kaIz5nSkRU985nFeLxz/fC0VqB0ooqrBzdCd1bOgIAVu9PxYdbz6KLtz0Sr9zEF8M7oKXaGvtScvHh1rOY1MMXswa0afDMNe0D0UemRETUNLRS2yC8jTNOXi1A9Etd0c7tf692e76bN7acvI5jaTexYnQnPPb/kzy0UttAKgHe23wWjtYKvPior1jx74tlSkREDWbZyI7QCgIs5fr1I5NKsGZcF+QUlaGV2kZv3fgwH+QUluGjbUnIKCjFOwP8YSYT/ZYfPSxTIiJqMEpz2T3XNbOSo5mVvNp1MyJaQ61S4v0tZ5GcpcHoEC+0VFvD28HKKIqVZUpEREZPIpFgXDdvtFRb480fT2DK94kAAEdrOZ4IdMOznT30Ths3eD7egHQ33oBERGTc8orKkJxViJ1J2fjjxHXcKC7H35E94e1oVae/p1FMdE9ERFQbDtYKdPNzxJwn2mLvW73RzEqO5bsu6NZrSitws7i8wfKwTImIqFFTmsvw0qO+2HTsGtLySlBaUYUX1x3FxPVHG+xdqrxmSkREjd6oEC+s3HMJS3amoKi0EsfT8/H9xJAavzrOUCxTIiJq9CzkMkzu6YsPtyZBJpXgq9Gd0NnbvsF+P8uUiIhMwqgQL+w4m4VhXTwR3lb94C/UIZYpERGZBAu5DNEv3f+NY/WFNyAREREZiGVKRERkIJYpERGRgVimREREBmKZEhERGYhlSkREZCCWKRERkYFYpkRERAZimRIRERmIZUpERGQglikREZGBWKZEREQGYpkSEREZiGVKRERkIJYpERGRgfg+02oIggAA0Gg0IichIiIx3emBO71wLyzTahQWFgIAPD09RU5CRETGoLCwELa2tvdcLxEeVLdNkFarxfXr12FjYwOJRFLr/Wg0Gnh6eiI9PR0qlaoOE9Y/ZhdPY87P7OJozNkB484vCAIKCwvh5uYGqfTeV0Y5Mq2GVCqFh4dHne1PpVIZ3X8gNcXs4mnM+ZldHI05O2C8+e83Ir2DNyAREREZiGVKRERkIJZpPVIoFJg/fz4UCoXYUR4as4unMedndnE05uxA488P8AYkIiIig3FkSkREZCCWKRERkYFYpkRERAZimRIRERmIZVpPli9fDm9vbyiVSoSEhCA+Pl7sSHeJiopCly5dYGNjA2dnZwwZMgTJycl625SWlmLq1KlwcHCAtbU1hg4diqysLJES39snn3wCiUSC6dOn65YZe/Zr165h9OjRcHBwgIWFBQICAnD06FHdekEQMG/ePLi6usLCwgLh4eFISUkRMfFtVVVVmDt3Lnx8fGBhYQE/Pz988MEHenOXGlP2vXv3YtCgQXBzc4NEIsFvv/2mt74mWW/cuIFRo0ZBpVLBzs4OEyZMQFFRkajZKyoq8PbbbyMgIABWVlZwc3PD2LFjcf36daPP/m+TJ0+GRCLB4sWL9ZaLlb02WKb1IDo6GpGRkZg/fz4SExMRFBSEiIgIZGdnix1Nz549ezB16lQcPnwYO3bsQEVFBR5//HEUFxfrtnn99dexefNm/PTTT9izZw+uX7+Op59+WsTUdzty5Ai++uorBAYG6i035uw3b95EWFgYzM3N8eeff+Ls2bNYuHAhmjVrptvm008/xZIlS7By5UrExcXBysoKERERKC0tFTE5sGDBAqxYsQLLli1DUlISFixYgE8//RRLly7VbWNM2YuLixEUFITly5dXu74mWUeNGoUzZ85gx44d2LJlC/bu3YtJkyaJmr2kpASJiYmYO3cuEhMTsWnTJiQnJ+PJJ5/U284Ys//Tr7/+isOHD8PNze2udWJlrxWB6lxwcLAwdepU3c9VVVWCm5ubEBUVJWKqB8vOzhYACHv27BEEQRDy8/MFc3Nz4aefftJtk5SUJAAQDh06JFZMPYWFhULLli2FHTt2CD179hRee+01QRCMP/vbb78tdO/e/Z7rtVqt4OLiInz22We6Zfn5+YJCoRB++OGHhoh4TwMHDhReeOEFvWVPP/20MGrUKEEQjDs7AOHXX3/V/VyTrGfPnhUACEeOHNFt8+effwoSiUS4du2aaNmrEx8fLwAQrly5IgiC8We/evWq4O7uLpw+fVrw8vISPv/8c906Y8leUxyZ1rHy8nIkJCQgPDxct0wqlSI8PByHDh0SMdmDFRQUAADs7e0BAAkJCaioqNA7Fn9/fzRv3txojmXq1KkYOHCgXkbA+LP/8ccf6Ny5M5599lk4OzujQ4cOWLVqlW59amoqMjMz9fLb2toiJCRE9PzdunVDbGwszp8/DwA4ceIE9u/fj/79+wMw7uz/VpOshw4dgp2dHTp37qzbJjw8HFKpFHFxcQ2e+X4KCgogkUhgZ2cHwLiza7VajBkzBjNmzEC7du3uWm/M2avDie7rWG5uLqqqqqBWq/WWq9VqnDt3TqRUD6bVajF9+nSEhYXhkUceAQBkZmZCLpfr/mLeoVarkZmZKUJKfRs3bkRiYiKOHDly1zpjz37p0iWsWLECkZGReOedd3DkyBG8+uqrkMvlGDdunC5jdf8diZ1/5syZ0Gg08Pf3h0wmQ1VVFT766COMGjUKAIw6+7/VJGtmZiacnZ311puZmcHe3t6ojqe0tBRvv/02RowYoZss3pizL1iwAGZmZnj11VerXW/M2avDMiUAt0d4p0+fxv79+8WOUiPp6el47bXXsGPHDiiVSrHjPDStVovOnTvj448/BgB06NABp0+fxsqVKzFu3DiR093fjz/+iO+//x4bNmxAu3btcPz4cUyfPh1ubm5Gn91UVVRU4LnnnoMgCFixYoXYcR4oISEBX3zxBRITEw16zaUx4WneOubo6AiZTHbXXaNZWVlwcXERKdX9TZs2DVu2bMGuXbv0Xj3n4uKC8vJy5Ofn621vDMeSkJCA7OxsdOzYEWZmZjAzM8OePXuwZMkSmJmZQa1WG212AHB1dUXbtm31lrVp0wZpaWkAoMtojP8dzZgxAzNnzsTw4cMREBCAMWPG4PXXX0dUVBQA487+bzXJ6uLictfNg5WVlbhx44ZRHM+dIr1y5Qp27Nih9wozY82+b98+ZGdno3nz5rq/v1euXMEbb7wBb29vAMab/V5YpnVMLpejU6dOiI2N1S3TarWIjY1FaGioiMnuJggCpk2bhl9//RU7d+6Ej4+P3vpOnTrB3Nxc71iSk5ORlpYm+rH07dsXp06dwvHjx3Wfzp07Y9SoUbo/G2t2AAgLC7vrMaTz58/Dy8sLAODj4wMXFxe9/BqNBnFxcaLnLykpueslyTKZDFqtFoBxZ/+3mmQNDQ1Ffn4+EhISdNvs3LkTWq0WISEhDZ75n+4UaUpKCv7++284ODjorTfW7GPGjMHJkyf1/v66ublhxowZ+OuvvwAYb/Z7EvsOKFO0ceNGQaFQCGvXrhXOnj0rTJo0SbCzsxMyMzPFjqZnypQpgq2trbB7924hIyND9ykpKdFtM3nyZKF58+bCzp07haNHjwqhoaFCaGioiKnv7Z938wqCcWePj48XzMzMhI8++khISUkRvv/+e8HS0lL47rvvdNt88skngp2dnfD7778LJ0+eFAYPHiz4+PgIt27dEjG5IIwbN05wd3cXtmzZIqSmpgqbNm0SHB0dhbfeeku3jTFlLywsFI4dOyYcO3ZMACAsWrRIOHbsmO6O15pk7devn9ChQwchLi5O2L9/v9CyZUthxIgRomYvLy8XnnzyScHDw0M4fvy43t/hsrIyo85enX/fzStm9tpgmdaTpUuXCs2bNxfkcrkQHBwsHD58WOxIdwFQ7eebb77RbXPr1i3h5ZdfFpo1ayZYWloKTz31lJCRkSFe6Pv4d5kae/bNmzcLjzzyiKBQKAR/f3/hv//9r956rVYrzJ07V1Cr1YJCoRD69u0rJCcni5T2fzQajfDaa68JzZs3F5RKpeDr6yvMnj1b7//AjSn7rl27qv3vfNy4cTXOmpeXJ4wYMUKwtrYWVCqVMH78eKGwsFDU7Kmpqff8O7xr1y6jzl6d6spUrOy1wVewERERGYjXTImIiAzEMiUiIjIQy5SIiMhALFMiIiIDsUyJiIgMxDIlIiIyEMuUiIjIQCxTIqpTu3fvhkQiuWteZCJTxjIlIiIyEMuUiIjIQCxTIhOj1WoRFRUFHx8fWFhYICgoCD///DOA/52C3bp1KwIDA6FUKtG1a1ecPn1abx+//PIL2rVrB4VCAW9vbyxcuFBvfVlZGd5++214enpCoVCgRYsWWL16td42CQkJ6Ny5MywtLdGtW7e73pJDZEpYpkQmJioqCuvXr8fKlStx5swZvP766xg9ejT27Nmj22bGjBlYuHAhjhw5AicnJwwaNAgVFRUAbpfgc889h+HDh+PUqVN49913MXfuXKxdu1b3/bFjx+KHH37AkiVLkJSUhK+++grW1tZ6OWbPno2FCxfi6NGjMDMzwwsvvNAgx08kCrFn2ieiulNaWipYWloKBw8e1Fs+YcIEYcSIEbo3eWzcuFG3Li8vT7CwsBCio6MFQRCEkSNHCo899pje92fMmCG0bdtWEARBSE5OFgAIO3bsqDbDnd/x999/65Zt3bpVACD66+OI6gtHpkQm5MKFCygpKcFjjz0Ga2tr3Wf9+vW4ePGibrt/vqTb3t4erVu3RlJSEgAgKSkJYWFhevsNCwtDSkoKqqqqcPz4cchkMvTs2fO+WQIDA3V/dnV1BQBkZ2cbfIxExshM7ABEVHeKiooAAFu3boW7u7veOoVCoVeotWVhYVGj7czNzXV/lkgkAG5fzyUyRRyZEpmQtm3bQqFQIC0tDS1atND7eHp66rY7fPiw7s83b97E+fPn0aZNGwBAmzZtcODAAb39HjhwAK1atYJMJkNAQAC0Wq3eNViipo4jUyITYmNjgzfffBOvv/46tFotunfvjoKCAhw4cAAqlQpeXl4AgPfffx8ODg5Qq9WYPXs2HB0dMWTIEADAG2+8gS5duuCDDz7AsGHDcOjQISxbtgxffvklAMDb2xvjxo3DCy+8gCVLliAoKAhXrlxBdnY2nnvuObEOnUhcYl+0JaK6pdVqhcWLFwutW7cWzM3NBScnJyEiIkLYs2eP7uagzZs3C+3atRPkcrkQHBwsnDhxQm8fP//8s9C2bVvB3NxcaN68ufDZZ5/prb9165bw+uuvC66uroJcLhdatGghrFmzRhCE/92AdPPmTd32x44dEwAIqamp9X34RKKQCIIgiNznRNRAdu/ejd69e+PmzZuws7MTOw6RyeA1UyIiIgOxTImIiAzE07xEREQG4siUiIjIQCxTIiIiA7FMiYiIDMQyJSIiMhDLlIiIyEAsUyIiIgOxTImIiAzEMiUiIjIQy5SIiMhA/wejJ1kTmxzQ8wAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Model Fit.\n", "# The training hyperparameters are passed to fit(), not to the constructor.\n", "# Contrastive divergence converges slowly, so the model is trained until the training\n", "# rmse flattens out; stopping after a handful of epochs leaves the model untrained.\n", "with Timer() as train_time:\n", " model.fit(\n", " Xtr,\n", " training_epoch=150,\n", " minibatch_size=100,\n", " learning_rate=0.001,\n", " l2=0.02,\n", " keep_prob=0.7,\n", " with_metrics=True,\n", " )\n", "\n", "print(f\"Took {train_time.interval:.2f} seconds for training.\")\n", "\n", "# Plot the train RMSE as a function of the epochs\n", "line_graph(values=model.rmse_train, labels=\"train\", x_name=\"epoch\", y_name=\"rmse_train\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "During training, we can optionally evaluate the root mean squared error to have an idea of how the learning is proceeding. We would generally like to see this quantity decreasing as a function of the learning epochs and eventually flattening out. To visualise this choose `with_metrics=True` in the `fit()` method.\n", "\n", "Once the model has been trained, we can recommend new items." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:34.656122Z", "iopub.status.busy": "2026-08-07T06:48:34.655529Z", "iopub.status.idle": "2026-08-07T06:48:34.804706Z", "shell.execute_reply": "2026-08-07T06:48:34.803831Z" }, "tags": [ "top_k" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Took 0.11 seconds for prediction.\n" ] } ], "source": [ "# number of top score elements to be recommended\n", "K = 10\n", "\n", "# Model prediction on the train set Xtr.\n", "#\n", "# The input of the model must be the ratings we are allowed to observe, i.e. the train set.\n", "# The model then infers the ratings of everything else, and the held-out ratings Xtst are\n", "# used only as the ground truth we score those recommendations against. Feeding Xtst here\n", "# instead would hand the model the answers, and the metrics would merely measure how well\n", "# the model echoes its own input rather than how well it recommends.\n", "with Timer() as prediction_time:\n", " top_k = model.recommend_k_items(Xtr)\n", "\n", "print(f\"Took {prediction_time.interval:.2f} seconds for prediction.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`top_k` returns the first K elements having the highest recommendation score. The score of an item is its expected rating under the learned distribution, $\\sum_l l \\, P(v=l|h)$, and items the user has already rated in the input are excluded. In order to inspect the prediction and use the evaluation metrics in this repository, we convert both top_k and Xtst to pandas dataframe format:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:34.807898Z", "iopub.status.busy": "2026-08-07T06:48:34.807703Z", "iopub.status.idle": "2026-08-07T06:48:34.979453Z", "shell.execute_reply": "2026-08-07T06:48:34.978583Z" } }, "outputs": [], "source": [ "top_k_df = am.map_back_sparse(top_k, kind=\"prediction\")\n", "test_df = am.map_back_sparse(Xtst, kind=\"ratings\")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:34.982763Z", "iopub.status.busy": "2026-08-07T06:48:34.982568Z", "iopub.status.idle": "2026-08-07T06:48:35.020609Z", "shell.execute_reply": "2026-08-07T06:48:35.019694Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIDmovieIDprediction
012094.794805
111004.776519
212854.830676
314084.824455
414834.693295
515114.651228
616574.732646
716544.650139
816034.703500
919214.659908
\n", "
" ], "text/plain": [ " userID movieID prediction\n", "0 1 209 4.794805\n", "1 1 100 4.776519\n", "2 1 285 4.830676\n", "3 1 408 4.824455\n", "4 1 483 4.693295\n", "5 1 511 4.651228\n", "6 1 657 4.732646\n", "7 1 654 4.650139\n", "8 1 603 4.703500\n", "9 1 921 4.659908" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "top_k_df.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4 Ranking metrics\n", "\n", "Here we evaluate the performance of the algorithm using the metrics provided in the `python_evaluation` module. Note that the following metrics take into account only the first K elements, therefore their value may be different from the one displayed from the `model.fit()` method. " ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.023799Z", "iopub.status.busy": "2026-08-07T06:48:35.023604Z", "iopub.status.idle": "2026-08-07T06:48:35.219850Z", "shell.execute_reply": "2026-08-07T06:48:35.218943Z" }, "tags": [ "ranking" ] }, "outputs": [], "source": [ "eval_map = map_at_k(\n", " test_df,\n", " top_k_df,\n", " col_user=\"userID\",\n", " col_item=\"movieID\",\n", " col_rating=\"rating\",\n", " col_prediction=\"prediction\",\n", " relevancy_method=\"top_k\",\n", " k=K,\n", ")\n", "\n", "eval_ndcg = ndcg_at_k(\n", " test_df,\n", " top_k_df,\n", " col_user=\"userID\",\n", " col_item=\"movieID\",\n", " col_rating=\"rating\",\n", " col_prediction=\"prediction\",\n", " relevancy_method=\"top_k\",\n", " k=K,\n", ")\n", "\n", "eval_precision = precision_at_k(\n", " test_df,\n", " top_k_df,\n", " col_user=\"userID\",\n", " col_item=\"movieID\",\n", " col_rating=\"rating\",\n", " col_prediction=\"prediction\",\n", " relevancy_method=\"top_k\",\n", " k=K,\n", ")\n", "\n", "eval_recall = recall_at_k(\n", " test_df,\n", " top_k_df,\n", " col_user=\"userID\",\n", " col_item=\"movieID\",\n", " col_rating=\"rating\",\n", " col_prediction=\"prediction\",\n", " relevancy_method=\"top_k\",\n", " k=K,\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.223081Z", "iopub.status.busy": "2026-08-07T06:48:35.222885Z", "iopub.status.idle": "2026-08-07T06:48:35.257680Z", "shell.execute_reply": "2026-08-07T06:48:35.256892Z" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model:\n", "Top K:\t\t 10\n", "MAP:\t\t 0.062163\n", "NDCG:\t\t 0.124090\n", "Precision@K:\t 0.110710\n", "Recall@K:\t 0.039830\n" ] } ], "source": [ "print(\n", " \"Model:\",\n", " f\"Top K:\\t\\t {K}\",\n", " f\"MAP:\\t\\t {eval_map:f}\",\n", " f\"NDCG:\\t\\t {eval_ndcg:f}\",\n", " f\"Precision@K:\\t {eval_precision:f}\",\n", " f\"Recall@K:\\t {eval_recall:f}\",\n", " sep=\"\\n\",\n", ")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.260965Z", "iopub.status.busy": "2026-08-07T06:48:35.260779Z", "iopub.status.idle": "2026-08-07T06:48:35.297208Z", "shell.execute_reply": "2026-08-07T06:48:35.296330Z" } }, "outputs": [ { "data": { "application/notebook_utils.json+json": { "data": 0.062163266508441485, "encoder": "json", "name": "map" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "map" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.12408980316962735, "encoder": "json", "name": "ndcg" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "ndcg" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.11071049840933193, "encoder": "json", "name": "precision" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "precision" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.0398300704213902, "encoder": "json", "name": "recall" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "recall" } }, "output_type": "display_data" } ], "source": [ "# Record results for tests - ignore this cell\n", "store_metadata(\"map\", eval_map)\n", "store_metadata(\"ndcg\", eval_ndcg)\n", "store_metadata(\"precision\", eval_precision)\n", "store_metadata(\"recall\", eval_recall)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5 Rating prediction\n", "\n", "The metrics above measure the quality of a *ranking*. The task the original paper evaluates is instead *rating prediction*: given the ratings a user has already given, predict the ratings they gave to the held-out movies and measure the error. `predict()` returns the expected rating $\\sum_l l \\, P(v=l|h)$ for every user/item pair, which is the estimator that minimises exactly this error.\n", "\n", "Here we use the rating metrics provided by the `python_evaluation` module: RMSE, MAE, R$^2$ and explained variance. They compare two dataframes of (user, item, rating) pairs, so we restrict the predicted matrix to the held-out entries and map it back to a dataframe with the same `AffinityMatrix` used to build the data.\n", "\n", "As a sanity check we compare the model against two trivial predictors: always predicting the global average rating, and predicting the average rating of each movie. A collaborative filtering model that does not beat the movie average is not learning anything useful about the users." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.300795Z", "iopub.status.busy": "2026-08-07T06:48:35.300572Z", "iopub.status.idle": "2026-08-07T06:48:35.891448Z", "shell.execute_reply": "2026-08-07T06:48:35.890247Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RBM:\n", "RMSE:\t\t\t 0.949651\n", "MAE:\t\t\t 0.751078\n", "R2:\t\t\t 0.290832\n", "Explained variance:\t 0.290849\n", "\n", "RMSE of the trivial predictors:\n", "global average:\t\t 1.127699\n", "movie average:\t\t 1.025201\n" ] } ], "source": [ "# Inferred ratings for every user/item pair, given the observed (train) ratings\n", "predictions = model.predict(Xtr)\n", "\n", "# the error is evaluated on the held-out ratings only\n", "test_mask = Xtst > 0\n", "\n", "\n", "def prediction_df(pred):\n", " \"\"\"Keep the held-out entries of a prediction matrix and map them back to a dataframe.\"\"\"\n", " return am.map_back_sparse(pred * test_mask, kind=\"prediction\")\n", "\n", "\n", "# trivial baselines, computed on the train set\n", "observed = Xtr > 0\n", "global_mean = Xtr[observed].mean()\n", "item_counts = observed.sum(axis=0)\n", "item_mean = np.where(\n", " item_counts > 0, Xtr.sum(axis=0) / np.maximum(item_counts, 1), global_mean\n", ")\n", "\n", "predictions_df = prediction_df(predictions)\n", "global_mean_df = prediction_df(np.full(Xtst.shape, global_mean))\n", "item_mean_df = prediction_df(np.broadcast_to(item_mean, Xtst.shape))\n", "\n", "eval_rmse = rmse(test_df, predictions_df, **header)\n", "eval_mae = mae(test_df, predictions_df, **header)\n", "eval_rsquared = rsquared(test_df, predictions_df, **header)\n", "eval_exp_var = exp_var(test_df, predictions_df, **header)\n", "\n", "print(\n", " \"RBM:\",\n", " f\"RMSE:\\t\\t\\t {eval_rmse:f}\",\n", " f\"MAE:\\t\\t\\t {eval_mae:f}\",\n", " f\"R2:\\t\\t\\t {eval_rsquared:f}\",\n", " f\"Explained variance:\\t {eval_exp_var:f}\",\n", " \"\",\n", " \"RMSE of the trivial predictors:\",\n", " f\"global average:\\t\\t {rmse(test_df, global_mean_df, **header):f}\",\n", " f\"movie average:\\t\\t {rmse(test_df, item_mean_df, **header):f}\",\n", " sep=\"\\n\",\n", ")" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.894538Z", "iopub.status.busy": "2026-08-07T06:48:35.894357Z", "iopub.status.idle": "2026-08-07T06:48:35.932248Z", "shell.execute_reply": "2026-08-07T06:48:35.931189Z" } }, "outputs": [ { "data": { "application/notebook_utils.json+json": { "data": 0.9496509209153413, "encoder": "json", "name": "rmse" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "rmse" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.7510775836686331, "encoder": "json", "name": "mae" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "mae" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.2908321225330761, "encoder": "json", "name": "rsquared" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "rsquared" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.2908490056242041, "encoder": "json", "name": "exp_var" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "exp_var" } }, "output_type": "display_data" } ], "source": [ "# Record results for tests - ignore this cell\n", "store_metadata(\"rmse\", eval_rmse)\n", "store_metadata(\"mae\", eval_mae)\n", "store_metadata(\"rsquared\", eval_rsquared)\n", "store_metadata(\"exp_var\", eval_exp_var)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6 Saving the model and Loading a pre-trained model\n", "Trained model checkpoint can be saved to a specified directory using the `save` function." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.935342Z", "iopub.status.busy": "2026-08-07T06:48:35.935159Z", "iopub.status.idle": "2026-08-07T06:48:35.989109Z", "shell.execute_reply": "2026-08-07T06:48:35.988056Z" } }, "outputs": [], "source": [ "model.save(file_path=\"./models/rbm_model.pt\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pre-trained RBM model can be loaded using the `load` function, which can be used to resume the training." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2026-08-07T06:48:35.992755Z", "iopub.status.busy": "2026-08-07T06:48:35.992568Z", "iopub.status.idle": "2026-08-07T06:48:36.072914Z", "shell.execute_reply": "2026-08-07T06:48:36.071840Z" } }, "outputs": [], "source": [ "# Initialize the model class\n", "model = RBM(\n", " possible_ratings=np.setdiff1d(np.unique(Xtr), np.array([0])),\n", " visible_units=Xtr.shape[1],\n", " hidden_units=200,\n", ")\n", "\n", "# Load the model checkpoint\n", "model.load(file_path=\"./models/rbm_model.pt\")" ] } ], "metadata": { "celltoolbar": "Tags", "interpreter": { "hash": "67434505f7f08e5031eee7757e853265d2f43dd6b5963eb755a27835ec0e1503" }, "kernel_info": { "name": "python3" }, "kernelspec": { "display_name": "Python (recommenders)", "language": "python", "name": "recommenders" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" }, "nteract": { "version": "0.12.3" } }, "nbformat": 4, "nbformat_minor": 4 }