{ "cells": [ { "cell_type": "markdown", "id": "9a197a09", "metadata": { "papermill": { "duration": 0.011196, "end_time": "2026-05-25T20:08:01.498080", "exception": false, "start_time": "2026-05-25T20:08:01.486884", "status": "completed" }, "tags": [] }, "source": [ "Copyright (c) Recommenders contributors.\n", "\n", "Licensed under the MIT License." ] }, { "cell_type": "markdown", "id": "942b2788", "metadata": { "papermill": { "duration": 0.004778, "end_time": "2026-05-25T20:08:01.516768", "exception": false, "start_time": "2026-05-25T20:08:01.511990", "status": "completed" }, "tags": [] }, "source": [ "# Benchmark with Movielens dataset\n", "\n", "This illustrative comparison applies to collaborative filtering algorithms available in this repository such as Spark ALS, SAR and others using the Movielens dataset. These algorithms are usable in a variety of recommendation tasks, including product or news recommendations.\n", "\n", "The main purpose of this notebook is not to produce comprehensive benchmarking results on multiple datasets. Rather, it is intended to illustrate on how one could evaluate different recommender algorithms using tools in this repository.\n", "\n", "## Experimentation setup:\n", "\n", "* Objective:\n", " * To compare how each collaborative filtering algorithm perform in predicting ratings and recommending relevant items.\n", "\n", "* Environment:\n", " * The comparison is run on a machine with 24 CPUs, 30Gb of RAM, and 1 GPU GeForce RTX 5090 GPU with 24Gb of memory.\n", " * It should be noted that a local machine is not supposed to run scalable benchmarking analysis. Either scaling up or out the computing instances is necessary to run the benchmarking in an run-time efficient way without any memory issue.\n", " * **NOTE ABOUT THE DEPENDENCIES TO INSTALL**: This notebook uses CPU, GPU and PySpark algorithms, so make sure you install the `full environment` as detailed in the [SETUP.md](../../SETUP.md). \n", " \n", "* Datasets:\n", " * [Movielens 100K](https://grouplens.org/datasets/movielens/100k/).\n", " * [Movielens 1M](https://grouplens.org/datasets/movielens/1m/).\n", "\n", "* Data split:\n", " * The data is split into train and test sets.\n", " * The split ratios are 75-25 for train and test datasets.\n", " * The splitting is stratified based on items. \n", "\n", "* Model training:\n", " * A recommendation model is trained by using each of the collaborative filtering algorithms. \n", " * Empirical parameter values reported [here](http://mymedialite.net/examples/datasets.html) are used in this notebook. More exhaustive hyper parameter tuning would be required to further optimize results.\n", "\n", "* Evaluation metrics:\n", " * Ranking metrics:\n", " * Precision@k.\n", " * Recall@k.\n", " * Normalized discounted cumulative gain@k (NDCG@k).\n", " * Mean-average-precision (MAP). \n", " * In the evaluation metrics above, k = 10. \n", " * Rating metrics:\n", " * Root mean squared error (RMSE).\n", " * Mean average error (MAE).\n", " * R squared.\n", " * Explained variance.\n", " * Run time performance\n", " * Elapsed for training a model and using a model for predicting/recommending k items. \n", " * The time may vary across different machines. " ] }, { "cell_type": "markdown", "id": "5b5d87ed", "metadata": { "papermill": { "duration": 0.004757, "end_time": "2026-05-25T20:08:01.526802", "exception": false, "start_time": "2026-05-25T20:08:01.522045", "status": "completed" }, "tags": [] }, "source": [ "## Globals settings" ] }, { "cell_type": "code", "execution_count": 1, "id": "94ec3aa1", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:31.273854Z", "iopub.status.busy": "2026-07-13T14:09:31.273475Z", "iopub.status.idle": "2026-07-13T14:09:31.294194Z", "shell.execute_reply": "2026-07-13T14:09:31.288908Z" }, "papermill": { "duration": 0.02636, "end_time": "2026-05-25T20:08:01.556677", "exception": false, "start_time": "2026-05-25T20:08:01.530317", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "# Remove warnings\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "import os\n", "os.environ[\"PYTHONWARNINGS\"] = \"ignore\"\n", "os.environ[\"SPARK_LOCAL_IP\"] = \"127.0.0.1\" # Set local IP to avoid hostname warnings\n", "import logging\n", "logging.basicConfig(level=logging.ERROR)\n", "logging.getLogger(\"py4j\").setLevel(logging.ERROR)\n", "logging.getLogger(\"pyspark\").setLevel(logging.ERROR)" ] }, { "cell_type": "code", "execution_count": 2, "id": "173762ab", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:31.299751Z", "iopub.status.busy": "2026-07-13T14:09:31.298834Z", "iopub.status.idle": "2026-07-13T14:09:45.187548Z", "shell.execute_reply": "2026-07-13T14:09:45.184268Z" }, "papermill": { "duration": 9.722963, "end_time": "2026-05-25T20:08:11.284149", "exception": false, "start_time": "2026-05-25T20:08:01.561186", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "System version: 3.11.14 (main, Jan 14 2026, 19:35:32) [Clang 21.1.4 ]\n", "Number of cores: 24\n", "NumPy version: 1.26.4\n", "Pandas version: 2.3.3\n", "Cornac version: 2.3.0\n", "PySpark version: 3.5.8\n", "CUDA version: 13.2\n", "CuDNN version: 92000\n", "PyTorch version: 2.13.0.dev20260521+cu132\n" ] } ], "source": [ "import sys\n", "import numpy as np\n", "import pandas as pd\n", "import cornac\n", "\n", "try:\n", " import pyspark\n", " from recommenders.utils.spark_utils import start_or_get_spark\n", "except ImportError:\n", " pass # skip this import if we are not in a Spark environment\n", "\n", "try:\n", " import torch\n", " from recommenders.utils.gpu_utils import get_cuda_version, get_cudnn_version\n", "except ImportError:\n", " pass # skip this import if we are not in a GPU environment\n", "\n", "current_path = os.path.join(os.getcwd(), \"examples\", \"06_benchmarks\") # To execute the notebook programmatically from root folder\n", "sys.path.append(current_path)\n", "from benchmark_utils import * \n", "from recommenders.datasets import movielens\n", "from recommenders.utils.general_utils import get_number_processors\n", "from recommenders.datasets.python_splitters import python_stratified_split\n", "from recommenders.utils.notebook_utils import store_metadata\n", "\n", "\n", "print(f\"System version: {sys.version}\")\n", "print(f\"Number of cores: {get_number_processors()}\")\n", "print(f\"NumPy version: {np.__version__}\")\n", "print(f\"Pandas version: {pd.__version__}\")\n", "print(f\"Cornac version: {cornac.__version__}\")\n", "\n", "try:\n", " print(f\"PySpark version: {pyspark.__version__}\")\n", "except NameError:\n", " pass # skip this import if we are not in a Spark environment\n", "\n", "try:\n", " print(f\"CUDA version: {get_cuda_version()}\")\n", " print(f\"CuDNN version: {get_cudnn_version()}\")\n", " print(f\"PyTorch version: {torch.__version__}\")\n", "except NameError:\n", " pass # skip this import if we are not in a GPU environment\n", "\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": 3, "id": "e4ae65b8", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:45.192530Z", "iopub.status.busy": "2026-07-13T14:09:45.191951Z", "iopub.status.idle": "2026-07-13T14:09:52.550988Z", "shell.execute_reply": "2026-07-13T14:09:52.547169Z" }, "papermill": { "duration": 7.139913, "end_time": "2026-05-25T20:08:18.428879", "exception": false, "start_time": "2026-05-25T20:08:11.288966", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Setting default log level to \"WARN\".\n", "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "26/07/13 16:09:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" ] } ], "source": [ "try:\n", " spark = start_or_get_spark(\"PySpark\", memory=\"32g\")\n", " spark.conf.set(\"spark.sql.analyzer.failAmbiguousSelfJoin\", \"false\")\n", " # Suppress Spark warnings\n", " spark.sparkContext.setLogLevel(\"ERROR\") \n", " log4j = spark._jvm.org.apache.log4j\n", " log4j.LogManager.getLogger(\"org\").setLevel(log4j.Level.ERROR)\n", " log4j.LogManager.getLogger(\"akka\").setLevel(log4j.Level.ERROR)\n", " log4j.LogManager.getLogger(\"org.apache.spark\").setLevel(log4j.Level.ERROR)\n", " log4j.LogManager.getLogger(\"org.spark_project\").setLevel(log4j.Level.ERROR)\n", "except NameError:\n", " pass # skip this import if we are not in a Spark environment" ] }, { "cell_type": "code", "execution_count": 4, "id": "e24d302c", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.556911Z", "iopub.status.busy": "2026-07-13T14:09:52.556251Z", "iopub.status.idle": "2026-07-13T14:09:52.603631Z", "shell.execute_reply": "2026-07-13T14:09:52.600618Z" }, "papermill": { "duration": 0.065061, "end_time": "2026-05-25T20:08:18.501342", "exception": false, "start_time": "2026-05-25T20:08:18.436281", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "# Fix random seeds to make sure out runs are reproducible\n", "np.random.seed(SEED)\n", "try:\n", " torch.manual_seed(SEED)\n", " torch.cuda.manual_seed_all(SEED)\n", "except NameError:\n", " pass # skip this import if we are not in a GPU environment" ] }, { "cell_type": "markdown", "id": "de46f11c", "metadata": { "papermill": { "duration": 0.011048, "end_time": "2026-05-25T20:08:18.516997", "exception": false, "start_time": "2026-05-25T20:08:18.505949", "status": "completed" }, "tags": [] }, "source": [ "## Parameters" ] }, { "cell_type": "code", "execution_count": 5, "id": "1be0289c", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.609468Z", "iopub.status.busy": "2026-07-13T14:09:52.608720Z", "iopub.status.idle": "2026-07-13T14:09:52.648661Z", "shell.execute_reply": "2026-07-13T14:09:52.645029Z" }, "papermill": { "duration": 0.050508, "end_time": "2026-05-25T20:08:18.572160", "exception": false, "start_time": "2026-05-25T20:08:18.521652", "status": "completed" }, "tags": [ "parameters" ] }, "outputs": [], "source": [ "data_sizes = [\"100k\"] # Movielens data size: 100k, 1m, 10m, or 20m\n", "algorithms = [\"als\", \"sar\", \"ncf\", \"embdotbias\", \"bpr\", \"bivae\", \"lightgcn\"]\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "39803d33", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.653432Z", "iopub.status.busy": "2026-07-13T14:09:52.653018Z", "iopub.status.idle": "2026-07-13T14:09:52.694649Z", "shell.execute_reply": "2026-07-13T14:09:52.690823Z" }, "papermill": { "duration": 0.061897, "end_time": "2026-05-25T20:08:18.639880", "exception": false, "start_time": "2026-05-25T20:08:18.577983", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "environments = {\n", " \"als\": \"pyspark\",\n", " \"sar\": \"python_cpu\",\n", " \"embdotbias\": \"python_gpu\",\n", " \"ncf\": \"python_gpu\",\n", " \"bpr\": \"python_cpu\",\n", " \"bivae\": \"python_gpu\",\n", " \"lightgcn\": \"python_gpu\",\n", "}\n", "\n", "metrics = {\n", " \"als\": [\"rating\", \"ranking\"],\n", " \"sar\": [\"rating\", \"ranking\"],\n", " \"embdotbias\": [\"rating\", \"ranking\"],\n", " \"ncf\": [\"ranking\"],\n", " \"bpr\": [\"ranking\"],\n", " \"bivae\": [\"ranking\"],\n", " \"lightgcn\": [\"ranking\"]\n", "}" ] }, { "cell_type": "markdown", "id": "1b382221", "metadata": { "papermill": { "duration": 0.010504, "end_time": "2026-05-25T20:08:18.656086", "exception": false, "start_time": "2026-05-25T20:08:18.645582", "status": "completed" }, "tags": [] }, "source": [ "Algorithm parameters" ] }, { "cell_type": "code", "execution_count": 7, "id": "3dbc6bfe", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.699991Z", "iopub.status.busy": "2026-07-13T14:09:52.699477Z", "iopub.status.idle": "2026-07-13T14:09:52.764495Z", "shell.execute_reply": "2026-07-13T14:09:52.760137Z" }, "papermill": { "duration": 0.055274, "end_time": "2026-05-25T20:08:18.715513", "exception": false, "start_time": "2026-05-25T20:08:18.660239", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "als_params = {\n", " \"rank\": 10,\n", " \"maxIter\": 20,\n", " \"implicitPrefs\": False,\n", " \"alpha\": 0.1,\n", " \"regParam\": 0.05,\n", " \"coldStartStrategy\": \"drop\",\n", " \"nonnegative\": False,\n", " \"userCol\": DEFAULT_USER_COL,\n", " \"itemCol\": DEFAULT_ITEM_COL,\n", " \"ratingCol\": DEFAULT_RATING_COL,\n", " \"seed\": SEED,\n", "}\n", "\n", "sar_params = {\n", " \"similarity_type\": \"jaccard\",\n", " \"time_decay_coefficient\": 30,\n", " \"time_now\": None,\n", " \"timedecay_formula\": True,\n", " \"col_user\": DEFAULT_USER_COL,\n", " \"col_item\": DEFAULT_ITEM_COL,\n", " \"col_rating\": DEFAULT_RATING_COL,\n", " \"col_timestamp\": DEFAULT_TIMESTAMP_COL,\n", " \"normalize\": True,\n", "}\n", "\n", "embdotbias_params = {\n", " \"n_factors\": 40, \n", " \"y_range\": [0,5.5], \n", " \"wd\": 1e-1,\n", " \"lr_max\": 5e-3,\n", " \"epochs\": 15\n", "}\n", "\n", "ncf_params = {\n", " \"model_type\": \"NeuMF\",\n", " \"n_factors\": 4,\n", " \"layer_sizes\": [16, 8, 4],\n", " \"n_epochs\": 15,\n", " \"batch_size\": 1024,\n", " \"learning_rate\": 1e-3,\n", " \"verbose\": 10,\n", " \"seed\": SEED,\n", "}\n", "\n", "bpr_params = {\n", " \"k\": 200,\n", " \"max_iter\": 200,\n", " \"learning_rate\": 0.01,\n", " \"lambda_reg\": 1e-3,\n", " \"seed\": SEED,\n", " \"verbose\": False\n", "}\n", "\n", "bivae_params = {\n", " \"k\": 100,\n", " \"encoder_structure\": [200],\n", " \"act_fn\": \"tanh\",\n", " \"likelihood\": \"pois\",\n", " \"n_epochs\": 500,\n", " \"batch_size\": 1024,\n", " \"learning_rate\": 0.001,\n", " \"seed\": SEED,\n", " \"use_gpu\": True,\n", " \"verbose\": False\n", "}\n", "\n", "lightgcn_param = {\n", " \"model_type\": \"lightgcn\",\n", " \"n_layers\": 3,\n", " \"batch_size\": 1024,\n", " \"embed_size\": 64,\n", " \"decay\": 0.0001,\n", " \"epochs\": 20,\n", " \"learning_rate\": 0.005,\n", " \"eval_epoch\": 5,\n", " \"top_k\": DEFAULT_K,\n", " \"metrics\": [\"recall\", \"ndcg\", \"precision\", \"map\"],\n", " \"save_model\":False,\n", " \"MODEL_DIR\":\".\",\n", " \"seed\": SEED,\n", "}\n", "\n", "params = {\n", " \"als\": als_params,\n", " \"sar\": sar_params,\n", " \"embdotbias\": embdotbias_params,\n", " \"ncf\": ncf_params,\n", " \"bpr\": bpr_params,\n", " \"bivae\": bivae_params,\n", " \"lightgcn\": lightgcn_param,\n", "}" ] }, { "cell_type": "code", "execution_count": 8, "id": "fe5de221", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.772237Z", "iopub.status.busy": "2026-07-13T14:09:52.771761Z", "iopub.status.idle": "2026-07-13T14:09:52.818463Z", "shell.execute_reply": "2026-07-13T14:09:52.814832Z" }, "papermill": { "duration": 0.063927, "end_time": "2026-05-25T20:08:18.784161", "exception": false, "start_time": "2026-05-25T20:08:18.720234", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "prepare_training_data = {\n", " \"als\": prepare_training_als,\n", " \"sar\": prepare_training_sar,\n", " \"embdotbias\": prepare_training_embdotbias,\n", " \"ncf\": prepare_training_ncf,\n", " \"bpr\": prepare_training_bpr,\n", " \"bivae\": prepare_training_cornac,\n", " \"lightgcn\": prepare_training_lightgcn,\n", "}" ] }, { "cell_type": "code", "execution_count": 9, "id": "3b7668ca", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.823499Z", "iopub.status.busy": "2026-07-13T14:09:52.823022Z", "iopub.status.idle": "2026-07-13T14:09:52.860148Z", "shell.execute_reply": "2026-07-13T14:09:52.857196Z" }, "papermill": { "duration": 0.057411, "end_time": "2026-05-25T20:08:18.845836", "exception": false, "start_time": "2026-05-25T20:08:18.788425", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "prepare_metrics_data = {\n", " \"als\": lambda train, test: prepare_metrics_als(train, test),\n", " \"embdotbias\": lambda train, test: prepare_metrics_embdotbias(train, test), \n", " \"bpr\": lambda train, test: prepare_metrics_bpr(train, test),\n", "}" ] }, { "cell_type": "code", "execution_count": 10, "id": "5d0683ad", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.865287Z", "iopub.status.busy": "2026-07-13T14:09:52.864955Z", "iopub.status.idle": "2026-07-13T14:09:52.909177Z", "shell.execute_reply": "2026-07-13T14:09:52.906267Z" }, "papermill": { "duration": 0.054007, "end_time": "2026-05-25T20:08:18.904358", "exception": false, "start_time": "2026-05-25T20:08:18.850351", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "trainer = {\n", " \"als\": lambda params, data: train_als(params, data),\n", " \"sar\": lambda params, data: train_sar(params, data), \n", " \"embdotbias\": lambda params, data: train_embdotbias(params, data),\n", " \"ncf\": lambda params, data: train_ncf(params, data),\n", " \"bpr\": lambda params, data: train_bpr(params, data),\n", " \"bivae\": lambda params, data: train_bivae(params, data),\n", " \"lightgcn\": lambda params, data: train_lightgcn(params, data),\n", "}" ] }, { "cell_type": "code", "execution_count": 11, "id": "e6567f34", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.913534Z", "iopub.status.busy": "2026-07-13T14:09:52.913163Z", "iopub.status.idle": "2026-07-13T14:09:52.957161Z", "shell.execute_reply": "2026-07-13T14:09:52.952857Z" }, "papermill": { "duration": 0.058662, "end_time": "2026-05-25T20:08:18.968534", "exception": false, "start_time": "2026-05-25T20:08:18.909872", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "rating_predictor = {\n", " \"als\": lambda model, test: predict_als(model, test),\n", " \"sar\": lambda model, test: predict_sar(model, test),\n", " \"embdotbias\": lambda model, test: predict_embdotbias(model, test),\n", "}" ] }, { "cell_type": "code", "execution_count": 12, "id": "c18a1c60", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:52.962034Z", "iopub.status.busy": "2026-07-13T14:09:52.961514Z", "iopub.status.idle": "2026-07-13T14:09:53.002600Z", "shell.execute_reply": "2026-07-13T14:09:52.999224Z" }, "papermill": { "duration": 0.05443, "end_time": "2026-05-25T20:08:19.030727", "exception": false, "start_time": "2026-05-25T20:08:18.976297", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "ranking_predictor = {\n", " \"als\": lambda model, test, train: recommend_k_als(model, test, train),\n", " \"sar\": lambda model, test, train: recommend_k_sar(model, test, train),\n", " \"embdotbias\": lambda model, test, train: recommend_k_embdotbias(model, test, train),\n", " \"ncf\": lambda model, test, train: recommend_k_ncf(model, test, train),\n", " \"bpr\": lambda model, test, train: recommend_k_bpr(model, test, train),\n", " \"bivae\": lambda model, test, train: recommend_k_bivae(model, test, train),\n", " \"lightgcn\": lambda model, test, train: recommend_k_lightgcn(model, test, train),\n", "}" ] }, { "cell_type": "code", "execution_count": 13, "id": "c28c8f24", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:53.007495Z", "iopub.status.busy": "2026-07-13T14:09:53.007047Z", "iopub.status.idle": "2026-07-13T14:09:53.054833Z", "shell.execute_reply": "2026-07-13T14:09:53.051892Z" }, "papermill": { "duration": 0.057779, "end_time": "2026-05-25T20:08:19.092771", "exception": false, "start_time": "2026-05-25T20:08:19.034992", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "rating_evaluator = {\n", " \"als\": lambda test, predictions: rating_metrics_pyspark(test, predictions),\n", " \"sar\": lambda test, predictions: rating_metrics_python(test, predictions),\n", " \"embdotbias\": lambda test, predictions: rating_metrics_python(test, predictions)\n", "}\n", " \n", " \n", "ranking_evaluator = {\n", " \"als\": lambda test, predictions, k: ranking_metrics_pyspark(test, predictions, k),\n", " \"sar\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n", " \"embdotbias\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n", " \"ncf\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n", " \"bpr\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n", " \"bivae\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n", " \"lightgcn\": lambda test, predictions, k: ranking_metrics_python(test, predictions, k),\n", "}" ] }, { "cell_type": "code", "execution_count": 14, "id": "7844fef3", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:53.059300Z", "iopub.status.busy": "2026-07-13T14:09:53.058915Z", "iopub.status.idle": "2026-07-13T14:09:53.109260Z", "shell.execute_reply": "2026-07-13T14:09:53.106104Z" }, "papermill": { "duration": 0.062643, "end_time": "2026-05-25T20:08:19.160696", "exception": false, "start_time": "2026-05-25T20:08:19.098053", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "def generate_summary(data, algo, k, train_time, time_rating, rating_metrics, time_ranking, ranking_metrics):\n", " summary = {\"Data\": data, \"Algo\": algo, \"K\": k, \"Train time (s)\": train_time, \"Predicting time (s)\": time_rating, \"Recommending time (s)\": time_ranking}\n", " if rating_metrics is None:\n", " rating_metrics = {\n", " \"RMSE\": np.nan,\n", " \"MAE\": np.nan,\n", " \"R2\": np.nan,\n", " \"Explained Variance\": np.nan,\n", " }\n", " if ranking_metrics is None:\n", " ranking_metrics = {\n", " \"MAP@k\": np.nan,\n", " \"nDCG@k\": np.nan,\n", " \"Precision@k\": np.nan,\n", " \"Recall@k\": np.nan,\n", " }\n", " summary.update(rating_metrics)\n", " summary.update(ranking_metrics)\n", " return summary" ] }, { "cell_type": "markdown", "id": "a9c1e827", "metadata": { "papermill": { "duration": 0.010834, "end_time": "2026-05-25T20:08:19.177607", "exception": false, "start_time": "2026-05-25T20:08:19.166773", "status": "completed" }, "tags": [] }, "source": [ "## Benchmark loop" ] }, { "cell_type": "code", "execution_count": 15, "id": "311ae038", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:09:53.114375Z", "iopub.status.busy": "2026-07-13T14:09:53.113928Z", "iopub.status.idle": "2026-07-13T14:14:20.253048Z", "shell.execute_reply": "2026-07-13T14:14:20.248888Z" }, "papermill": { "duration": 345.861122, "end_time": "2026-05-25T20:14:05.042768", "exception": false, "start_time": "2026-05-25T20:08:19.181646", "status": "completed" }, "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", " 0%| | 0.00/4.81k [00:00 (0 + 24) / 24]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 0:======================================> (16 + 8) / 24]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 13.3561s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Rating prediction time: 0.2430s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Ranking prediction time: 0.3328s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 577:==============>(23 + 1) / 24][Stage 578:==============>(23 + 1) / 24]\r", "\r", "[Stage 625:> (0 + 24) / 24]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 681:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 683:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 685:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 687:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 689:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computing sar algorithm on Movielens 100k\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 0.6155s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Rating prediction time: 0.2295s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Ranking prediction time: 0.2267s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computing ncf algorithm on Movielens 100k\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 70.3349s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Ranking prediction time: 11.2570s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computing embdotbias algorithm on Movielens 100k\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 77.6305s\n", "Rating prediction time: 0.0357s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Ranking prediction time: 2.0636s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computing bpr algorithm on Movielens 100k\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 3.8147s\n", "Ranking prediction time: 0.1744s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computing bivae algorithm on Movielens 100k\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 31.5243s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Ranking prediction time: 1.8528s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computing lightgcn algorithm on Movielens 100k\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training time: 28.9207s\n", "Ranking prediction time: 0.0759s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Computation finished\n", "CPU times: user 8min 24s, sys: 26.3 s, total: 8min 51s\n", "Wall time: 4min 27s\n" ] } ], "source": [ "%%time\n", "\n", "# For each data size and each algorithm, a recommender is evaluated. \n", "cols = [\"Data\", \"Algo\", \"K\", \"Train time (s)\", \"Predicting time (s)\", \"RMSE\", \"MAE\", \"R2\", \"Explained Variance\", \"Recommending time (s)\", \"MAP@k\", \"nDCG@k\", \"Precision@k\", \"Recall@k\"]\n", "df_results = pd.DataFrame(columns=cols)\n", "\n", "for data_size in data_sizes:\n", " # Load the dataset\n", " df = movielens.load_pandas_df(\n", " size=data_size,\n", " header=[DEFAULT_USER_COL, DEFAULT_ITEM_COL, DEFAULT_RATING_COL, DEFAULT_TIMESTAMP_COL]\n", " )\n", " print(\"Size of Movielens {}: {}\".format(data_size, df.shape))\n", " \n", " # Split the dataset\n", " df_train, df_test = python_stratified_split(df,\n", " ratio=0.75, \n", " min_rating=1, \n", " filter_by=\"item\", \n", " col_user=DEFAULT_USER_COL, \n", " col_item=DEFAULT_ITEM_COL,\n", " seed=SEED,\n", " )\n", " \n", " # Loop through the algos\n", " for algo in algorithms:\n", " print(f\"\\nComputing {algo} algorithm on Movielens {data_size}\")\n", " \n", " try:\n", " # Data prep for training set\n", " train = prepare_training_data.get(algo, lambda x,y:(x,y))(df_train, df_test)\n", " \n", " # Get model parameters\n", " model_params = params[algo]\n", " \n", " # Train the model\n", " model, time_train = trainer[algo](model_params, train)\n", " print(f\"Training time: {time_train}s\")\n", " \n", " # Predict and evaluate\n", " train, test = prepare_metrics_data.get(algo, lambda x,y:(x,y))(df_train, df_test)\n", " \n", " if \"rating\" in metrics[algo]: \n", " # Predict for rating\n", " preds, time_rating = rating_predictor[algo](model, test)\n", " print(f\"Rating prediction time: {time_rating}s\")\n", " \n", " # Evaluate for rating\n", " ratings = rating_evaluator[algo](test, preds)\n", " else:\n", " ratings = None\n", " time_rating = np.nan\n", " \n", " if \"ranking\" in metrics[algo]:\n", " # Predict for ranking\n", " top_k_scores, time_ranking = ranking_predictor[algo](model, test, train)\n", " print(f\"Ranking prediction time: {time_ranking}s\")\n", " \n", " # Evaluate for ranking\n", " rankings = ranking_evaluator[algo](test, top_k_scores, DEFAULT_K)\n", " else:\n", " rankings = None\n", " time_ranking = np.nan\n", " \n", " # Record results\n", " summary = generate_summary(data_size, algo, DEFAULT_K, time_train, time_rating, ratings, time_ranking, rankings)\n", " df_results.loc[df_results.shape[0] + 1] = summary\n", " except Exception as e:\n", " print(f\" ERROR: {e}\")\n", " summary = generate_summary(data_size, algo, DEFAULT_K, np.nan, np.nan, None, np.nan, None)\n", " df_results.loc[df_results.shape[0] + 1] = summary\n", " \n", "print(\"\\nComputation finished\")\n" ] }, { "cell_type": "markdown", "id": "22d050fd", "metadata": { "papermill": { "duration": 0.011693, "end_time": "2026-05-25T20:14:05.063463", "exception": false, "start_time": "2026-05-25T20:14:05.051770", "status": "completed" }, "tags": [] }, "source": [ "## Results" ] }, { "cell_type": "code", "execution_count": 16, "id": "70f0e51e", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:14:20.259058Z", "iopub.status.busy": "2026-07-13T14:14:20.258400Z", "iopub.status.idle": "2026-07-13T14:14:20.340121Z", "shell.execute_reply": "2026-07-13T14:14:20.336326Z" }, "papermill": { "duration": 0.099913, "end_time": "2026-05-25T20:14:05.171993", "exception": false, "start_time": "2026-05-25T20:14:05.072080", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DataAlgoKTrain time (s)Predicting time (s)RMSEMAER2Explained VarianceRecommending time (s)MAP@knDCG@kPrecision@kRecall@k
1100kals1013.35610.24300.9637250.7513640.2722310.2681100.33280.0113990.0369070.0430540.015254
2100ksar100.61550.22951.2291031.035996-0.4878430.0972830.22670.2584520.3938190.3406150.185377
3100kncf1070.3349NaNNaNNaNNaNNaN11.25700.2767980.4114240.3604450.188897
4100kembdotbias1077.63050.03570.9872290.7712670.2319730.2319892.06360.0549460.1189620.1069990.042326
5100kbpr103.8147NaNNaNNaNNaNNaN0.17440.2291740.3623010.2854680.251667
6100kbivae1031.5243NaNNaNNaNNaNNaN1.85280.3299620.4717220.4113470.224300
7100klightgcn1028.9207NaNNaNNaNNaNNaN0.07590.2810620.4199470.3623540.193273
\n", "
" ], "text/plain": [ " Data Algo K Train time (s) Predicting time (s) RMSE \\\n", "1 100k als 10 13.3561 0.2430 0.963725 \n", "2 100k sar 10 0.6155 0.2295 1.229103 \n", "3 100k ncf 10 70.3349 NaN NaN \n", "4 100k embdotbias 10 77.6305 0.0357 0.987229 \n", "5 100k bpr 10 3.8147 NaN NaN \n", "6 100k bivae 10 31.5243 NaN NaN \n", "7 100k lightgcn 10 28.9207 NaN NaN \n", "\n", " MAE R2 Explained Variance Recommending time (s) MAP@k \\\n", "1 0.751364 0.272231 0.268110 0.3328 0.011399 \n", "2 1.035996 -0.487843 0.097283 0.2267 0.258452 \n", "3 NaN NaN NaN 11.2570 0.276798 \n", "4 0.771267 0.231973 0.231989 2.0636 0.054946 \n", "5 NaN NaN NaN 0.1744 0.229174 \n", "6 NaN NaN NaN 1.8528 0.329962 \n", "7 NaN NaN NaN 0.0759 0.281062 \n", "\n", " nDCG@k Precision@k Recall@k \n", "1 0.036907 0.043054 0.015254 \n", "2 0.393819 0.340615 0.185377 \n", "3 0.411424 0.360445 0.188897 \n", "4 0.118962 0.106999 0.042326 \n", "5 0.362301 0.285468 0.251667 \n", "6 0.471722 0.411347 0.224300 \n", "7 0.419947 0.362354 0.193273 " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_results" ] }, { "cell_type": "code", "execution_count": 17, "id": "61a7d26e", "metadata": { "execution": { "iopub.execute_input": "2026-07-13T14:14:20.344663Z", "iopub.status.busy": "2026-07-13T14:14:20.344282Z", "iopub.status.idle": "2026-07-13T14:14:20.420535Z", "shell.execute_reply": "2026-07-13T14:14:20.417662Z" }, "papermill": { "duration": 0.098009, "end_time": "2026-05-25T20:14:05.278251", "exception": false, "start_time": "2026-05-25T20:14:05.180242", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "application/notebook_utils.json+json": { "data": 0.03690703376630467, "encoder": "json", "name": "als" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "als" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.3938185694234946, "encoder": "json", "name": "sar" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "sar" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.4114242141639425, "encoder": "json", "name": "ncf" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "ncf" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.1189617743709156, "encoder": "json", "name": "embdotbias" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "embdotbias" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.3623011798298851, "encoder": "json", "name": "bpr" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "bpr" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.4717219365198055, "encoder": "json", "name": "bivae" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "bivae" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.41994665586993424, "encoder": "json", "name": "lightgcn" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "lightgcn" } }, "output_type": "display_data" } ], "source": [ "# Record results for tests - ignore this cell\n", "for algo in algorithms:\n", " store_metadata(algo, df_results.loc[df_results[\"Algo\"] == algo, \"nDCG@k\"].values[0])\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.14" }, "papermill": { "default_parameters": {}, "duration": 368.536939, "end_time": "2026-05-25T20:14:08.235046", "environment_variables": {}, "exception": null, "input_path": "examples/06_benchmarks/movielens.ipynb", "output_path": "examples/06_benchmarks/movielens.ipynb", "parameters": {}, "start_time": "2026-05-25T20:07:59.698107", "version": "2.6.0" } }, "nbformat": 4, "nbformat_minor": 5 }