{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright (c) Recommenders contributors.\n", "\n", "Licensed under the MIT License." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# LightGBM: A Highly Efficient Gradient Boosting Decision Tree\n", "This notebook will give you an example of how to train a LightGBM model to estimate click-through rates on an e-commerce advertisement. We will train a LightGBM based model on the Criteo dataset.\n", "\n", "[LightGBM](https://github.com/Microsoft/LightGBM) is a gradient boosting framework that uses tree-based learning algorithms. It is designed to be distributed and efficient with the following advantages:\n", "* Fast training speed and high efficiency.\n", "* Low memory usage.\n", "* Great accuracy.\n", "* Support of parallel and GPU learning.\n", "* Capable of handling large-scale data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Global Settings and Imports" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "System version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]\n", "LightGBM version: 4.4.0\n" ] } ], "source": [ "import os\n", "import sys\n", "import numpy as np\n", "import lightgbm as lgb\n", "import category_encoders as ce\n", "from tempfile import TemporaryDirectory\n", "from sklearn.metrics import roc_auc_score, log_loss\n", "\n", "import recommenders.datasets.criteo as criteo\n", "import recommenders.models.lightgbm.lightgbm_utils as lgb_utils\n", "from recommenders.utils.notebook_utils import store_metadata\n", "\n", "print(\"System version: {}\".format(sys.version))\n", "print(\"LightGBM version: {}\".format(lgb.__version__))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parameter Setting\n", "Let's set the main related parameters for LightGBM now. Basically, the task is a binary classification (predicting click or no click), so the objective function is set to binary logloss, and 'AUC' metric, is used as a metric which is less effected by imbalance in the classes of the dataset.\n", "\n", "Generally, we can adjust the number of leaves (MAX_LEAF), the minimum number of data in each leaf (MIN_DATA), maximum number of trees (NUM_OF_TREES), the learning rate of trees (TREE_LEARNING_RATE) and EARLY_STOPPING_ROUNDS (to avoid overfitting) in the model to get better performance.\n", "\n", "Besides, we can also adjust some other listed parameters to optimize the results. [In this link](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst), a list of all the parameters is shown. Also, some advice on how to tune these parameters can be found [in this url](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters-Tuning.rst). " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [ "parameters" ] }, "outputs": [], "source": [ "MAX_LEAF = 64\n", "MIN_DATA = 20\n", "NUM_OF_TREES = 100\n", "TREE_LEARNING_RATE = 0.15\n", "EARLY_STOPPING_ROUNDS = 20\n", "METRIC = \"auc\"\n", "SIZE = \"sample\"" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "params = {\n", " \"task\": \"train\",\n", " \"boosting_type\": \"gbdt\",\n", " \"num_class\": 1,\n", " \"objective\": \"binary\",\n", " \"metric\": METRIC,\n", " \"num_leaves\": MAX_LEAF,\n", " \"min_data\": MIN_DATA,\n", " \"boost_from_average\": True,\n", " # set it according to your cpu cores.\n", " \"num_threads\": 20,\n", " \"feature_fraction\": 0.8,\n", " \"learning_rate\": TREE_LEARNING_RATE,\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Preparation\n", "Here we use CSV format as the example data input. Our example data is a sample (about 100 thousand samples) from [Criteo dataset](https://www.kaggle.com/c/criteo-display-ad-challenge). The Criteo dataset is a well-known industry benchmarking dataset for developing CTR prediction models, and it's frequently adopted as evaluation dataset by research papers. The original dataset is too large for a lightweight demo, so we sample a small portion from it as a demo dataset.\n", "\n", "Specifically, there are 39 columns of features in Criteo, where 13 columns are numerical features (I1-I13) and the other 26 columns are categorical features (C1-C26)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 8.58k/8.58k [00:00<00:00, 10.4kKB/s]\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
LabelI1I2I3I4I5I6I7I8I9...C17C18C19C20C21C22C23C24C25C26
001.015.00.01382.04.015.02.0181.0...e5ba7672f54016b921ddcdc9b1252a9d07b5194cNaN3a171ecbc5c50484e8b834079727dd16
102.0044.01.0102.08.02.02.04.0...07c540c4b04e467021ddcdc95840adea60f6221eNaN3a171ecb43f13e8be8b83407731c3655
202.001.014.0767.089.04.02.0245.0...8efede7f3412118dNaNNaNe587c466ad3062eb3a171ecb3b183c5cNaNNaN
30NaN893NaNNaN4392.0NaN0.00.00.0...1e88c74f74ef3502NaNNaN6b3a5ca6NaN3a171ecb9117a34aNaNNaN
403.0-1NaN0.02.00.03.00.00.0...1e88c74f26b3c7a7NaNNaN21c9516aNaN32c7478eb34f3128NaNNaN
\n", "

5 rows × 40 columns

\n", "
" ], "text/plain": [ " Label I1 I2 I3 I4 I5 I6 I7 I8 I9 ... C17 \\\n", "0 0 1.0 1 5.0 0.0 1382.0 4.0 15.0 2.0 181.0 ... e5ba7672 \n", "1 0 2.0 0 44.0 1.0 102.0 8.0 2.0 2.0 4.0 ... 07c540c4 \n", "2 0 2.0 0 1.0 14.0 767.0 89.0 4.0 2.0 245.0 ... 8efede7f \n", "3 0 NaN 893 NaN NaN 4392.0 NaN 0.0 0.0 0.0 ... 1e88c74f \n", "4 0 3.0 -1 NaN 0.0 2.0 0.0 3.0 0.0 0.0 ... 1e88c74f \n", "\n", " C18 C19 C20 C21 C22 C23 C24 \\\n", "0 f54016b9 21ddcdc9 b1252a9d 07b5194c NaN 3a171ecb c5c50484 \n", "1 b04e4670 21ddcdc9 5840adea 60f6221e NaN 3a171ecb 43f13e8b \n", "2 3412118d NaN NaN e587c466 ad3062eb 3a171ecb 3b183c5c \n", "3 74ef3502 NaN NaN 6b3a5ca6 NaN 3a171ecb 9117a34a \n", "4 26b3c7a7 NaN NaN 21c9516a NaN 32c7478e b34f3128 \n", "\n", " C25 C26 \n", "0 e8b83407 9727dd16 \n", "1 e8b83407 731c3655 \n", "2 NaN NaN \n", "3 NaN NaN \n", "4 NaN NaN \n", "\n", "[5 rows x 40 columns]" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "nume_cols = [\"I\" + str(i) for i in range(1, 14)]\n", "cate_cols = [\"C\" + str(i) for i in range(1, 27)]\n", "label_col = \"Label\"\n", "\n", "header = [label_col] + nume_cols + cate_cols\n", "with TemporaryDirectory() as tmp:\n", " all_data = criteo.load_pandas_df(size=SIZE, local_cache_path=tmp, header=header)\n", "display(all_data.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we cut three sets (train_data (first 80%), valid_data (middle 10%) and test_data (last 10%)), cut from the original all data.
\n", "Notably, considering the Criteo is a kind of time-series streaming data, which is also very common in recommendation scenario, we split the data by its order." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# split data to 3 sets \n", "length = len(all_data)\n", "train_data = all_data.loc[:0.8*length-1]\n", "valid_data = all_data.loc[0.8*length:0.9*length-1]\n", "test_data = all_data.loc[0.9*length:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic Usage\n", "### Ordinal Encoding\n", "Considering LightGBM could handle the low-frequency features and missing value by itself, for basic usage, we only encode the string-like categorical features by an ordinal encoder." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train Data Shape: X: (80000, 39); Y: (80000,).\n", "Valid Data Shape: X: (10000, 39); Y: (10000,).\n", "Test Data Shape: X: (10000, 39); Y: (10000,).\n", "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
I1I2I3I4I5I6I7I8I9I10...C17C18C19C20C21C22C23C24C25C26
01.015.00.01382.04.015.02.0181.01.0...11111111111
12.0044.01.0102.08.02.02.04.01.0...22122111212
22.001.014.0767.089.04.02.0245.01.0...33122243113498199
3NaN893NaNNaN4392.0NaN0.00.00.0NaN...441222441114498199
43.0-1NaN0.02.00.03.00.00.01.0...451222451125498199
\n", "

5 rows × 39 columns

\n", "
" ], "text/plain": [ " I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 ... C17 C18 \\\n", "0 1.0 1 5.0 0.0 1382.0 4.0 15.0 2.0 181.0 1.0 ... 1 1 \n", "1 2.0 0 44.0 1.0 102.0 8.0 2.0 2.0 4.0 1.0 ... 2 2 \n", "2 2.0 0 1.0 14.0 767.0 89.0 4.0 2.0 245.0 1.0 ... 3 3 \n", "3 NaN 893 NaN NaN 4392.0 NaN 0.0 0.0 0.0 NaN ... 4 4 \n", "4 3.0 -1 NaN 0.0 2.0 0.0 3.0 0.0 0.0 1.0 ... 4 5 \n", "\n", " C19 C20 C21 C22 C23 C24 C25 C26 \n", "0 1 1 1 11 1 1 1 1 \n", "1 1 2 2 11 1 2 1 2 \n", "2 1222 4 3 1 1 3 49 8199 \n", "3 1222 4 4 11 1 4 49 8199 \n", "4 1222 4 5 11 2 5 49 8199 \n", "\n", "[5 rows x 39 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ord_encoder = ce.ordinal.OrdinalEncoder(cols=cate_cols)\n", "\n", "def encode_csv(df, encoder, label_col, typ=\"fit\"):\n", " if typ == \"fit\":\n", " df = encoder.fit_transform(df)\n", " else:\n", " df = encoder.transform(df)\n", " y = df[label_col].values\n", " del df[label_col]\n", " return df, y\n", "\n", "train_x, train_y = encode_csv(train_data, ord_encoder, label_col)\n", "valid_x, valid_y = encode_csv(valid_data, ord_encoder, label_col, \"transform\")\n", "test_x, test_y = encode_csv(test_data, ord_encoder, label_col, \"transform\")\n", "\n", "print(\"Train Data Shape: X: {trn_x_shape}; Y: {trn_y_shape}.\\nValid Data Shape: X: {vld_x_shape}; Y: {vld_y_shape}.\\nTest Data Shape: X: {tst_x_shape}; Y: {tst_y_shape}.\\n\"\n", " .format(trn_x_shape=train_x.shape,\n", " trn_y_shape=train_y.shape,\n", " vld_x_shape=valid_x.shape,\n", " vld_y_shape=valid_y.shape,\n", " tst_x_shape=test_x.shape,\n", " tst_y_shape=test_y.shape,))\n", "train_x.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create model\n", "When both hyper-parameters and data are ready, we can create a model:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[LightGBM] [Warning] Categorical features with more bins than the configured maximum bin number found.\n", "[LightGBM] [Warning] For categorical features, max_bin and max_bin_by_feature may be ignored with a large number of categories.\n", "[LightGBM] [Info] Number of positive: 17958, number of negative: 62042\n", "[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.027934 seconds.\n", "You can set `force_col_wise=true` to remove the overhead.\n", "[LightGBM] [Info] Total Bins 38971\n", "[LightGBM] [Info] Number of data points in the train set: 80000, number of used features: 39\n", "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.224475 -> initscore=-1.239776\n", "[LightGBM] [Info] Start training from score -1.239776\n", "Training until validation scores don't improve for 20 rounds\n", "Early stopping, best iteration is:\n", "[18]\tvalid_0's auc: 0.759658\n" ] } ], "source": [ "lgb_train = lgb.Dataset(train_x, train_y.reshape(-1), params=params, categorical_feature=cate_cols)\n", "lgb_valid = lgb.Dataset(valid_x, valid_y.reshape(-1), reference=lgb_train, categorical_feature=cate_cols)\n", "lgb_test = lgb.Dataset(test_x, test_y.reshape(-1), reference=lgb_train, categorical_feature=cate_cols)\n", "lgb_model = lgb.train(params,\n", " lgb_train,\n", " num_boost_round=NUM_OF_TREES,\n", " valid_sets=lgb_valid,\n", " callbacks=[lgb.early_stopping(EARLY_STOPPING_ROUNDS)])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's see what is the model's performance:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'auc': 0.7655408801711783, 'logloss': 0.4682583178835999}\n" ] } ], "source": [ "test_preds = lgb_model.predict(test_x)\n", "auc = roc_auc_score(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n", "logloss = log_loss(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n", "res_basic = {\"auc\": auc, \"logloss\": logloss}\n", "print(res_basic)\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/notebook_utils.json+json": { "data": 0.7655408801711783, "encoder": "json", "name": "auc_basic" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "auc_basic" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.4682583178835999, "encoder": "json", "name": "logloss_basic" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "logloss_basic" } }, "output_type": "display_data" } ], "source": [ "# Record results for tests - ignore this cell\n", "store_metadata(\"auc_basic\", res_basic[\"auc\"])\n", "store_metadata(\"logloss_basic\", res_basic[\"logloss\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Optimized Usage\n", "### Label-encoding and Binary-encoding\n", "Next, since LightGBM has a better capability in handling dense numerical features effectively, we try to convert all the categorical features in original data into numerical ones, by label-encoding [3] and binary-encoding [4]. Also due to the sequence property of Criteo, the label-encoding we adopted is executed one-by-one, which means we encode the samples in order, by the information of the previous samples before each sample (sequential label-encoding and sequential count-encoding). Besides, we also filter the low-frequency categorical features and fill the missing values by the mean of corresponding columns for the numerical features. (consulting `lgb_utils.NumEncoder`)\n", "\n", "Specifically, in `lgb_utils.NumEncoder`, the main steps are as follows.\n", "* Firstly, we convert the low-frequency categorical features to `\"LESS\"` and the missing categorical features to `\"UNK\"`. \n", "* Secondly, we convert the missing numerical features into the mean of corresponding columns. \n", "* Thirdly, the string-like categorical features are ordinal encoded like the example shown in basic usage. \n", "* And then, we target encode the categorical features in the samples order one-by-one. For each sample, we add the label and count information of its former samples into the data and produce new features. Formally, for $i=1,2,...,n$, we add $\\frac{\\sum\\nolimits_{j=1}^{i-1} I(x_j=c) \\cdot y}{\\sum\\nolimits_{j=1}^{i-1} I(x_j=c)}$ as a new label feature for current sample $x_i$, where $c$ is a category to encode in current sample, so $(i-1)$ is the number of former samples, and $I(\\cdot)$ is the indicator function that check the former samples contain $c$ (whether $x_j=c$) or not. At the meantime, we also add the count frequency of $c$, which is $\\frac{\\sum\\nolimits_{j=1}^{i-1} I(x_j=c)}{i-1}$, as a new count feature. \n", "* Finally, based on the results of ordinal encoding, we add the binary encoding results as new columns into the data.\n", "\n", "Note that the statistics used in the above process only updates when fitting the training set, while maintaining static when transforming the testing set because the label of test data should be considered as unknown." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-03-31 22:23:47,402 [INFO] Filtering and fillna features\n", "100%|██████████| 26/26 [00:05<00:00, 4.94it/s]\n", "100%|██████████| 13/13 [00:00<00:00, 175.39it/s]\n", "2025-03-31 22:23:52,763 [INFO] Ordinal encoding cate features\n", "2025-03-31 22:23:54,739 [INFO] Target encoding cate features\n", "100%|██████████| 26/26 [00:06<00:00, 4.23it/s]\n", "2025-03-31 22:24:00,890 [INFO] Start manual binary encoding\n", "100%|██████████| 65/65 [00:08<00:00, 7.79it/s]\n", "100%|██████████| 26/26 [00:05<00:00, 4.91it/s]\n", "2025-03-31 22:24:14,799 [INFO] Filtering and fillna features\n", "100%|██████████| 26/26 [00:00<00:00, 111.90it/s]\n", "100%|██████████| 13/13 [00:00<00:00, 1011.14it/s]\n", "2025-03-31 22:24:15,061 [INFO] Ordinal encoding cate features\n", "2025-03-31 22:24:15,171 [INFO] Target encoding cate features\n", "100%|██████████| 26/26 [00:00<00:00, 38.43it/s]\n", "2025-03-31 22:24:15,853 [INFO] Start manual binary encoding\n", "100%|██████████| 65/65 [00:09<00:00, 7.16it/s]\n", "100%|██████████| 26/26 [00:04<00:00, 6.37it/s]\n", "2025-03-31 22:24:29,249 [INFO] Filtering and fillna features\n", "100%|██████████| 26/26 [00:00<00:00, 53.84it/s]\n", "100%|██████████| 13/13 [00:00<00:00, 939.65it/s]\n", "2025-03-31 22:24:29,773 [INFO] Ordinal encoding cate features\n", "2025-03-31 22:24:30,059 [INFO] Target encoding cate features\n", "100%|██████████| 26/26 [00:00<00:00, 28.50it/s]\n", "2025-03-31 22:24:30,985 [INFO] Start manual binary encoding\n", "100%|██████████| 65/65 [00:09<00:00, 7.03it/s]\n", "100%|██████████| 26/26 [00:04<00:00, 6.28it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train Data Shape: X: (80000, 268); Y: (80000, 1).\n", "Valid Data Shape: X: (10000, 268); Y: (10000, 1).\n", "Test Data Shape: X: (10000, 268); Y: (10000, 1).\n", "\n" ] } ], "source": [ "label_col = \"Label\"\n", "num_encoder = lgb_utils.NumEncoder(cate_cols, nume_cols, label_col)\n", "train_x, train_y = num_encoder.fit_transform(train_data)\n", "valid_x, valid_y = num_encoder.transform(valid_data)\n", "test_x, test_y = num_encoder.transform(test_data)\n", "del num_encoder\n", "print(\"Train Data Shape: X: {trn_x_shape}; Y: {trn_y_shape}.\\nValid Data Shape: X: {vld_x_shape}; Y: {vld_y_shape}.\\nTest Data Shape: X: {tst_x_shape}; Y: {tst_y_shape}.\\n\"\n", " .format(trn_x_shape=train_x.shape,\n", " trn_y_shape=train_y.shape,\n", " vld_x_shape=valid_x.shape,\n", " vld_y_shape=valid_y.shape,\n", " tst_x_shape=test_x.shape,\n", " tst_y_shape=test_y.shape,))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training and Evaluation" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[LightGBM] [Info] Number of positive: 17958, number of negative: 62042\n", "[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.235469 seconds.\n", "You can set `force_row_wise=true` to remove the overhead.\n", "And if memory is not enough, you can set `force_col_wise=true`.\n", "[LightGBM] [Info] Total Bins 15787\n", "[LightGBM] [Info] Number of data points in the train set: 80000, number of used features: 267\n", "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.224475 -> initscore=-1.239776\n", "[LightGBM] [Info] Start training from score -1.239776\n", "Training until validation scores don't improve for 20 rounds\n", "Early stopping, best iteration is:\n", "[43]\tvalid_0's auc: 0.77085\n" ] } ], "source": [ "lgb_train = lgb.Dataset(train_x, train_y.reshape(-1), params=params)\n", "lgb_valid = lgb.Dataset(valid_x, valid_y.reshape(-1), reference=lgb_train)\n", "lgb_model = lgb.train(params,\n", " lgb_train,\n", " num_boost_round=NUM_OF_TREES,\n", " valid_sets=lgb_valid,\n", " callbacks=[lgb.early_stopping(EARLY_STOPPING_ROUNDS)])" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "test_preds = lgb_model.predict(test_x)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'auc': 0.7758548016657666, 'logloss': 0.46030887404896165}\n" ] } ], "source": [ "auc = roc_auc_score(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n", "logloss = log_loss(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n", "res_optim = {\"auc\": auc, \"logloss\": logloss}\n", "\n", "print(res_optim)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "application/notebook_utils.json+json": { "data": 0.7758548016657666, "encoder": "json", "name": "auc_opt" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "auc_opt" } }, "output_type": "display_data" }, { "data": { "application/notebook_utils.json+json": { "data": 0.46030887404896165, "encoder": "json", "name": "logloss_opt" } }, "metadata": { "notebook_utils": { "data": true, "display": false, "name": "logloss_opt" } }, "output_type": "display_data" } ], "source": [ "# Record results for tests - ignore this cell\n", "store_metadata(\"auc_opt\", res_optim[\"auc\"])\n", "store_metadata(\"logloss_opt\", res_optim[\"logloss\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model saving and loading\n", "Now we finish the basic training and testing for LightGBM, next let's try to save and reload the model, and then evaluate it again." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "with TemporaryDirectory() as tmp:\n", " save_file = os.path.join(tmp, \"finished.model\")\n", " lgb_model.save_model(save_file)\n", " loaded_model = lgb.Booster(model_file=save_file)\n", "\n", "# eval the performance again\n", "test_preds = loaded_model.predict(test_x)\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'auc': 0.7758548016657666, 'logloss': 0.46030887404896165}\n" ] } ], "source": [ "auc = roc_auc_score(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n", "logloss = log_loss(np.asarray(test_y.reshape(-1)), np.asarray(test_preds))\n", "\n", "print({\"auc\": auc, \"logloss\": logloss})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Additional Reading\n", "\n", "\\[1\\] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems. 3146–3154.
\n", "\\[2\\] The parameters of LightGBM: https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst
\n", "\\[3\\] Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018. CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363 (2018).
\n", "\\[4\\] Scikit-learn. 2018. categorical_encoding. https://github.com/scikit-learn-contrib/categorical-encoding
\n" ] } ], "metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "recommenders", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 2 }