{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copyright (c) Recommenders contributors.\n",
"\n",
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# LSTUR: Neural News Recommendation with Long- and Short-term User Representations\n",
"LSTUR \\[1\\] is a news recommendation approach capturing users' both long-term preferences and short-term interests. The core of LSTUR is a news encoder and a user encoder. In the news encoder, we learn representations of news from their titles. In user encoder, we propose to learn long-term\n",
"user representations from the embeddings of their IDs. In addition, we propose to learn short-term user representations from their recently browsed news via GRU network. Besides, we propose two methods to combine\n",
"long-term and short-term user representations. The first one is using the long-term user representation to initialize the hidden state of the GRU network in short-term user representation. The second one is concatenating both\n",
"long- and short-term user representations as a unified user vector.\n",
"\n",
"## Properties of LSTUR:\n",
"- LSTUR captures users' both long-term and short term preference.\n",
"- It uses embeddings of users' IDs to learn long-term user representations.\n",
"- It uses users' recently browsed news via GRU network to learn short-term user representations.\n",
"\n",
"## Data format:\n",
"For quicker training and evaluaiton, we sample MINDdemo dataset of 5k users from [MIND small dataset](https://msnews.github.io/). The MINDdemo dataset has the same file format as MINDsmall and MINDlarge. If you want to try experiments on MINDsmall and MINDlarge, please change the dowload source. Select the MIND_type parameter from ['large', 'small', 'demo'] to choose dataset.\n",
" \n",
"**MINDdemo_train** is used for training, and **MINDdemo_dev** is used for evaluation. Training data and evaluation data are composed of a news file and a behaviors file. You can find more detailed data description in [MIND repo](https://github.com/msnews/msnews.github.io/blob/master/assets/doc/introduction.md)\n",
"\n",
"### news data\n",
"This file contains news information including newsid, category, subcatgory, news title, news abstarct, news url and entities in news title, entities in news abstarct.\n",
"One simple example:
\n",
"\n",
"`N46466\tlifestyle\tlifestyleroyals\tThe Brands Queen Elizabeth, Prince Charles, and Prince Philip Swear By\tShop the notebooks, jackets, and more that the royals can't live without.\thttps://www.msn.com/en-us/lifestyle/lifestyleroyals/the-brands-queen-elizabeth,-prince-charles,-and-prince-philip-swear-by/ss-AAGH0ET?ocid=chopendata\t[{\"Label\": \"Prince Philip, Duke of Edinburgh\", \"Type\": \"P\", \"WikidataId\": \"Q80976\", \"Confidence\": 1.0, \"OccurrenceOffsets\": [48], \"SurfaceForms\": [\"Prince Philip\"]}, {\"Label\": \"Charles, Prince of Wales\", \"Type\": \"P\", \"WikidataId\": \"Q43274\", \"Confidence\": 1.0, \"OccurrenceOffsets\": [28], \"SurfaceForms\": [\"Prince Charles\"]}, {\"Label\": \"Elizabeth II\", \"Type\": \"P\", \"WikidataId\": \"Q9682\", \"Confidence\": 0.97, \"OccurrenceOffsets\": [11], \"SurfaceForms\": [\"Queen Elizabeth\"]}]\t[]`\n",
"
\n",
"\n",
"In general, each line in data file represents information of one piece of news:
\n",
"\n",
"`[News ID] [Category] [Subcategory] [News Title] [News Abstrct] [News Url] [Entities in News Title] [Entities in News Abstract] ...`\n",
"\n",
"
\n",
"\n",
"We generate a word_dict file to tranform words in news title to word indexes, and a embedding matrix is initted from pretrained glove embeddings.\n",
"\n",
"### behaviors data\n",
"One simple example:
\n",
"`1\tU82271\t11/11/2019 3:28:58 PM\tN3130 N11621 N12917 N4574 N12140 N9748\tN13390-0 N7180-0 N20785-0 N6937-0 N15776-0 N25810-0 N20820-0 N6885-0 N27294-0 N18835-0 N16945-0 N7410-0 N23967-0 N22679-0 N20532-0 N26651-0 N22078-0 N4098-0 N16473-0 N13841-0 N15660-0 N25787-0 N2315-0 N1615-0 N9087-0 N23880-0 N3600-0 N24479-0 N22882-0 N26308-0 N13594-0 N2220-0 N28356-0 N17083-0 N21415-0 N18671-0 N9440-0 N17759-0 N10861-0 N21830-0 N8064-0 N5675-0 N15037-0 N26154-0 N15368-1 N481-0 N3256-0 N20663-0 N23940-0 N7654-0 N10729-0 N7090-0 N23596-0 N15901-0 N16348-0 N13645-0 N8124-0 N20094-0 N27774-0 N23011-0 N14832-0 N15971-0 N27729-0 N2167-0 N11186-0 N18390-0 N21328-0 N10992-0 N20122-0 N1958-0 N2004-0 N26156-0 N17632-0 N26146-0 N17322-0 N18403-0 N17397-0 N18215-0 N14475-0 N9781-0 N17958-0 N3370-0 N1127-0 N15525-0 N12657-0 N10537-0 N18224-0`\n",
"
\n",
"\n",
"In general, each line in data file represents one instance of an impression. The format is like:
\n",
"\n",
"`[Impression ID] [User ID] [Impression Time] [User Click History] [Impression News]`\n",
"\n",
"
\n",
"\n",
"User Click History is the user historical clicked news before Impression Time. Impression News is the displayed news in an impression, which format is:
\n",
"\n",
"`[News ID 1]-[label1] ... [News ID n]-[labeln]`\n",
"\n",
"
\n",
"Label represents whether the news is clicked by the user. All information of news in User Click History and Impression News can be found in news data file."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Global settings and imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/anaconda/envs/tf2/lib/python3.7/site-packages/papermill/iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.\n",
" from pyarrow import HadoopFileSystem\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"System version: 3.7.11 (default, Jul 27 2021, 14:32:16) \n",
"[GCC 7.5.0]\n",
"Tensorflow version: 2.6.1\n"
]
}
],
"source": [
"import os\n",
"import sys\n",
"import numpy as np\n",
"import zipfile\n",
"from tqdm import tqdm\n",
"from tempfile import TemporaryDirectory\n",
"import tensorflow as tf\n",
"tf.get_logger().setLevel('ERROR') # only show error messages\n",
"\n",
"from recommenders.models.deeprec.deeprec_utils import download_deeprec_resources \n",
"from recommenders.models.newsrec.newsrec_utils import prepare_hparams\n",
"from recommenders.models.newsrec.models.lstur import LSTURModel\n",
"from recommenders.models.newsrec.io.mind_iterator import MINDIterator\n",
"from recommenders.models.newsrec.newsrec_utils import get_mind_data_set\n",
"from recommenders.utils.notebook_utils import store_metadata\n",
"\n",
"print(\"System version: {}\".format(sys.version))\n",
"print(\"Tensorflow version: {}\".format(tf.__version__))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Prepare Parameters"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"epochs = 5\n",
"seed = 40\n",
"batch_size = 32\n",
"\n",
"# Options: demo, small, large\n",
"MIND_type = \"demo\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Download and load data"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 17.0k/17.0k [00:01<00:00, 9.67kKB/s]\n",
"100%|██████████| 9.84k/9.84k [00:01<00:00, 8.34kKB/s]\n",
"100%|██████████| 95.0k/95.0k [00:08<00:00, 11.4kKB/s]\n"
]
}
],
"source": [
"tmpdir = TemporaryDirectory()\n",
"data_path = tmpdir.name\n",
"\n",
"train_news_file = os.path.join(data_path, 'train', r'news.tsv')\n",
"train_behaviors_file = os.path.join(data_path, 'train', r'behaviors.tsv')\n",
"valid_news_file = os.path.join(data_path, 'valid', r'news.tsv')\n",
"valid_behaviors_file = os.path.join(data_path, 'valid', r'behaviors.tsv')\n",
"wordEmb_file = os.path.join(data_path, \"utils\", \"embedding.npy\")\n",
"userDict_file = os.path.join(data_path, \"utils\", \"uid2index.pkl\")\n",
"wordDict_file = os.path.join(data_path, \"utils\", \"word_dict.pkl\")\n",
"yaml_file = os.path.join(data_path, \"utils\", r'lstur.yaml')\n",
"\n",
"mind_url, mind_train_dataset, mind_dev_dataset, mind_utils = get_mind_data_set(MIND_type)\n",
"\n",
"if not os.path.exists(train_news_file):\n",
" download_deeprec_resources(mind_url, os.path.join(data_path, 'train'), mind_train_dataset)\n",
" \n",
"if not os.path.exists(valid_news_file):\n",
" download_deeprec_resources(mind_url, \\\n",
" os.path.join(data_path, 'valid'), mind_dev_dataset)\n",
"if not os.path.exists(yaml_file):\n",
" download_deeprec_resources(r'https://huggingface.co/datasets/Recommenders/MIND/resolve/main/', \\\n",
" os.path.join(data_path, 'utils'), mind_utils)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create hyper-parameters"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"data_format=news,iterator_type=None,support_quick_scoring=True,wordEmb_file=/tmp/tmpcpgw9veg/utils/embedding.npy,wordDict_file=/tmp/tmpcpgw9veg/utils/word_dict.pkl,userDict_file=/tmp/tmpcpgw9veg/utils/uid2index.pkl,vertDict_file=None,subvertDict_file=None,title_size=30,body_size=None,word_emb_dim=300,word_size=None,user_num=None,vert_num=None,subvert_num=None,his_size=50,npratio=4,dropout=0.2,attention_hidden_dim=200,head_num=4,head_dim=100,cnn_activation=relu,dense_activation=None,filter_num=400,window_size=3,vert_emb_dim=100,subvert_emb_dim=100,gru_unit=400,type=ini,user_emb_dim=50,learning_rate=0.0001,loss=cross_entropy_loss,optimizer=adam,epochs=5,batch_size=32,show_step=100000,metrics=['group_auc', 'mean_mrr', 'ndcg@5;10']\n"
]
}
],
"source": [
"hparams = prepare_hparams(yaml_file, \n",
" wordEmb_file=wordEmb_file,\n",
" wordDict_file=wordDict_file, \n",
" userDict_file=userDict_file,\n",
" batch_size=batch_size,\n",
" epochs=epochs)\n",
"print(hparams)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"iterator = MINDIterator"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Train the LSTUR model"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tensor(\"conv1d/Relu:0\", shape=(?, 30, 400), dtype=float32)\n",
"Tensor(\"att_layer2/Sum_1:0\", shape=(?, 400), dtype=float32)\n"
]
}
],
"source": [
"model = LSTURModel(hparams, iterator, seed=seed)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"586it [00:03, 155.76it/s]\n",
"236it [00:09, 26.08it/s]\n",
"7538it [00:00, 7590.51it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'group_auc': 0.5201, 'mean_mrr': 0.2214, 'ndcg@5': 0.2292, 'ndcg@10': 0.2912}\n"
]
}
],
"source": [
"print(model.run_eval(valid_news_file, valid_behaviors_file))"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"1086it [02:23, 7.55it/s]\n",
"586it [00:01, 430.29it/s]\n",
"236it [00:08, 28.16it/s]\n",
"7538it [00:01, 6738.86it/s]\n",
"1it [00:00, 7.26it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"at epoch 1\n",
"train info: logloss loss:1.4868141242592814\n",
"eval info: group_auc:0.5973, mean_mrr:0.2622, ndcg@10:0.3501, ndcg@5:0.2861\n",
"at epoch 1 , train time: 143.8 eval time: 18.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"1086it [02:18, 7.85it/s]\n",
"586it [00:01, 455.05it/s]\n",
"236it [00:08, 28.32it/s]\n",
"7538it [00:01, 6669.92it/s]\n",
"1it [00:00, 8.64it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"at epoch 2\n",
"train info: logloss loss:1.3999453176011916\n",
"eval info: group_auc:0.6219, mean_mrr:0.2803, ndcg@10:0.3726, ndcg@5:0.3099\n",
"at epoch 2 , train time: 138.3 eval time: 19.2\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"1086it [02:18, 7.83it/s]\n",
"586it [00:01, 448.54it/s]\n",
"236it [00:08, 28.40it/s]\n",
"7538it [00:00, 8089.03it/s]\n",
"1it [00:00, 8.04it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"at epoch 3\n",
"train info: logloss loss:1.3563778104044455\n",
"eval info: group_auc:0.6281, mean_mrr:0.285, ndcg@10:0.3785, ndcg@5:0.3159\n",
"at epoch 3 , train time: 138.7 eval time: 18.2\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"1086it [02:18, 7.84it/s]\n",
"586it [00:01, 431.78it/s]\n",
"236it [00:08, 28.00it/s]\n",
"7538it [00:01, 7187.47it/s]\n",
"1it [00:00, 8.33it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"at epoch 4\n",
"train info: logloss loss:1.3173029956786892\n",
"eval info: group_auc:0.6369, mean_mrr:0.2913, ndcg@10:0.3851, ndcg@5:0.3225\n",
"at epoch 4 , train time: 138.5 eval time: 18.5\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"1086it [02:18, 7.84it/s]\n",
"586it [00:01, 416.18it/s]\n",
"236it [00:08, 28.36it/s]\n",
"7538it [00:01, 7087.70it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"at epoch 5\n",
"train info: logloss loss:1.2810899292017655\n",
"eval info: group_auc:0.6462, mean_mrr:0.3031, ndcg@10:0.3983, ndcg@5:0.3349\n",
"at epoch 5 , train time: 138.5 eval time: 18.4\n",
"CPU times: user 25min 40s, sys: 2min 21s, total: 28min 2s\n",
"Wall time: 13min 10s\n"
]
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"model.fit(train_news_file, train_behaviors_file, valid_news_file, valid_behaviors_file)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"586it [00:01, 440.26it/s]\n",
"236it [00:08, 28.51it/s]\n",
"7538it [00:00, 9166.73it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'group_auc': 0.6462, 'mean_mrr': 0.3031, 'ndcg@5': 0.3349, 'ndcg@10': 0.3983}\n",
"CPU times: user 37.1 s, sys: 2.69 s, total: 39.8 s\n",
"Wall time: 18.1 s\n"
]
}
],
"source": [
"%%time\n",
"res_syn = model.run_eval(valid_news_file, valid_behaviors_file)\n",
"print(res_syn)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Record results for tests - ignore this cell\n",
"store_metadata(\"group_auc\", res_syn['group_auc'])\n",
"store_metadata(\"mean_mrr\", res_syn['mean_mrr'])\n",
"store_metadata(\"ndcg@5\", res_syn['ndcg@5'])\n",
"store_metadata(\"ndcg@10\", res_syn['ndcg@10'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Save the model"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"model_path = os.path.join(data_path, \"model\")\n",
"os.makedirs(model_path, exist_ok=True)\n",
"\n",
"model.model.save_weights(os.path.join(model_path, \"lstur_ckpt\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Output Prediction File\n",
"This code segment is used to generate the prediction.zip file, which is in the same format in [MIND Competition Submission Tutorial](https://competitions.codalab.org/competitions/24122#learn_the_details-submission-guidelines).\n",
"\n",
"Please change the `MIND_type` parameter to `large` if you want to submit your prediction to [MIND Competition](https://msnews.github.io/competition.html)."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"586it [00:01, 438.04it/s]\n",
"236it [00:08, 28.26it/s]\n",
"7538it [00:00, 8876.72it/s]\n"
]
}
],
"source": [
"group_impr_indexes, group_labels, group_preds = model.run_fast_eval(valid_news_file, valid_behaviors_file)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"7538it [00:00, 44730.54it/s]\n"
]
}
],
"source": [
"with open(os.path.join(data_path, 'prediction.txt'), 'w') as f:\n",
" for impr_index, preds in tqdm(zip(group_impr_indexes, group_preds)):\n",
" impr_index += 1\n",
" pred_rank = (np.argsort(np.argsort(preds)[::-1]) + 1).tolist()\n",
" pred_rank = '[' + ','.join([str(i) for i in pred_rank]) + ']'\n",
" f.write(' '.join([str(impr_index), pred_rank])+ '\\n')"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"f = zipfile.ZipFile(os.path.join(data_path, 'prediction.zip'), 'w', zipfile.ZIP_DEFLATED)\n",
"f.write(os.path.join(data_path, 'prediction.txt'), arcname='prediction.txt')\n",
"f.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reference\n",
"\\[1\\] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu and Xing Xie: Neural News Recommendation with Long- and Short-term User Representations, ACL 2019
\n",
"\\[2\\] Wu, Fangzhao, et al. \"MIND: A Large-scale Dataset for News Recommendation\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://msnews.github.io/competition.html
\n",
"\\[3\\] GloVe: Global Vectors for Word Representation. https://nlp.stanford.edu/projects/glove/"
]
}
],
"metadata": {
"celltoolbar": "Tags",
"interpreter": {
"hash": "3a9a0c422ff9f08d62211b9648017c63b0a26d2c935edc37ebb8453675d13bb5"
},
"kernelspec": {
"display_name": "Python 3.7.11 64-bit ('tf2': conda)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}