{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "\n", "# Recommendation Systems: Two Tower Deep Learning Models with RedisVL\n", "\n", "\"Open" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Recommendation systems are a common application of machine learning and serve many industries from e-commerce to music streaming platforms.\n", "\n", "There are many different architectures that can be followed to build a recommendation system. In previous example notebooks we demonstrated two common approaches that leverage different methods. Our first showed how to do [content filtering with RedisVL](content_filtering.ipynb) where an item's underlying features determine what gets recommended.\n", "Next, we showcased how RedisVL can be used to build a [collaborative filtering recommender](collaborative_filtering.ipynb), which leverages users' ratings of items to create personalized recommendations. Before continuing with this notebook we encourage you to start with the previous two.\n", "\n", "In this notebook we'll demonstrate how to build a [two tower recommendation system](https://cloud.google.com/blog/products/ai-machine-learning/scaling-deep-retrieval-tensorflow-two-towers-architecture)\n", "and compare it to architectures we've seen before." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To mix things up a bit, instead of using our movies dataset like the previous two examples, we'll look at brick & mortar restaurants in San Francisco as our items to recommend." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Environment Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -q redis \"redisvl>=0.4.1\" pandas torch requests scikit-learn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Install Redis Stack\n", "\n", "Later in this tutorial, Redis will be used to store, index, and query vector\n", "embeddings. **We need to make sure we have a Redis instance available.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Redis in Colab\n", "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "%%sh\n", "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", "sudo apt-get update > /dev/null 2>&1\n", "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", "redis-stack-server --daemonize yes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Other ways to get Redis\n", "There are many ways to get the necessary redis-stack instance running\n", "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.io/try-free/). Or, if you have your\n", "own version of Redis Enterprise running, that works too!\n", "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define the Redis Connection URL\n", "\n", "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import requests\n", "import pandas as pd\n", "import json\n", "\n", "# Replace values below with your own if using Redis Cloud instance\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", "\n", "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def fetch_data(file_name):\n", " dataset_path = 'datasets/two_towers/'\n", " try:\n", " with open(dataset_path + file_name, 'r') as f:\n", " return json.load(f)\n", " except:\n", " url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/two-towers/'\n", " r = requests.get(url + file_name)\n", " if not os.path.exists(dataset_path):\n", " os.makedirs(dataset_path)\n", " with open(dataset_path + file_name, 'wb') as f:\n", " f.write(r.content)\n", " return json.loads(r.content.decode('utf-8'))" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "we have 147 restaurants in our dataset, with 14700 total reviews\n" ] } ], "source": [ "# the original dataset can be found here: https://www.kaggle.com/datasets/jkgatt/restaurant-data-with-100-trip-advisor-reviews-each\n", "\n", "restaurant_data = fetch_data('factual_tripadvisor_restaurant_data_all_100_reviews.json')\n", "\n", "print(f\"we have {restaurant_data['restaurant_count']} restaurants in our dataset, with {restaurant_data['total_review_count']} total reviews\")\n", "\n", "restaurant_data = restaurant_data[\"restaurants\"] # ignore the count fields" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameaddresslocalitylatitudelongitudecuisinepriceratinghoursparking...meal_takeoutmeal_cateroptions_healthyoptions_organicoptions_vegetarianoptions_veganoptions_glutenfreeoptions_lowfatreviewsunique_name
021st Amendment Brewery & Restaurant563 2nd StSan Francisco37.782448-122.392576[Cafe, Pub Food, American, Burgers, Pizza]24.0{'monday': [['11:30', '23:59']], 'tuesday': [[...True...TrueFalseTrueFalseTrueFalseFalseFalse[{'review_website': 'TripAdvisor', 'review_url...21st Amendment Brewery & Restaurant 563 2nd St
1Absinthe Brasserie & Bar398 Hayes StSan Francisco37.777083-122.422882[French, Californian, Mediterranean, Cafe, Ame...34.0{'tuesday': [['11:30', '23:59']], 'wednesday':...True...TrueTrueTrueFalseTrueFalseFalseFalse[{'review_website': 'TripAdvisor', 'review_url...Absinthe Brasserie & Bar 398 Hayes St
2Amber India Restaurant25 Yerba Buena LnSan Francisco37.785772-122.404401[Indian, Chinese, Vegetarian, Asian, Pakistani]24.5{'monday': [['11:30', '14:30'], ['17:00', '22:...True...TrueTrueTrueFalseTrueTrueTrueFalse[{'review_website': 'TripAdvisor', 'review_url...Amber India Restaurant 25 Yerba Buena Ln
3Americano8 Mission StSan Francisco37.793620-122.392915[Italian, American, Californian, Pub Food, Cafe]33.5{'monday': [['6:30', '10:30'], ['11:30', '14:3...True...TrueTrueTrueFalseTrueFalseFalseFalse[{'review_website': 'TripAdvisor', 'review_url...Americano 8 Mission St
4Anchor & Hope83 Minna StSan Francisco37.787848-122.398812[Seafood, American, Cafe, Chowder, Californian]34.0{'monday': [['11:30', '14:00'], ['17:30', '22:...True...TrueTrueTrueFalseTrueTrueTrueFalse[{'review_website': 'TripAdvisor', 'review_url...Anchor & Hope 83 Minna St
\n", "

5 rows × 33 columns

\n", "
" ], "text/plain": [ " name address locality \\\n", "0 21st Amendment Brewery & Restaurant 563 2nd St San Francisco \n", "1 Absinthe Brasserie & Bar 398 Hayes St San Francisco \n", "2 Amber India Restaurant 25 Yerba Buena Ln San Francisco \n", "3 Americano 8 Mission St San Francisco \n", "4 Anchor & Hope 83 Minna St San Francisco \n", "\n", " latitude longitude cuisine \\\n", "0 37.782448 -122.392576 [Cafe, Pub Food, American, Burgers, Pizza] \n", "1 37.777083 -122.422882 [French, Californian, Mediterranean, Cafe, Ame... \n", "2 37.785772 -122.404401 [Indian, Chinese, Vegetarian, Asian, Pakistani] \n", "3 37.793620 -122.392915 [Italian, American, Californian, Pub Food, Cafe] \n", "4 37.787848 -122.398812 [Seafood, American, Cafe, Chowder, Californian] \n", "\n", " price rating hours parking \\\n", "0 2 4.0 {'monday': [['11:30', '23:59']], 'tuesday': [[... True \n", "1 3 4.0 {'tuesday': [['11:30', '23:59']], 'wednesday':... True \n", "2 2 4.5 {'monday': [['11:30', '14:30'], ['17:00', '22:... True \n", "3 3 3.5 {'monday': [['6:30', '10:30'], ['11:30', '14:3... True \n", "4 3 4.0 {'monday': [['11:30', '14:00'], ['17:30', '22:... True \n", "\n", " ... meal_takeout meal_cater options_healthy options_organic \\\n", "0 ... True False True False \n", "1 ... True True True False \n", "2 ... True True True False \n", "3 ... True True True False \n", "4 ... True True True False \n", "\n", " options_vegetarian options_vegan options_glutenfree options_lowfat \\\n", "0 True False False False \n", "1 True False False False \n", "2 True True True False \n", "3 True False False False \n", "4 True True True False \n", "\n", " reviews \\\n", "0 [{'review_website': 'TripAdvisor', 'review_url... \n", "1 [{'review_website': 'TripAdvisor', 'review_url... \n", "2 [{'review_website': 'TripAdvisor', 'review_url... \n", "3 [{'review_website': 'TripAdvisor', 'review_url... \n", "4 [{'review_website': 'TripAdvisor', 'review_url... \n", "\n", " unique_name \n", "0 21st Amendment Brewery & Restaurant 563 2nd St \n", "1 Absinthe Brasserie & Bar 398 Hayes St \n", "2 Amber India Restaurant 25 Yerba Buena Ln \n", "3 Americano 8 Mission St \n", "4 Anchor & Hope 83 Minna St \n", "\n", "[5 rows x 33 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame(restaurant_data)\n", "\n", "df.fillna('', inplace=True)\n", "\n", "df.drop(columns=['region', 'country', 'tel','fax', 'email', 'website', 'address_extended', 'chain_name','trip_advisor_url'], inplace=True)\n", "df['unique_name'] = df['name'] +' ' + df['address'] # some restaurants are chains or have more than one location\n", "df.head()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you wanted to build a content filtering system now would be a good time to extract the text from the reviews, join them together and generate semantic embeddings from them like we did in our previous notebook.\n", "\n", "This would be a great approach, but to demonstrate the two tower architecture we won't use a pre-trained embedding model, and instead use the other columns as our raw features - but we will at least extract the numerical ratings from the reviews.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "df['min_rating'] = df['reviews'].apply(lambda x: np.min([r[\"review_rating\"] for r in x]))\n", "df['max_rating'] = df['reviews'].apply(lambda x: np.max([r[\"review_rating\"] for r in x]))\n", "df['avg_rating'] = df['reviews'].apply(lambda x: np.mean([r[\"review_rating\"] for r in x]))\n", "df['stddev_rating'] = df['reviews'].apply(lambda x: np.std([r[\"review_rating\"] for r in x]))\n", "df['price'] = df['price'].astype(int)\n", "\n", "# now take all the features we have and build a raw feature vector for each restaurant\n", "numerical_cols = df.select_dtypes(include=['float64', 'int64']).columns\n", "boolean_cols = df.select_dtypes(include=['bool']).columns\n", "\n", "# convert boolean columns to integers\n", "df[boolean_cols] = df[boolean_cols].astype(int)\n", "\n", "# combine numerical and boolean columns into a single vector\n", "df['feature_vector'] = df[numerical_cols.tolist() + boolean_cols.tolist()].values.tolist()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now have feature vectors with 30 features for each restaurant. The next step is to construct our raw feature vectors for our users.\n", "\n", "We don't have publicly available user data to correspond with this list of restaurants, so instead we'll generate some using the popular testing tool Faker." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" ] } ], "source": [ "!pip install Faker --quiet" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idnameusernameemailaddressphone_numberbirthdatelikesaccount_created_onprice_bracketnewsletternotificationsprofile_visibilitydata_sharing
02d72a59b-5b12-430f-a5c3-49f8b093cb3dKayla Clarkdebbie11teresa54@example.net8873 Thompson Cape\\nOsborneport, NV 34895231.228.4452x0081982-06-10[pizza, pasta, shakes, brunch, bbq, ethiopian,...2017-04-18middleTrueTruepublicTrue
1034b2b2f-1949-478d-abd6-add4b3275efeLeah Hopkinswilliamsanchezdarryl77@example.net353 Kimberly Green\\nRoachfort, FM 3438546690946321999-03-07[brunch, ethiopian, breweries]1970-06-21lowFalseTruepublicFalse
25d674492-3026-4cc9-b216-be675cf8d360Mason Pattersonjamescurtislopezchristopher@example.com945 Bryan Locks Suite 200\\nValenzuelaburgh, MI...885-983-45731914-02-14[cocktails, fine dining, pizza, shakes, ethiop...2013-03-10highFalseFalsefriends-onlyFalse
361e17d13-9e18-431f-8f06-208bd0469892Aaron Dixonmarshallkristenbecky20@example.org42388 Russell Harbors Suite 340\\nNorth Andrewc...448.270.3034x5831959-05-01[breweries, cocktails, fine dining]1973-12-11middleFalseTrueprivateTrue
48cc208b6-0f4f-459c-a8f5-31d3ca6deca6Loretta Eatonphatfieldaaustin@example.orgPSC 2899, Box 5115\\nAPO AE 79916663-371-4597x722951923-07-02[brunch, italian, bbq, mexican, burgers, pizza]2023-04-29highTrueTrueprivateTrue
\n", "
" ], "text/plain": [ " user_id name username \\\n", "0 2d72a59b-5b12-430f-a5c3-49f8b093cb3d Kayla Clark debbie11 \n", "1 034b2b2f-1949-478d-abd6-add4b3275efe Leah Hopkins williamsanchez \n", "2 5d674492-3026-4cc9-b216-be675cf8d360 Mason Patterson jamescurtis \n", "3 61e17d13-9e18-431f-8f06-208bd0469892 Aaron Dixon marshallkristen \n", "4 8cc208b6-0f4f-459c-a8f5-31d3ca6deca6 Loretta Eaton phatfield \n", "\n", " email \\\n", "0 teresa54@example.net \n", "1 darryl77@example.net \n", "2 lopezchristopher@example.com \n", "3 becky20@example.org \n", "4 aaustin@example.org \n", "\n", " address phone_number \\\n", "0 8873 Thompson Cape\\nOsborneport, NV 34895 231.228.4452x008 \n", "1 353 Kimberly Green\\nRoachfort, FM 34385 4669094632 \n", "2 945 Bryan Locks Suite 200\\nValenzuelaburgh, MI... 885-983-4573 \n", "3 42388 Russell Harbors Suite 340\\nNorth Andrewc... 448.270.3034x583 \n", "4 PSC 2899, Box 5115\\nAPO AE 79916 663-371-4597x72295 \n", "\n", " birthdate likes \\\n", "0 1982-06-10 [pizza, pasta, shakes, brunch, bbq, ethiopian,... \n", "1 1999-03-07 [brunch, ethiopian, breweries] \n", "2 1914-02-14 [cocktails, fine dining, pizza, shakes, ethiop... \n", "3 1959-05-01 [breweries, cocktails, fine dining] \n", "4 1923-07-02 [brunch, italian, bbq, mexican, burgers, pizza] \n", "\n", " account_created_on price_bracket newsletter notifications \\\n", "0 2017-04-18 middle True True \n", "1 1970-06-21 low False True \n", "2 2013-03-10 high False False \n", "3 1973-12-11 middle False True \n", "4 2023-04-29 high True True \n", "\n", " profile_visibility data_sharing \n", "0 public True \n", "1 public False \n", "2 friends-only False \n", "3 private True \n", "4 private True " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from faker import Faker\n", "from uuid import uuid4\n", "\n", "fake = Faker()\n", "\n", "def generate_user():\n", " return {\n", " \"user_id\": str(uuid4()),\n", " \"name\": fake.name(),\n", " \"username\": fake.user_name(),\n", " \"email\": fake.email(),\n", " \"address\": fake.address(),\n", " \"phone_number\": fake.phone_number(),\n", " \"birthdate\": fake.date_of_birth().isoformat(),\n", " \"likes\": fake.random_elements(elements=['burgers', 'shakes', 'pizza', 'italian', 'mexican', 'fine dining', 'bbq', 'cocktails', 'breweries', 'ethiopian', 'pasta', 'brunch','fast food'], unique=True),\n", " \"account_created_on\": fake.date() ,\n", " \"price_bracket\": fake.random_element(elements=(\"low\", \"middle\", \"high\")),\n", " \"newsletter\": fake.boolean(),\n", " \"notifications\": fake.boolean(),\n", " \"profile_visibility\": fake.random_element(elements=(\"public\", \"private\", \"friends-only\")),\n", " \"data_sharing\": fake.boolean()\n", " }\n", "\n", "users = [generate_user() for _ in range(1000)]\n", "\n", "users_df = pd.DataFrame(users)\n", "users_df.head()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idnameusernameemailaddressphone_numberbirthdatelikesaccount_created_onnewsletter...pastapizzashakesprice_bracket_highprice_bracket_lowprice_bracket_middleprofile_visibility_friends-onlyprofile_visibility_privateprofile_visibility_publicfeature_vector
02d72a59b-5b12-430f-a5c3-49f8b093cb3dKayla Clarkdebbie11teresa54@example.net8873 Thompson Cape\\nOsborneport, NV 34895231.228.4452x0081982-06-10[pizza, pasta, shakes, brunch, bbq, ethiopian,...2017-04-181...111001001[1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, ...
1034b2b2f-1949-478d-abd6-add4b3275efeLeah Hopkinswilliamsanchezdarryl77@example.net353 Kimberly Green\\nRoachfort, FM 3438546690946321999-03-07[brunch, ethiopian, breweries]1970-06-210...000010001[0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, ...
25d674492-3026-4cc9-b216-be675cf8d360Mason Pattersonjamescurtislopezchristopher@example.com945 Bryan Locks Suite 200\\nValenzuelaburgh, MI...885-983-45731914-02-14[cocktails, fine dining, pizza, shakes, ethiop...2013-03-100...111100100[0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, ...
361e17d13-9e18-431f-8f06-208bd0469892Aaron Dixonmarshallkristenbecky20@example.org42388 Russell Harbors Suite 340\\nNorth Andrewc...448.270.3034x5831959-05-01[breweries, cocktails, fine dining]1973-12-110...000001010[0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, ...
48cc208b6-0f4f-459c-a8f5-31d3ca6deca6Loretta Eatonphatfieldaaustin@example.orgPSC 2899, Box 5115\\nAPO AE 79916663-371-4597x722951923-07-02[brunch, italian, bbq, mexican, burgers, pizza]2023-04-291...010100010[1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, ...
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " user_id name username \\\n", "0 2d72a59b-5b12-430f-a5c3-49f8b093cb3d Kayla Clark debbie11 \n", "1 034b2b2f-1949-478d-abd6-add4b3275efe Leah Hopkins williamsanchez \n", "2 5d674492-3026-4cc9-b216-be675cf8d360 Mason Patterson jamescurtis \n", "3 61e17d13-9e18-431f-8f06-208bd0469892 Aaron Dixon marshallkristen \n", "4 8cc208b6-0f4f-459c-a8f5-31d3ca6deca6 Loretta Eaton phatfield \n", "\n", " email \\\n", "0 teresa54@example.net \n", "1 darryl77@example.net \n", "2 lopezchristopher@example.com \n", "3 becky20@example.org \n", "4 aaustin@example.org \n", "\n", " address phone_number \\\n", "0 8873 Thompson Cape\\nOsborneport, NV 34895 231.228.4452x008 \n", "1 353 Kimberly Green\\nRoachfort, FM 34385 4669094632 \n", "2 945 Bryan Locks Suite 200\\nValenzuelaburgh, MI... 885-983-4573 \n", "3 42388 Russell Harbors Suite 340\\nNorth Andrewc... 448.270.3034x583 \n", "4 PSC 2899, Box 5115\\nAPO AE 79916 663-371-4597x72295 \n", "\n", " birthdate likes \\\n", "0 1982-06-10 [pizza, pasta, shakes, brunch, bbq, ethiopian,... \n", "1 1999-03-07 [brunch, ethiopian, breweries] \n", "2 1914-02-14 [cocktails, fine dining, pizza, shakes, ethiop... \n", "3 1959-05-01 [breweries, cocktails, fine dining] \n", "4 1923-07-02 [brunch, italian, bbq, mexican, burgers, pizza] \n", "\n", " account_created_on newsletter ... pasta pizza shakes \\\n", "0 2017-04-18 1 ... 1 1 1 \n", "1 1970-06-21 0 ... 0 0 0 \n", "2 2013-03-10 0 ... 1 1 1 \n", "3 1973-12-11 0 ... 0 0 0 \n", "4 2023-04-29 1 ... 0 1 0 \n", "\n", " price_bracket_high price_bracket_low price_bracket_middle \\\n", "0 0 0 1 \n", "1 0 1 0 \n", "2 1 0 0 \n", "3 0 0 1 \n", "4 1 0 0 \n", "\n", " profile_visibility_friends-only profile_visibility_private \\\n", "0 0 0 \n", "1 0 0 \n", "2 1 0 \n", "3 0 1 \n", "4 0 1 \n", "\n", " profile_visibility_public \\\n", "0 1 \n", "1 1 \n", "2 0 \n", "3 0 \n", "4 0 \n", "\n", " feature_vector \n", "0 [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, ... \n", "1 [0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, ... \n", "2 [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, ... \n", "3 [0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, ... \n", "4 [1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, ... \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.preprocessing import MultiLabelBinarizer\n", "\n", "# use a MultiLabelBinarizer to one-hot encode our user's 'likes' column, which has a list of users' food preferences\n", "mlb = MultiLabelBinarizer()\n", "\n", "likes_encoded = mlb.fit_transform(users_df['likes'])\n", "likes_df = pd.DataFrame(likes_encoded, columns=mlb.classes_)\n", "\n", "# concatenate the original users_df with the new one-hot encoded likes_df\n", "users_df = pd.concat([users_df, likes_df], axis=1)\n", "\n", "# one-hot encode categorical columns\n", "categorical_cols = ['price_bracket', 'profile_visibility']\n", "users_df = pd.get_dummies(users_df, columns=categorical_cols)\n", "\n", "# convert boolean columns to integers\n", "boolean_cols = users_df.select_dtypes(include=['boolean']).columns\n", "users_df[boolean_cols] = users_df[boolean_cols].astype(int)\n", "\n", "# combine all numerical columns into a single feature vector\n", "numerical_cols = users_df.select_dtypes(include=['int64', 'uint8']).columns\n", "users_df['feature_vector'] = users_df[numerical_cols].values.tolist()\n", "users_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Because two tower models are also trained on interaction data like our SVD collaborative filtering model we need to generate some purchases.\n", "\n", "This will be a 1 or -1 to indicate if a user has eaten at this restaurant before.\n", "\n", "Once again we're generating random labels for this example to go along with our random users." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "import random\n", "\n", "user_ids = users_df['user_id'].tolist()\n", "restaurant_names = df[\"unique_name\"].tolist()\n", "\n", "# generate purchases by randomly selecting users and businesses\n", "purchases = [\n", " (user_ids[random.randrange(0, len(user_ids))],\n", " restaurant_names[random.randrange(0, len(restaurant_names))]\n", " )\n", " for _ in range(200)\n", "]\n", "\n", "positive_labels = []\n", "for i in range(len(purchases)):\n", " user_index = users_df[users_df['user_id'] == purchases[i][0]].index.item()\n", " restaurant_index = df[df['unique_name'] == purchases[i][1]].index.item()\n", " positive_labels.append((user_index, restaurant_index, 1.))\n", "\n", "# generate an equal number of negative examples\n", "negative_labels = []\n", "for i in range(len(purchases)):\n", " user_index = random.randint(0, len(user_ids)-1)\n", " restaurant_index = random.randint(0, len(restaurant_names)-1)\n", " negative_labels.append((user_index, restaurant_index, -1.))\n", "\n", "labels = positive_labels + negative_labels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have all of our data. The next steps are to define a the model and train it." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "import torch\n", "from torch.utils.data import DataLoader, Dataset\n", "\n", "import torch.nn as nn\n", "import torch.optim as optim\n", "\n", "# define a custom dataset\n", "class PurchaseDataset(Dataset):\n", " def __init__(self, user_features, restaurant_features, labels):\n", " self.user_features = user_features\n", " self.restaurant_features = restaurant_features\n", " self.labels = labels\n", "\n", " def __len__(self):\n", " return len(self.labels)\n", "\n", " def __getitem__(self, idx):\n", " user_index, restaurant_index, label = self.labels[idx]\n", " return self.user_features[user_index], self.restaurant_features[restaurant_index], torch.tensor(label, dtype=torch.float32)\n", "\n", "# define the two tower model\n", "class TwoTowerModel(nn.Module):\n", " def __init__(self, user_input_dim, restaurant_input_dim, hidden_dim):\n", " super(TwoTowerModel, self).__init__()\n", " self.user_tower = nn.Sequential(\n", " nn.Linear(user_input_dim, hidden_dim),\n", " nn.ReLU(),\n", " nn.Dropout(p=0.5),\n", " nn.Linear(hidden_dim, hidden_dim),\n", " nn.ReLU(),\n", " nn.Dropout(p=0.5),\n", " nn.Linear(hidden_dim, hidden_dim),\n", " )\n", " self.restaurant_tower = nn.Sequential(\n", " nn.Linear(restaurant_input_dim, hidden_dim),\n", " nn.ReLU(),\n", " nn.Dropout(p=0.5),\n", " nn.Linear(hidden_dim, hidden_dim),\n", " nn.ReLU(),\n", " nn.Dropout(p=0.5),\n", " nn.Linear(hidden_dim, hidden_dim),\n", " )\n", "\n", " def get_user_embeddings(self, user_features):\n", " return nn.functional.normalize(self.user_tower(user_features), dim=1)\n", "\n", " def get_restaurant_embeddings(self, restaurant_features):\n", " return nn.functional.normalize(self.restaurant_tower(restaurant_features), dim=1)\n", "\n", " def forward(self, user_features, restaurant_features):\n", " user_embedding = self.get_user_embeddings(user_features)\n", " restaurant_embedding = self.get_restaurant_embeddings(restaurant_features)\n", " return user_embedding, restaurant_embedding" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# prepare the data and data loader\n", "user_features = torch.tensor(users_df['feature_vector'].tolist(), dtype=torch.float32)\n", "restaurant_features = torch.tensor(df['feature_vector'].tolist(), dtype=torch.float32)\n", "\n", "dataset = PurchaseDataset(user_features, restaurant_features, labels)\n", "dataloader = DataLoader(dataset, batch_size=64, shuffle=True)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch [1/200], loss: 0.4948478043079376\n", "epoch [11/200], loss: 0.5350240468978882\n", "epoch [21/200], loss: 0.32322847843170166\n", "epoch [31/200], loss: 0.47431042790412903\n", "epoch [41/200], loss: 0.39620476961135864\n", "epoch [51/200], loss: 0.43342289328575134\n", "epoch [61/200], loss: 0.1380709707736969\n", "epoch [71/200], loss: 0.25389307737350464\n", "epoch [81/200], loss: 0.029272809624671936\n", "epoch [91/200], loss: 0.3498039245605469\n", "epoch [101/200], loss: 0.303999662399292\n", "epoch [111/200], loss: 0.3710485100746155\n", "epoch [121/200], loss: 0.1330445408821106\n", "epoch [131/200], loss: 0.14256471395492554\n", "epoch [141/200], loss: 0.16317707300186157\n", "epoch [151/200], loss: 0.3127524256706238\n", "epoch [161/200], loss: 0.26822173595428467\n", "epoch [171/200], loss: 0.13817712664604187\n", "epoch [181/200], loss: 0.31456106901168823\n", "epoch [191/200], loss: 0.3622739613056183\n" ] } ], "source": [ "# initialize the model, loss function and optimizer\n", "model = TwoTowerModel(user_input_dim=user_features.shape[1], restaurant_input_dim=restaurant_features.shape[1], hidden_dim=128)\n", "cosine_criterion = nn.CosineEmbeddingLoss()\n", "\n", "optimizer = optim.Adam(model.parameters(), lr=0.001)\n", "\n", "# train model\n", "num_epochs = 200\n", "losses = []\n", "for epoch in range(num_epochs):\n", " for user_batch, restaurant_batch, label_batch in dataloader:\n", " optimizer.zero_grad()\n", " user_embeddings, restaurant_embeddings = model(user_batch, restaurant_batch)\n", " loss = cosine_criterion(user_embeddings, restaurant_embeddings, label_batch)\n", " loss.backward()\n", " optimizer.step()\n", " if epoch % 10 == 0:\n", " print(f'epoch [{epoch+1}/{num_epochs}], loss: {loss.item()}')\n", " losses.append(loss.item())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why use two towers instead of content or collaborative filtering?\n", "This seems rather complicated compared to other recommender system architectures, so why go through with all of this effort? The best way to answer this is to compare with other recommendation system approaches.\n", "\n", "### Shortcomings of content filtering\n", "The simplest machine learning approach to recommendations is content filtering. It's also an approach that doesn't take into account user behaviors beyond finding similar content. This may not sound too bad, but can quickly lead to users getting trapped into content bubbles, where once they interact with a certain item - even if it was just randomly - they only see similar items.\n", "\n", "### Shortcomings of collaborative filtering\n", "Collaborative filtering approaches like Singular Value Decomposition (SVD) take the opposite approach and _only_ consider user behaviors to make recommendations. This has clear advantages, but one major drawback; SVD can't handle brand new users or brand new content. Each time a new user joins, or a new content is added to your library they won't have associated vectors. There also won't be meaningful new interaction data to re-train a model and generate vectors. It can be bad enough that a model needs frequent re-training; it can be an even bigger issue if you can't make recommendations for new users and content.\n", "\n", "Two tower models overcome these obstacles and to better understand how let's dive into what this type of architectures is really doing." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Two Towers Separate Embedding Vector Creation from Model Training\n", "Now that we have a trained model we can use each tower in our two tower model to generate embeddings for our users and items.\n", "Unlike SVD, we don't have to retrain our model to get these vectors. We also don't need new interaction data for our users or content." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "user_embeddings = model.get_user_embeddings(user_features=torch.tensor(users_df['feature_vector'].tolist(), dtype=torch.float32))\n", "restaurant_embeddings = model.get_restaurant_embeddings(restaurant_features=torch.tensor(df['feature_vector'].tolist(), dtype=torch.float32))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Best of Both Worlds\n", "Two tower models are a triple whammy when it comes to solving the above problems:\n", "- They are trained on interaction data, aka our labels, so learn not to fall into content bubbles\n", "- They directly consider the user features _and_ content features\n", "- they can handle brand new users and content that don't yet have interaction data. No retraining necessary\n", "\n", "While we need some interaction data to train our model initially, it's totally fine if not all users or restaurants are included in our labelled data. Only a sample is needed.\n", "This is why we can handle new users and content without retraining. Only their raw features are needed to generate embeddings\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading into Redis\n", "With two sets of vectors we'll load the restaurant data into a Redis vector store to search over, and the user vectors into a regular key look up for quick access.\n", "We'll handle our restaurants opening and closing hours, as well as their location in longitude and latitude. We'll want these for later." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'name': '21st Amendment Brewery & Restaurant', 'address': '563 2nd St', 'locality': 'San Francisco', 'location': '-122.392576,37.782448', 'cuisine': ['Cafe', 'Pub Food', 'American', 'Burgers', 'Pizza'], 'price': 2, 'rating': 4.0, 'sunday_open': 1000, 'sunday_close': 2359, 'monday_open': 1130, 'monday_close': 2359, 'tuesday_open': 1130, 'tuesday_close': 2359, 'wednesday_open': 1130, 'wednesday_close': 2359, 'thursday_open': 1130, 'thursday_close': 2359, 'friday_open': 1130, 'friday_close': 2359, 'saturday_open': 1130, 'saturday_close': 2359, 'embedding': [0.04085610806941986, -0.07978134602308273, 0.043692223727703094, 0.07572835683822632, 0.012072183191776276, -0.13669149577617645, -0.025233915075659752, -0.08203943073749542, 0.004281577654182911, 0.053853441029787064, 0.10153290629386902, -0.0029142447747290134, -0.03880758956074715, -0.047814156860113144, -0.06192268431186676, -0.013051072135567665, 0.08474208414554596, 0.11760099232196808, -0.04501175880432129, 0.036522794514894485, 0.07012218236923218, 0.07932834327220917, -0.11235840618610382, -0.06628117710351944, -0.036081865429878235, 0.10264216363430023, 0.006768162362277508, -0.1377549171447754, 0.11204114556312561, -0.05779130011796951, 0.1014084592461586, -0.011839451268315315, 0.06367754936218262, 0.1345064789056778, 0.04285123571753502, 0.12564392387866974, -0.018177764490246773, -0.023292746394872665, -0.11306607723236084, 0.07133293896913528, -0.0793255865573883, 0.10723698139190674, 0.025939466431736946, 0.005317146424204111, 0.08136926591396332, -0.08177289366722107, 0.1532663106918335, -0.01752050220966339, -0.10502904653549194, -0.11620310693979263, 0.030231507495045662, 0.14732813835144043, 0.005023199133574963, -0.09555873274803162, 0.15709209442138672, -0.02062702737748623, 0.04334118962287903, -0.0390237420797348, 0.02523123100399971, 0.0641607716679573, 0.016466626897454262, -0.08375702798366547, -0.1619223952293396, -0.09513241052627563, -0.024845421314239502, 0.029008952900767326, -0.025478294119238853, -0.027486223727464676, 0.06510215252637863, -0.16730202734470367, 0.10255678743124008, 0.08258558064699173, -0.16539154946804047, 0.05780792608857155, 0.08961871266365051, -0.11477161943912506, -0.0031035130377858877, 0.10795316845178604, -0.06544432789087296, 0.06262046098709106, -0.035685282200574875, -0.12535151839256287, 0.15087392926216125, 0.054519712924957275, -0.06867150962352753, 0.01560500543564558, 0.1242009848356247, -0.03294835612177849, 0.09918906539678574, 0.07899592816829681, 0.016121378168463707, 0.012538410723209381, 0.04713483154773712, -0.016400208696722984, 0.1835649609565735, 0.16933025419712067, -0.06303106993436813, 0.09937309473752975, -0.07579631358385086, 0.11997299641370773, 0.01682763174176216, -0.09757581353187561, 0.09561226516962051, -0.04486346244812012, -0.16034992039203644, -0.10858193784952164, -0.026912961155176163, 0.11371364444494247, 0.16520418226718903, -0.08449505269527435, 0.11462333798408508, -0.0775909498333931, 0.0306625384837389, -0.061352867633104324, 0.03166527301073074, 0.013918583281338215, -0.13196627795696259, -0.084035724401474, -0.14768821001052856, 0.07229673862457275, 0.12507523596286774, 0.09107068926095963, 0.011143358424305916, -0.17356498539447784, 0.1589767187833786, 0.028815122321248055, -0.10924888402223587, 0.044692039489746094]}\n" ] } ], "source": [ "# extract opening and closing times from the 'hours' column\n", "def extract_opening_closing_times(hours, day):\n", " # convert to a simple numeric representation of times\n", " if day in hours:\n", " return int(hours[day][0][0].replace(':','')), int(hours[day][0][1].replace(':',''))\n", " else:\n", " # we don't know their hours, assume a reasonable default of 9:00am to 8:00pm\n", " return 900, 2000\n", "\n", "# create new columns for opening and closing times for each day of the week\n", "for day in ['sunday', 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday']:\n", " df[f'{day}_open'], df[f'{day}_close'] = zip(*df['hours'].apply(lambda x: extract_opening_closing_times(x, day)))\n", "\n", "# combine 'longitude' and 'latitude' into a single 'location' column\n", "df['location'] = df.apply(lambda row: f\"{row['longitude']},{row['latitude']}\", axis=1)\n", "\n", "# drop the original 'hours' separate 'latitude' and 'longitude' columns as we don't need them anymore\n", "df.drop(columns=['hours', 'latitude', 'longitude'], inplace=True)\n", "\n", "# ensure the 'embedding' column is in the correct format (list of floats)\n", "df['embedding'] = restaurant_embeddings.detach().numpy().tolist()\n", "\n", "# ensure all columns are in the correct order as defined in the schema\n", "df = df[['name', 'address', 'locality', 'location', 'cuisine', 'price', 'rating', 'sunday_open', 'sunday_close', 'monday_open', 'monday_close', 'tuesday_open', 'tuesday_close', 'wednesday_open', 'wednesday_close', 'thursday_open', 'thursday_close', 'friday_open', 'friday_close', 'saturday_open', 'saturday_close', 'embedding']]\n", "\n", "# print the first record to verify the format\n", "print(df.to_dict(orient='records')[0])" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from redis import Redis\n", "from redisvl.schema import IndexSchema\n", "from redisvl.index import SearchIndex\n", "\n", "client = Redis.from_url(REDIS_URL)\n", "\n", "restaurant_schema = IndexSchema.from_dict({\n", " 'index': {\n", " 'name': 'restaurants',\n", " 'prefix': 'restaurant',\n", " 'storage_type': 'json'\n", " },\n", " 'fields': [\n", " {'name': 'name', 'type': 'text'},\n", " {'name': 'address', 'type': 'text'},\n", " {'name': 'locality', 'type': 'tag'},\n", " {'name': 'location', 'type': 'geo'},\n", " {'name': 'cuisine', 'type': 'tag'},\n", " {'name': 'price', 'type': 'numeric'},\n", " {'name': 'rating', 'type': 'numeric'},\n", " {'name': 'sunday_open', 'type': 'numeric'},\n", " {'name': 'sunday_close', 'type': 'numeric'},\n", " {'name': 'monday_open', 'type': 'numeric'},\n", " {'name': 'monday_close', 'type': 'numeric'},\n", " {'name': 'tuesday_open', 'type': 'numeric'},\n", " {'name': 'tuesday_close', 'type': 'numeric'},\n", " {'name': 'wednesday_open', 'type': 'numeric'},\n", " {'name': 'wednesday_close', 'type': 'numeric'},\n", " {'name': 'thursday_open', 'type': 'numeric'},\n", " {'name': 'thursday_close', 'type': 'numeric'},\n", " {'name': 'friday_open', 'type': 'numeric'},\n", " {'name': 'friday_close', 'type': 'numeric'},\n", " {'name': 'saturday_open', 'type': 'numeric'},\n", " {'name': 'saturday_close', 'type': 'numeric'},\n", " {\n", " 'name': 'embedding',\n", " 'type': 'vector',\n", " 'attrs': {\n", " 'dims': 128,\n", " 'algorithm': 'flat',\n", " 'datatype': 'float32',\n", " 'distance_metric': 'cosine'\n", " }\n", " }\n", " ]\n", "})\n", "\n", "restaurant_index = SearchIndex(restaurant_schema, redis_client=client)\n", "restaurant_index.create(overwrite=True, drop=True)\n", "\n", "restaurant_keys = restaurant_index.load(df.to_dict(orient='records'))" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "# load the user vectors into a regular redis space\n", "from redis.commands.json.path import Path\n", "\n", "with client.pipeline() as pipe:\n", " for user_id, embedding in zip(users_df['user_id'], user_embeddings):\n", " user_key = f\"user:{user_id}\"\n", "\n", " user_data = {\n", " \"user_embedding\": embedding.tolist(),\n", " }\n", " pipe.json().set(user_key, Path.root_path(), user_data)\n", " pipe.execute()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The power of deep learning with the speed of Redis\n", "\n", "I can hear you say it, \"deep learning is cool and all, but I need my system to be fast. I don't want to call a deep neural network to get recommendations.\"\n", "\n", "Well not to fear my friend, you won't have to! While training our model may take a while, you won't need to do this often.\n", "And if you look closely you'll see that both the user and content embedding vectors can be generated once and reused again and again.\n", "Only the vector search is happening when generating recommendations.\n", "These embeddings will only change if your user or content features change and if you select your features wisely this won't be often." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Location Aware Recommendations\n", "\n", "We've shown how Redis can apply filters on top of vector similarity search to further refine results, but did you know it can also refine search results by location?\n", "Using the `Geo` field type on our index definition we can apply a `GeoRadius` filter to find only places nearby, which seems mighty useful for a restaurant recommendation system.\n", "\n", "Combining `GeoRadius` with `Num` tags we can find places that are personally relevant to us, nearby _and_ open for business right now.\n", "\n", "We have all our data and vectors ready to go. Now let's put it all together with query logic." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "from redisvl.query.filter import Tag, Num, Geo, GeoRadius\n", "import datetime\n", "\n", "def get_filter(user_long,\n", " user_lat,\n", " current_date_time,\n", " radius=1000,\n", " low_price=0.0,\n", " high_price=5.0,\n", " rating=0.0,\n", " cuisines=[]):\n", "\n", " geo_filter = Geo(\"location\") == GeoRadius(user_long, user_lat, radius, unit=\"m\") # use a distance unit of meters\n", "\n", " open_filter = Num(f\"{current_date_time.strftime('%A').lower()}_open\") < current_date_time.hour*100 + current_date_time.minute\n", " close_filter = Num(f\"{current_date_time.strftime('%A').lower()}_close\") > current_date_time.hour*100 + current_date_time.minute\n", " time_filter = open_filter & close_filter\n", "\n", " price_filter = (Num('price') >= low_price) & (Num('price') <= high_price)\n", "\n", " rating_filter = Num('rating') >= rating\n", "\n", " cuisine_filter = Tag('cuisine') == cuisines\n", "\n", " return geo_filter & time_filter & price_filter & rating_filter & cuisine_filter\n" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "found 9 results from our query\n", "{'id': 'restaurant:38e3523a7e44410e9f9dadeaaedf7bcd', 'name': 'La Boulange', 'address': '2043 Fillmore St', 'location': '-122.43386,37.788408'}\n", "{'id': 'restaurant:6ed782a2b4ae49ff951fffc6b061fe23', 'name': 'Pizzeria Delfina', 'address': '2406 California St', 'location': '-122.434241,37.788925'}\n", "{'id': 'restaurant:5021e47f61f74bef8d0c35580c295a91', 'name': 'Burgermeister', 'address': '138 Church St', 'location': '-122.42914,37.768755'}\n", "{'id': 'restaurant:9d29fd36d9b34ac08b25cefe2c59bc58', 'name': 'Double Decker', 'address': '465 Grove St', 'location': '-122.424033,37.777531'}\n", "{'id': 'restaurant:631427789b384b2b82aaad718d6df0ff', 'name': 'Cafe Du Soleil', 'address': '200 Fillmore St', 'location': '-122.430158,37.771303'}\n", "{'id': 'restaurant:48c70513a1f44065ace6f4c9ea4b8f20', 'name': 'Nanis Coffee', 'address': '2739 Geary Blvd', 'location': '-122.448613,37.782187'}\n", "{'id': 'restaurant:d1959bf9ddb24edaa1bc0bd261d57cc6', 'name': 'Memphis Minnies BBQ Joint', 'address': '576 Haight St', 'location': '-122.431702,37.772058'}\n", "{'id': 'restaurant:d9b546847eb14ea28d6181fd68e93127', 'name': 'Panini', 'address': '1457 Haight St', 'location': '-122.44629,37.770036'}\n", "{'id': 'restaurant:1e1649915f734ebfae7f6c6e08ce5184', 'name': 'Magnolia Pub and Brewery', 'address': '1398 Haight St', 'location': '-122.445238,37.770276'}\n" ] } ], "source": [ "from redisvl.query import VectorQuery\n", "\n", "random_user = random.choice(users_df['user_id'].tolist())\n", "user_vector = client.json().get(f\"user:{random_user}\")[\"user_embedding\"]\n", "\n", "# get a location for this user. Your app may call an API, here we'll set one randomly to within San Francisco\n", "# San Francisco is within the longitude and latitude bounding box of:\n", "# Lower corner: (-122.5137, 37.7099) in (longitude, latitude) format\n", "# Upper corner: (-122.3785, 37.8101)\n", "\n", "longitude = random.uniform(-122.5137, -122.3785)\n", "latitude = random.uniform(37.7099, 37.8101)\n", "longitude, latitude = -122.439, 37.779\n", "radius = 1500\n", "\n", "full_filter = get_filter(user_long=longitude,\n", " user_lat=latitude,\n", " radius=radius,\n", " current_date_time=datetime.datetime.today())\n", "\n", "query = VectorQuery(vector=user_vector,\n", " vector_field_name='embedding',\n", " num_results=10,\n", " return_score=False,\n", " return_fields=['name', 'address', 'location', 'distance'],\n", " filter_expression=full_filter,\n", " )\n", "\n", "results = restaurant_index.query(query)\n", "print(f\"found {len(results)} results from our query\")\n", "for r in results:\n", " print(r)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Seeing Is Believing\n", "\n", "With our vectors loaded and helper functions defined we can get some nearby recommendations. That's all well and good, but don't you wish you could see these recommendations? I sure do. So let's visualize them on an interactive map." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install folium --quiet" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import folium\n", "from IPython.display import display\n", "\n", "# create a map centered around San Francisco\n", "figure = folium.Figure(width=700, height=600)\n", "sf_map = folium.Map(location=[37.7749, -122.4194],\n", " zoom_start=13,\n", " max_bounds=True,\n", " min_lat= 37.709 - 0.1,\n", " max_lat= 37.8101 + 0.1,\n", " min_lon= -122.3785 - 0.3,\n", " max_lon= -122.5137 + 0.3,\n", " )\n", "\n", "sf_map.add_to(figure)\n", "\n", "# add markers for each restaurant in blue\n", "for idx, row in df.iterrows():\n", " lat, lon = map(float, row['location'].split(','))\n", " folium.Marker([lon, lat], popup=row['name']).add_to(sf_map)\n", "\n", "\n", "# get personalized recommendations\n", "user = users_df['user_id'].tolist()[42]\n", "user_vector = client.json().get(f\"user:{user}\")[\"user_embedding\"]\n", "\n", "# get a location for this user. Your app may call an API, here we'll set one randomly to within San Francisco\n", "# lower corner: (-122.5137, 37.7099) in (longitude, latitude) format\n", "# upper corner: (-122.3785, 37.8101)\n", "\n", "longitude, latitude = -122.439, 37.779\n", "num_results = 25\n", "radius = 2000\n", "\n", "# draw a circle centered on our user\n", "folium.Circle(\n", " location=[latitude, longitude],\n", " radius=radius,\n", " color=\"green\",\n", " weight=3,\n", " fill=True,\n", " fill_opacity=0.3,\n", " opacity=1,\n", ").add_to(sf_map)\n", "\n", "\n", "full_filter = get_filter(user_long=longitude,\n", " user_lat=latitude,\n", " radius=radius,\n", " current_date_time=datetime.datetime.today()\n", " )\n", "\n", "query = VectorQuery(vector=user_vector,\n", " vector_field_name='embedding',\n", " num_results=num_results,\n", " return_score=False,\n", " return_fields=['name', 'address', 'location', 'rating'],\n", " filter_expression=full_filter,\n", " )\n", "\n", "results = restaurant_index.query(query)\n", "\n", "# now show our recommended places in red\n", "for restaurant in results:\n", " lat, lon = map(float, restaurant['location'].split(','))\n", " folium.Marker([lon, lat], popup=restaurant['name'] + ' ' + restaurant['rating'] + ' stars', icon=folium.Icon(color='red')).add_to(sf_map)\n", "\n", "display(sf_map)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "That's it! You've built a deep learning restaurant recommendation system with Redis. It's personalized, location aware, adaptable, and fast." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deleted 147 keys\n" ] }, { "data": { "text/plain": [ "1000" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# clean up your index\n", "while remaining := restaurant_index.clear():\n", " print(f\"Deleted {remaining} keys\")\n", "\n", "client.delete(*[f\"user:{user_id}\" for user_id in users_df['user_id'].tolist()])" ] } ], "metadata": { "kernelspec": { "display_name": "redis-ai-res", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 2 }