{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "# Card Transaction Search and Analytics with RedisVL\n", "\n", "In this recipe, we will explore a dataset of card transactions generated by multiple users, across multiple vendors over a period of time. We will showcase the power, speed, and flexibility of the Redis Query Engine for search, filtering, vector similarity, and complex aggregations using RedisVL (Redis Vector Library).\n", "\n", "Transaction search and analytics have many use cases - but primarily this data is useful for building realtime feature stores that can power fraud or anomaly detection machine learning models.\n", "\n", "## What we'll cover\n", "1. Loading transaction data into Redis\n", "2. Vectorizing transaction data for semantic similarity search\n", "3. Search techniques\n", " - Exact match filtering\n", " - Vector search\n", " - Full text search and fuzzy matching\n", " - Hybrid search\n", "4. Complex aggregation queries\n", " - Calculate average transaction volume per week\n", " - Identify spending patterns\n", " - Generate user spending profiles" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Let's Begin!\n", "\"Open" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare data\n", "\n", "Our dataset is a list of 200 credit card transactions (fake)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n", "!mv temp_repo/python-recipes/feature-store/resources .\n", "!rm -rf temp_repo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Install Required Packages" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -q \"redisvl==0.6.0\" sentence-transformers pandas nltk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Install Redis Stack\n", "\n", "In this tutorial, Redis will be used to store, index, and query vector\n", "embeddings created from transaction data. **We need to make sure we have a Redis\n", "instance available**.\n", "\n", "#### For Colab\n", "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "%%sh\n", "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", "sudo apt-get update > /dev/null 2>&1\n", "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", "redis-stack-server --daemonize yes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### For Alternative Environments\n", "There are many ways to get the necessary redis-stack instance running\n", "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", "own version of Redis Enterprise running, that works too!\n", "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define the Redis Connection URL\n", "\n", "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import os\n", "import warnings\n", "\n", "warnings.filterwarnings('ignore')\n", "\n", "# Replace values below with your own if using Redis Cloud instance\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", "\n", "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Redis client and test connection" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redis import Redis\n", "\n", "client = Redis.from_url(REDIS_URL)\n", "client.ping()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Optional: clear all data in Redis if needed\n", "client.flushall()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load Transaction Dataset" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded 200 transaction entries\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
transaction_iduser_idmerchant_iditem_nameamountcurrencytimestamplatloncard_providerlocation
0txn_0001u_002m_009Headphones1154.59USD174618255127.584806-71.730465VISA-71.730465, 27.584806
1txn_0002u_013m_018Dinner501.64USD174697095128.831898-104.441434AMEX-104.441434, 28.831898
2txn_0003u_008m_006Laptop1359.33USD174684135146.087128-102.099503VISA-102.099503, 46.087128
3txn_0004u_011m_024Gaming Console157.54USD174700335127.226349-115.753846VISA-115.753846, 27.226349
4txn_0005u_010m_014Concert Ticket718.00USD174543375145.108103-79.409905AMEX-79.409905, 45.108103
\n", "
" ], "text/plain": [ " transaction_id user_id merchant_id item_name amount currency \\\n", "0 txn_0001 u_002 m_009 Headphones 1154.59 USD \n", "1 txn_0002 u_013 m_018 Dinner 501.64 USD \n", "2 txn_0003 u_008 m_006 Laptop 1359.33 USD \n", "3 txn_0004 u_011 m_024 Gaming Console 157.54 USD \n", "4 txn_0005 u_010 m_014 Concert Ticket 718.00 USD \n", "\n", " timestamp lat lon card_provider location \n", "0 1746182551 27.584806 -71.730465 VISA -71.730465, 27.584806 \n", "1 1746970951 28.831898 -104.441434 AMEX -104.441434, 28.831898 \n", "2 1746841351 46.087128 -102.099503 VISA -102.099503, 46.087128 \n", "3 1747003351 27.226349 -115.753846 VISA -115.753846, 27.226349 \n", "4 1745433751 45.108103 -79.409905 AMEX -79.409905, 45.108103 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "# Load transactions from JSON file\n", "df = pd.read_json(\"resources/transactions_200.json\")\n", "print(f\"Loaded {len(df)} transaction entries\")\n", "\n", "# # Convert timestamp to datetime for easier manipulation\n", "df[\"timestamp\"] = df[\"timestamp\"].apply(lambda s: int(pd.to_datetime(s).timestamp()))\n", "df['location'] = df.apply(lambda r: f\"{r.lon}, {r.lat}\", axis=1)\n", "\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's examine the transaction data" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 200.00000\n", "mean 747.57135\n", "std 426.08199\n", "min 26.63000\n", "25% 373.06250\n", "50% 696.15500\n", "75% 1130.19750\n", "max 1499.87000\n", "Name: amount, dtype: float64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Basic statistics on transaction amounts\n", "df['amount'].describe()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "card_provider\n", "DISCOVER 54\n", "AMEX 52\n", "VISA 51\n", "MASTERCARD 43\n", "Name: count, dtype: int64" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Count of transactions by card provider\n", "df['card_provider'].value_counts()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "item_name\n", "Plane Ticket 15\n", "Hotel Stay 15\n", "Groceries 14\n", "Dinner 14\n", "Headphones 13\n", "Gym Membership 13\n", "Bicycle 12\n", "Gaming Console 11\n", "Streaming Subscription 11\n", "Smartphone 9\n", "Name: count, dtype: int64" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Look at the most common items purchased\n", "df['item_name'].value_counts().head(10)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "user_id\n", "u_013 16\n", "u_012 14\n", "u_008 13\n", "u_006 13\n", "u_014 12\n", "u_018 12\n", "u_007 11\n", "u_011 10\n", "u_010 10\n", "u_020 10\n", "u_009 9\n", "u_002 9\n", "u_016 9\n", "u_001 9\n", "u_017 8\n", "u_015 8\n", "u_005 7\n", "u_019 7\n", "u_003 7\n", "u_004 6\n", "Name: count, dtype: int64" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Look at how many users there are\n", "df['user_id'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Vectorize Transaction Data for Semantic Search\n", "\n", "We'll use a Hugging Face sentence transformer to create vector embeddings for transaction data. The text we'll vectorize will be a combination of:\n", "- Merchant name\n", "- Item purchased\n", "- Transaction amount" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "13:15:47 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n", "13:15:47 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Batches: 100%|██████████| 1/1 [00:00<00:00, 7.46it/s]\n" ] } ], "source": [ "from redisvl.utils.vectorize import HFTextVectorizer\n", "\n", "# Set environment variable to avoid parallelism warnings\n", "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n", "\n", "# Initialize the vectorizer with a small but powerful model\n", "hf = HFTextVectorizer(\"sentence-transformers/all-MiniLM-L6-v2\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
transaction_idvector_text
0txn_0001Merchant m_009 selling Headphones for $1154.59
1txn_0002Merchant m_018 selling Dinner for $501.64
2txn_0003Merchant m_006 selling Laptop for $1359.33
\n", "
" ], "text/plain": [ " transaction_id vector_text\n", "0 txn_0001 Merchant m_009 selling Headphones for $1154.59\n", "1 txn_0002 Merchant m_018 selling Dinner for $501.64\n", "2 txn_0003 Merchant m_006 selling Laptop for $1359.33" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a combined text field for vectorization\n", "def create_text_for_vectorization(row):\n", " return f\"Merchant {row['merchant_id']} selling {row['item_name']} for ${row['amount']:.2f}\"\n", "\n", "df['vector_text'] = df.apply(create_text_for_vectorization, axis=1)\n", "\n", "# Display some examples\n", "df[['transaction_id', 'vector_text']].head(3)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Generating vectors for transactions...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Batches: 100%|██████████| 1/1 [00:00<00:00, 8.56it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 13.23it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 77.89it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 70.84it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 68.89it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 68.21it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 76.25it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 82.86it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 87.50it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 72.29it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 74.83it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 70.85it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 72.97it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 83.08it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 87.37it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 85.30it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 73.55it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 77.35it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 78.44it/s]\n", "Batches: 100%|██████████| 1/1 [00:00<00:00, 79.18it/s]\n" ] } ], "source": [ "# Generate vectors for each transaction\n", "print(\"Generating vectors for transactions...\")\n", "df[\"vector\"] = hf.embed_many(df[\"vector_text\"].tolist(), as_buffer=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Define Redis Index Schema\n", "\n", "We'll create a schema that includes both standard fields and vector field for our transaction data." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from redisvl.schema import IndexSchema\n", "from redisvl.index import SearchIndex\n", "\n", "\n", "# Define the index schema with fields we want to search and filter on\n", "schema = IndexSchema.from_dict({\n", " \"index\": {\n", " \"name\": \"transactions\",\n", " \"prefix\": \"transactions:entry\",\n", " \"storage_type\": \"hash\"\n", " },\n", " \"fields\": [\n", " {\n", " \"name\": \"transaction_id\",\n", " \"type\": \"tag\",\n", " \"attrs\": {\n", " \"sortable\": True\n", " }\n", " },\n", " {\n", " \"name\": \"user_id\",\n", " \"type\": \"tag\",\n", " \"attrs\": {\n", " \"sortable\": True\n", " }\n", " },\n", " {\n", " \"name\": \"merchant_id\",\n", " \"type\": \"tag\",\n", " \"attrs\": {\n", " \"sortable\": True\n", " }\n", " },\n", " {\n", " \"name\": \"item_name\",\n", " \"type\": \"text\",\n", " \"attrs\": {\n", " \"sortable\": True\n", " }\n", " },\n", " {\n", " \"name\": \"amount\",\n", " \"type\": \"numeric\",\n", " \"attrs\": {\n", " \"sortable\": True\n", " }\n", " },\n", " {\n", " \"name\": \"currency\",\n", " \"type\": \"tag\",\n", " },\n", " {\n", " \"name\": \"timestamp\",\n", " \"type\": \"numeric\",\n", " \"attrs\": {\n", " \"sortable\": True\n", " }\n", " },\n", " {\n", " \"name\": \"card_provider\",\n", " \"type\": \"tag\",\n", " },\n", " {\n", " \"name\": \"location\",\n", " \"type\": \"geo\",\n", " },\n", " {\n", " \"name\": \"vector\",\n", " \"type\": \"vector\",\n", " \"attrs\": {\n", " \"dims\": 384, # Based on the all-MiniLM-L6-v2 model\n", " \"distance_metric\": \"cosine\",\n", " \"algorithm\": \"flat\",\n", " \"datatype\": \"float32\"\n", " }\n", " }\n", " ]\n", "})\n", "\n", "# Create the index\n", "index = SearchIndex(schema, client)\n", "index.create(overwrite=True, drop=True)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "Index Information:\n", "╭────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────╮\n", "│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │\n", "├────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┤\n", "| transactions | HASH | ['transactions:entry'] | [] | 0 |\n", "╰────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────╯\n", "Index Fields:\n", "╭─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────╮\n", "│ Name │ Attribute │ Type │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │\n", "├─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤\n", "│ transaction_id │ transaction_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │\n", "│ user_id │ user_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │\n", "│ merchant_id │ merchant_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │\n", "│ item_name │ item_name │ TEXT │ WEIGHT │ 1 │ │ │ │ │ │ │\n", "│ amount │ amount │ NUMERIC │ SORTABLE │ UNF │ │ │ │ │ │ │\n", "│ currency │ currency │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │\n", "│ timestamp │ timestamp │ NUMERIC │ SORTABLE │ UNF │ │ │ │ │ │ │\n", "│ card_provider │ card_provider │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │\n", "│ location │ location │ GEO │ │ │ │ │ │ │ │ │\n", "│ vector │ vector │ VECTOR │ algorithm │ FLAT │ data_type │ FLOAT32 │ dim │ 384 │ distance_metric │ COSINE │\n", "╰─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────╯\n" ] } ], "source": [ "# Check the index information\n", "!rvl index info -i transactions -u {REDIS_URL}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Populate Redis with Transaction Data\n", "\n", "Now that our index is created, let's load the transaction data into Redis." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded 200 transactions into Redis\n" ] }, { "data": { "text/plain": [ "['transactions:entry:txn_0001',\n", " 'transactions:entry:txn_0002',\n", " 'transactions:entry:txn_0003',\n", " 'transactions:entry:txn_0004',\n", " 'transactions:entry:txn_0005']" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load data into Redis\n", "transaction_ids = index.load(\n", " data=df.to_dict(orient=\"records\"),\n", " id_field=\"transaction_id\"\n", ")\n", "print(f\"Loaded {len(transaction_ids)} transactions into Redis\")\n", "\n", "# Display the first few transaction IDs loaded\n", "transaction_ids[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Part I: Transaction Search Techniques\n", "\n", "Now that we have our data loaded into Redis, let's explore different search techniques." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Exact Match Queryies & Sorting\n", "\n", "Let's start with some basic exact match filtering to find transactions with specific properties." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtransaction_iduser_idmerchant_iditem_nameamountcard_provider
0transactions:entry:txn_0002txn_0002u_013m_018Dinner501.64AMEX
1transactions:entry:txn_0005txn_0005u_010m_014Concert Ticket718AMEX
2transactions:entry:txn_0006txn_0006u_017m_016Hotel Stay1232.8AMEX
3transactions:entry:txn_0015txn_0015u_001m_018Clothing114.86AMEX
4transactions:entry:txn_0032txn_0032u_013m_002Concert Ticket585.69AMEX
\n", "
" ], "text/plain": [ " id transaction_id user_id merchant_id \\\n", "0 transactions:entry:txn_0002 txn_0002 u_013 m_018 \n", "1 transactions:entry:txn_0005 txn_0005 u_010 m_014 \n", "2 transactions:entry:txn_0006 txn_0006 u_017 m_016 \n", "3 transactions:entry:txn_0015 txn_0015 u_001 m_018 \n", "4 transactions:entry:txn_0032 txn_0032 u_013 m_002 \n", "\n", " item_name amount card_provider \n", "0 Dinner 501.64 AMEX \n", "1 Concert Ticket 718 AMEX \n", "2 Hotel Stay 1232.8 AMEX \n", "3 Clothing 114.86 AMEX \n", "4 Concert Ticket 585.69 AMEX " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redisvl.query import FilterQuery\n", "from redisvl.query.filter import Tag, Num\n", "\n", "# Find all AMEX transactions\n", "card_filter = Tag(\"card_provider\") == \"AMEX\"\n", "\n", "query = FilterQuery(\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", " filter_expression=card_filter,\n", " num_results=5\n", ")\n", "\n", "results = index.query(query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idamounttransaction_iduser_idmerchant_iditem_namecard_provider
0transactions:entry:txn_00611499.87txn_0061u_006m_004GroceriesVISA
1transactions:entry:txn_01111471.73txn_0111u_014m_006CoffeeVISA
2transactions:entry:txn_01471462.78txn_0147u_018m_003DinnerMASTERCARD
3transactions:entry:txn_00191462.52txn_0019u_012m_005DinnerDISCOVER
4transactions:entry:txn_01681450.52txn_0168u_016m_014GroceriesAMEX
\n", "
" ], "text/plain": [ " id amount transaction_id user_id merchant_id \\\n", "0 transactions:entry:txn_0061 1499.87 txn_0061 u_006 m_004 \n", "1 transactions:entry:txn_0111 1471.73 txn_0111 u_014 m_006 \n", "2 transactions:entry:txn_0147 1462.78 txn_0147 u_018 m_003 \n", "3 transactions:entry:txn_0019 1462.52 txn_0019 u_012 m_005 \n", "4 transactions:entry:txn_0168 1450.52 txn_0168 u_016 m_014 \n", "\n", " item_name card_provider \n", "0 Groceries VISA \n", "1 Coffee VISA \n", "2 Dinner MASTERCARD \n", "3 Dinner DISCOVER \n", "4 Groceries AMEX " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Find high-value transactions (over $1000)\n", "amount_filter = Num(\"amount\") > 1000\n", "\n", "query = FilterQuery(\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", " filter_expression=amount_filter,\n", " num_results=5,\n", ").sort_by(\"amount\", asc=False)\n", "\n", "results = index.query(query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idamounttransaction_iduser_idmerchant_iditem_nametimestamp
0transactions:entry:txn_01901413.99txn_0190u_013m_005Plane Ticket1745570551
1transactions:entry:txn_01451382.3txn_0145u_013m_024Hotel Stay1746927751
2transactions:entry:txn_01031360.17txn_0103u_013m_015Coffee1744911751
3transactions:entry:txn_00671311.19txn_0067u_013m_010Headphones1746715351
4transactions:entry:txn_00651231.32txn_0065u_013m_012Plane Ticket1746675751
5transactions:entry:txn_00601094.44txn_0060u_013m_001Ride Share1744857751
6transactions:entry:txn_01501075.13txn_0150u_013m_003Plane Ticket1746812551
7transactions:entry:txn_01251032.48txn_0125u_013m_018Shoes1747143751
8transactions:entry:txn_0058916.96txn_0058u_013m_024Plane Ticket1745523751
9transactions:entry:txn_0113733.8txn_0113u_013m_001Software License1745566951
\n", "
" ], "text/plain": [ " id amount transaction_id user_id merchant_id \\\n", "0 transactions:entry:txn_0190 1413.99 txn_0190 u_013 m_005 \n", "1 transactions:entry:txn_0145 1382.3 txn_0145 u_013 m_024 \n", "2 transactions:entry:txn_0103 1360.17 txn_0103 u_013 m_015 \n", "3 transactions:entry:txn_0067 1311.19 txn_0067 u_013 m_010 \n", "4 transactions:entry:txn_0065 1231.32 txn_0065 u_013 m_012 \n", "5 transactions:entry:txn_0060 1094.44 txn_0060 u_013 m_001 \n", "6 transactions:entry:txn_0150 1075.13 txn_0150 u_013 m_003 \n", "7 transactions:entry:txn_0125 1032.48 txn_0125 u_013 m_018 \n", "8 transactions:entry:txn_0058 916.96 txn_0058 u_013 m_024 \n", "9 transactions:entry:txn_0113 733.8 txn_0113 u_013 m_001 \n", "\n", " item_name timestamp \n", "0 Plane Ticket 1745570551 \n", "1 Hotel Stay 1746927751 \n", "2 Coffee 1744911751 \n", "3 Headphones 1746715351 \n", "4 Plane Ticket 1746675751 \n", "5 Ride Share 1744857751 \n", "6 Plane Ticket 1746812551 \n", "7 Shoes 1747143751 \n", "8 Plane Ticket 1745523751 \n", "9 Software License 1745566951 " ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Combine filters: Find transactions for a specific user with high amounts\n", "user_filter = Tag(\"user_id\") == \"u_013\" # Specific user\n", "amount_filter = Num(\"amount\") > 500 # High amount threshold\n", "\n", "# Combine filters with logical AND\n", "combined_filter = user_filter & amount_filter\n", "\n", "query = FilterQuery(\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"timestamp\"],\n", " filter_expression=combined_filter,\n", ").sort_by(\"amount\", asc=False)\n", "\n", "results = index.query(query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Vector Search\n", "\n", "Now let's use vector search to find transactions semantically similar to a query." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Batches: 100%|██████████| 1/1 [00:00<00:00, 10.19it/s]\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idvector_distanceamounttransaction_idmerchant_iditem_name
0transactions:entry:txn_00570.5002093911171201.71txn_0057m_011Laptop
1transactions:entry:txn_00980.497323393822503.01txn_0098m_012Headphones
2transactions:entry:txn_01700.500393152237374.23txn_0170m_010Headphones
3transactions:entry:txn_00400.495004236698159.33txn_0040m_017Headphones
4transactions:entry:txn_01690.494512319565153.22txn_0169m_008Headphones
\n", "
" ], "text/plain": [ " id vector_distance amount transaction_id \\\n", "0 transactions:entry:txn_0057 0.500209391117 1201.71 txn_0057 \n", "1 transactions:entry:txn_0098 0.497323393822 503.01 txn_0098 \n", "2 transactions:entry:txn_0170 0.500393152237 374.23 txn_0170 \n", "3 transactions:entry:txn_0040 0.495004236698 159.33 txn_0040 \n", "4 transactions:entry:txn_0169 0.494512319565 153.22 txn_0169 \n", "\n", " merchant_id item_name \n", "0 m_011 Laptop \n", "1 m_012 Headphones \n", "2 m_010 Headphones \n", "3 m_017 Headphones \n", "4 m_008 Headphones " ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redisvl.query import VectorQuery\n", "\n", "# Search for expensive electronics\n", "user_query = \"Expensive electronics purchase\"\n", "\n", "# Vectorize the user's query\n", "embedded_user_query = hf.embed(user_query, as_buffer=True)\n", "\n", "# Create vector query\n", "vec_query = VectorQuery(\n", " vector=embedded_user_query,\n", " vector_field_name=\"vector\",\n", " num_results=5,\n", " return_fields=[\"transaction_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", " return_score=True,\n", ").sort_by(\"amount\", asc=False)\n", "\n", "# Execute the query\n", "results = index.query(vec_query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Vector Search with Filters\n", "\n", "We can combine vector search with exact match filters to get more precise results." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Batches: 100%|██████████| 1/1 [00:00<00:00, 70.97it/s]\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idvector_distanceamounttransaction_iduser_idmerchant_iditem_name
0transactions:entry:txn_00420.533764362335742.36txn_0042u_017m_012Groceries
1transactions:entry:txn_00410.511252999306612.59txn_0041u_017m_003Clothing
2transactions:entry:txn_00080.563050031662564.91txn_0008u_017m_022Shoes
3transactions:entry:txn_00180.553927659988462.71txn_0018u_017m_013Dinner
4transactions:entry:txn_01950.5286039114429.52txn_0195u_017m_002Groceries
\n", "
" ], "text/plain": [ " id vector_distance amount transaction_id user_id \\\n", "0 transactions:entry:txn_0042 0.533764362335 742.36 txn_0042 u_017 \n", "1 transactions:entry:txn_0041 0.511252999306 612.59 txn_0041 u_017 \n", "2 transactions:entry:txn_0008 0.563050031662 564.91 txn_0008 u_017 \n", "3 transactions:entry:txn_0018 0.553927659988 462.71 txn_0018 u_017 \n", "4 transactions:entry:txn_0195 0.5286039114 429.52 txn_0195 u_017 \n", "\n", " merchant_id item_name \n", "0 m_012 Groceries \n", "1 m_003 Clothing \n", "2 m_022 Shoes \n", "3 m_013 Dinner \n", "4 m_002 Groceries " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Search for expensive purchases by a specific user\n", "user_query = \"Large purchases\"\n", "embedded_user_query = hf.embed(user_query)\n", "\n", "# Filter for a specific user\n", "user_filter = Tag(\"user_id\") == \"u_017\"\n", "\n", "# Create vector query with filter\n", "vec_query = VectorQuery(\n", " vector=embedded_user_query,\n", " vector_field_name=\"vector\",\n", " num_results=5,\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", " return_score=True,\n", " filter_expression=user_filter\n", ").sort_by(\"amount\", asc=False)\n", "\n", "# Execute the query\n", "results = index.query(vec_query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Batches: 100%|██████████| 1/1 [00:00<00:00, 14.38it/s]\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idvector_distancetransaction_iduser_idmerchant_iditem_nameamount
0transactions:entry:txn_00460.681029856205txn_0046u_013m_018Hotel Stay588.18
1transactions:entry:txn_01580.68172955513txn_0158u_015m_020Hotel Stay835.78
2transactions:entry:txn_00480.690501689911txn_0048u_010m_012Hotel Stay588.93
3transactions:entry:txn_01910.707059979439txn_0191u_001m_014Plane Ticket912.33
4transactions:entry:txn_00580.725707709789txn_0058u_013m_024Plane Ticket916.96
\n", "
" ], "text/plain": [ " id vector_distance transaction_id user_id \\\n", "0 transactions:entry:txn_0046 0.681029856205 txn_0046 u_013 \n", "1 transactions:entry:txn_0158 0.68172955513 txn_0158 u_015 \n", "2 transactions:entry:txn_0048 0.690501689911 txn_0048 u_010 \n", "3 transactions:entry:txn_0191 0.707059979439 txn_0191 u_001 \n", "4 transactions:entry:txn_0058 0.725707709789 txn_0058 u_013 \n", "\n", " merchant_id item_name amount \n", "0 m_018 Hotel Stay 588.18 \n", "1 m_020 Hotel Stay 835.78 \n", "2 m_012 Hotel Stay 588.93 \n", "3 m_014 Plane Ticket 912.33 \n", "4 m_024 Plane Ticket 916.96 " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Search for travel expenses with price range filter\n", "user_query = \"Travel-related expenses\"\n", "embedded_user_query = hf.embed(user_query)\n", "\n", "# Price range filter\n", "min_amount = Num(\"amount\") >= 500\n", "max_amount = Num(\"amount\") <= 1000\n", "price_range = min_amount & max_amount\n", "\n", "# Create vector query with filter\n", "vec_query = VectorQuery(\n", " vector=embedded_user_query,\n", " vector_field_name=\"vector\",\n", " num_results=5,\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", " return_score=True,\n", " filter_expression=price_range\n", ")\n", "\n", "# Execute the query\n", "results = index.query(vec_query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Full Text Search\n", "\n", "Redis also provides powerful full-text search capabilities." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idscoretransaction_iduser_idmerchant_iditem_nameamount
0transactions:entry:txn_00043.246753txn_0004u_011m_024Gaming Console157.54
1transactions:entry:txn_00133.246753txn_0013u_001m_005Gaming Console293.8
2transactions:entry:txn_00333.246753txn_0033u_008m_021Gaming Console402.54
3transactions:entry:txn_00363.246753txn_0036u_020m_002Gaming Console758.65
4transactions:entry:txn_00723.246753txn_0072u_007m_003Gaming Console68.88
5transactions:entry:txn_00883.246753txn_0088u_011m_015Gaming Console26.63
6transactions:entry:txn_00963.246753txn_0096u_014m_021Gaming Console1393.99
7transactions:entry:txn_01023.246753txn_0102u_016m_021Gaming Console697.55
8transactions:entry:txn_01093.246753txn_0109u_007m_020Gaming Console43.49
9transactions:entry:txn_01273.246753txn_0127u_001m_021Gaming Console508.48
10transactions:entry:txn_01403.246753txn_0140u_014m_008Gaming Console884.5
11transactions:entry:txn_00122.435065txn_0012u_018m_016Plane Ticket234.87
12transactions:entry:txn_00582.435065txn_0058u_013m_024Plane Ticket916.96
13transactions:entry:txn_00652.435065txn_0065u_013m_012Plane Ticket1231.32
14transactions:entry:txn_00702.435065txn_0070u_003m_005Plane Ticket1000.78
\n", "
" ], "text/plain": [ " id score transaction_id user_id merchant_id \\\n", "0 transactions:entry:txn_0004 3.246753 txn_0004 u_011 m_024 \n", "1 transactions:entry:txn_0013 3.246753 txn_0013 u_001 m_005 \n", "2 transactions:entry:txn_0033 3.246753 txn_0033 u_008 m_021 \n", "3 transactions:entry:txn_0036 3.246753 txn_0036 u_020 m_002 \n", "4 transactions:entry:txn_0072 3.246753 txn_0072 u_007 m_003 \n", "5 transactions:entry:txn_0088 3.246753 txn_0088 u_011 m_015 \n", "6 transactions:entry:txn_0096 3.246753 txn_0096 u_014 m_021 \n", "7 transactions:entry:txn_0102 3.246753 txn_0102 u_016 m_021 \n", "8 transactions:entry:txn_0109 3.246753 txn_0109 u_007 m_020 \n", "9 transactions:entry:txn_0127 3.246753 txn_0127 u_001 m_021 \n", "10 transactions:entry:txn_0140 3.246753 txn_0140 u_014 m_008 \n", "11 transactions:entry:txn_0012 2.435065 txn_0012 u_018 m_016 \n", "12 transactions:entry:txn_0058 2.435065 txn_0058 u_013 m_024 \n", "13 transactions:entry:txn_0065 2.435065 txn_0065 u_013 m_012 \n", "14 transactions:entry:txn_0070 2.435065 txn_0070 u_003 m_005 \n", "\n", " item_name amount \n", "0 Gaming Console 157.54 \n", "1 Gaming Console 293.8 \n", "2 Gaming Console 402.54 \n", "3 Gaming Console 758.65 \n", "4 Gaming Console 68.88 \n", "5 Gaming Console 26.63 \n", "6 Gaming Console 1393.99 \n", "7 Gaming Console 697.55 \n", "8 Gaming Console 43.49 \n", "9 Gaming Console 508.48 \n", "10 Gaming Console 884.5 \n", "11 Plane Ticket 234.87 \n", "12 Plane Ticket 916.96 \n", "13 Plane Ticket 1231.32 \n", "14 Plane Ticket 1000.78 " ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redisvl.query import TextQuery\n", "from redisvl.query.filter import Text\n", "\n", "# Search for specific items\n", "text_query = TextQuery(\n", " text=\"Gaming system, plane tickets, and hotel rooms\",\n", " text_field_name=\"item_name\",\n", " text_scorer=\"BM25\",\n", " num_results=15,\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\"],\n", ")\n", "\n", "results = index.query(text_query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fuzzy search is another popular technique to help with record linkage tasks." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtransaction_iduser_idmerchant_idamountitem_name
0transactions:entry:txn_0032txn_0032u_013m_002585.69Concert Ticket
1transactions:entry:txn_0058txn_0058u_013m_024916.96Plane Ticket
2transactions:entry:txn_0065txn_0065u_013m_0121231.32Plane Ticket
3transactions:entry:txn_0150txn_0150u_013m_0031075.13Plane Ticket
4transactions:entry:txn_0190txn_0190u_013m_0051413.99Plane Ticket
\n", "
" ], "text/plain": [ " id transaction_id user_id merchant_id amount \\\n", "0 transactions:entry:txn_0032 txn_0032 u_013 m_002 585.69 \n", "1 transactions:entry:txn_0058 txn_0058 u_013 m_024 916.96 \n", "2 transactions:entry:txn_0065 txn_0065 u_013 m_012 1231.32 \n", "3 transactions:entry:txn_0150 txn_0150 u_013 m_003 1075.13 \n", "4 transactions:entry:txn_0190 txn_0190 u_013 m_005 1413.99 \n", "\n", " item_name \n", "0 Concert Ticket \n", "1 Plane Ticket \n", "2 Plane Ticket \n", "3 Plane Ticket \n", "4 Plane Ticket " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redisvl.query.filter import Text\n", "\n", "user_filter = Tag(\"user_id\") == \"u_013\" # Specific user\n", "fuzzy = Text(\"item_name\") % \"%%tickt%%\"\n", "\n", "fuzzy_match = FilterQuery(\n", " filter_expression=user_filter & fuzzy,\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"amount\", \"item_name\"]\n", ")\n", "\n", "results = index.query(fuzzy_match)\n", "pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtransaction_iduser_idmerchant_iditem_nameamountcard_provider
0transactions:entry:txn_0004txn_0004u_011m_024Gaming Console157.54VISA
1transactions:entry:txn_0036txn_0036u_020m_002Gaming Console758.65VISA
2transactions:entry:txn_0102txn_0102u_016m_021Gaming Console697.55VISA
\n", "
" ], "text/plain": [ " id transaction_id user_id merchant_id \\\n", "0 transactions:entry:txn_0004 txn_0004 u_011 m_024 \n", "1 transactions:entry:txn_0036 txn_0036 u_020 m_002 \n", "2 transactions:entry:txn_0102 txn_0102 u_016 m_021 \n", "\n", " item_name amount card_provider \n", "0 Gaming Console 157.54 VISA \n", "1 Gaming Console 758.65 VISA \n", "2 Gaming Console 697.55 VISA " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Exact text match combined with other filters\n", "text_filter = Text(\"item_name\") % \"Gaming\" # Full text search for Laptop\n", "card_filter = Tag(\"card_provider\") == \"VISA\" # Only VISA card transactions\n", "combined_filter = text_filter & card_filter\n", "\n", "query = FilterQuery(\n", " return_fields=[\"transaction_id\", \"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", " filter_expression=combined_filter,\n", ")\n", "\n", "results = index.query(query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Part II: Record Linkage Examples\n", "\n", "Let's use our various search techniques to tackle a simple record linkage task. Below we have a \"fake\" transaction without a unique transaction ID. It may or may not be a duplicate of the data in our index already.\n", "\n", "Because Redis is fast we can perform fast record linkage techniques and serve transaction search clients as well.\n", "\n", "**Record linkage techniques in Redis:**\n", "- Exact match & fuzzy text search & timestamp range\n", "- Semantic search with vectors\n", "- Bloom filters (probabalistic data structures -- not shown here)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "# Create a duplicate transaction that's similar to an existing one\n", "fake_transaction = {\n", " \"user_id\": \"u_013\", # Same user as txn_0032\n", " \"merchant_id\": \"m_002\", # Same merchant as txn_0032\n", " \"item_name\": \"Concert Tickt\", # Same item slightly mispelled\n", " \"amount\": 585.69, # Same amount\n", " \"currency\": \"USD\",\n", " \"timestamp\": 1746765800, # Very close timestamp\n", " \"card_provider\": \"AMEX\", # Same card provider\n", " \"lat\": 36.173155, \n", " \"lon\": -79.595479, \n", " \"location\": \"36.173155,-79.595479\" \n", "}\n", "\n", "# In this example, the transaction is a mistaken duplicate charge by the vendor" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Batches: 100%|██████████| 1/1 [00:00<00:00, 14.15it/s]\n" ] } ], "source": [ "# User ID and Merchant ID must be the same\n", "exact_matches = (Tag(\"user_id\")==\"u_013\") & (Tag(\"merchant_id\")==\"m_002\")\n", "\n", "# Fuzzy match on Item Name\n", "terms = fake_transaction['item_name'].split()\n", "fuzzy_item_name = \" | \".join([f\"%%{term}%%\" for term in terms])\n", "fuzzy_match = Text(\"item_name\") % fuzzy_item_name\n", "\n", "# Timestamp range - create 60 second window on either side of transaction timestamp\n", "from redisvl.query.filter import Timestamp\n", "\n", "start_ts = fake_transaction['timestamp'] - 60\n", "end_ts = fake_transaction['timestamp'] + 60\n", "timestamp_range = Timestamp(\"timestamp\").between(start_ts, end_ts)\n", "\n", "# Make transaction vector\n", "transaction_vector = hf.embed(create_text_for_vectorization(fake_transaction), as_buffer=True)\n", "\n", "# Build query\n", "query = VectorQuery(\n", " vector=transaction_vector,\n", " vector_field_name=\"vector\",\n", " filter_expression=exact_matches & fuzzy_match & timestamp_range,\n", " return_fields=[\"user_id\", \"merchant_id\", \"item_name\", \"amount\", \"timestamp\", \"location\"],\n", " num_results=3\n", ")" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'(((@user_id:{u_013} @merchant_id:{m_002}) @item_name:(%%Concert%% | %%Tickt%%)) @timestamp:[1746765740.0 1746765860.0])=>[KNN 3 @vector $vector AS vector_distance] RETURN 7 user_id merchant_id item_name amount timestamp location vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 3'" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "str(query)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': 'transactions:entry:txn_0032',\n", " 'vector_distance': '0.0979611873627',\n", " 'user_id': 'u_013',\n", " 'merchant_id': 'm_002',\n", " 'item_name': 'Concert Ticket',\n", " 'amount': '585.69',\n", " 'timestamp': '1746765751',\n", " 'location': '-79.595479, 36.173155'}]" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "index.query(query)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This kind of search op for entity resolution can be very fast!" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "479 μs ± 19.7 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)\n" ] } ], "source": [ "%%timeit\n", "\n", "index.query(query)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Part III: Complex Aggregations\n", "\n", "Now let's explore Redis's powerful aggregation capabilities to analyze transaction data. This can be useful for feature store workloads, anomaly detection models, and even basic realtime analytics." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Basic Aggregations\n", "\n", "First, let's look at some simple aggregations to understand spending patterns." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "import redis.commands.search.reducers as reducers\n", "\n", "from redisvl.redis.utils import convert_bytes, make_dict\n", "from redisvl.query.aggregate import AggregationQuery" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
card_provideravg_amounttotal_amountcount
0DISCOVER661.07148148135697.8654
1VISA717.11470588236572.8551
2MASTERCARD800.72976744234431.3843
3AMEX823.31115384642812.1852
\n", "
" ], "text/plain": [ " card_provider avg_amount total_amount count\n", "0 DISCOVER 661.071481481 35697.86 54\n", "1 VISA 717.114705882 36572.85 51\n", "2 MASTERCARD 800.729767442 34431.38 43\n", "3 AMEX 823.311153846 42812.18 52" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calculate average transaction amount by card provider\n", "agg_query = AggregationQuery(\"*\") \\\n", " .group_by(\n", " \"@card_provider\",\n", " reducers.avg(\"amount\").alias(\"avg_amount\"),\n", " reducers.sum(\"amount\").alias(\"total_amount\"),\n", " reducers.count().alias(\"count\")\n", " ) \\\n", " .sort_by(\"@avg_amount\")\n", "\n", "results = index.aggregate(agg_query)\n", "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", "\n", "pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
item_nametotal_spentcount
0furniture28793
1camera2700.875
2laptop4597.796
3concert ticket3771.986
4software license6070.978
5shoes7181.089
6coffee7959.159
7clothing5670.879
8smartphone6312.799
9ride share6462.659
10book7423.419
11gaming console5236.0511
12streaming subscription10386.9311
13bicycle8212.1212
14gym membership11243.3213
15headphones6981.0513
16dinner10573.2714
17groceries10242.8414
18plane ticket11890.2815
19hotel stay13717.8515
\n", "
" ], "text/plain": [ " item_name total_spent count\n", "0 furniture 2879 3\n", "1 camera 2700.87 5\n", "2 laptop 4597.79 6\n", "3 concert ticket 3771.98 6\n", "4 software license 6070.97 8\n", "5 shoes 7181.08 9\n", "6 coffee 7959.15 9\n", "7 clothing 5670.87 9\n", "8 smartphone 6312.79 9\n", "9 ride share 6462.65 9\n", "10 book 7423.41 9\n", "11 gaming console 5236.05 11\n", "12 streaming subscription 10386.93 11\n", "13 bicycle 8212.12 12\n", "14 gym membership 11243.32 13\n", "15 headphones 6981.05 13\n", "16 dinner 10573.27 14\n", "17 groceries 10242.84 14\n", "18 plane ticket 11890.28 15\n", "19 hotel stay 13717.85 15" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Count transactions by item category\n", "agg_query = AggregationQuery(\"*\") \\\n", " .group_by(\n", " \"@item_name\",\n", " reducers.sum(\"amount\").alias(\"total_spent\"),\n", " reducers.count().alias(\"count\")\n", " ) \\\n", " .sort_by(\"@count\", max=20)\n", "\n", "\n", "results = index.aggregate(agg_query)\n", "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", "\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. User Transaction Features\n", "\n", "Let's analyze spending profiles by user. Probably most useful for feature store workloads" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idavg_transaction_amounttransaction_countstdev_transaction_amount
0u_013882.147516423.525319582
\n", "
" ], "text/plain": [ " user_id avg_transaction_amount transaction_count stdev_transaction_amount\n", "0 u_013 882.1475 16 423.525319582" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Let's look at some user spending features\n", "\n", "user_filter = Tag(\"user_id\") == \"u_013\"\n", "\n", "agg_query = AggregationQuery(str(user_filter)) \\\n", " .group_by(\n", " \"@user_id\",\n", " reducers.avg(\"amount\").alias(\"avg_transaction_amount\"),\n", " reducers.count().alias(\"transaction_count\"),\n", " reducers.stddev(\"amount\").alias(\"stdev_transaction_amount\")\n", " )\n", "\n", "\n", "results = index.aggregate(agg_query)\n", "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", "\n", "pd.DataFrame(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's analyze a user's recent transactions to build a feature for fraud detection." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtimestamptransaction_idmerchant_iditem_nameamountcard_provider
0transactions:entry:txn_01211746312151txn_0121m_012Furniture1277.46AMEX
1transactions:entry:txn_00261746214951txn_0026m_016Clothing1166.55DISCOVER
2transactions:entry:txn_01021746168151txn_0102m_021Gaming Console697.55VISA
3transactions:entry:txn_00091745919751txn_0009m_022Ride Share259.34DISCOVER
4transactions:entry:txn_01981745797351txn_0198m_005Furniture1042.48AMEX
5transactions:entry:txn_01681745462551txn_0168m_014Groceries1450.52AMEX
6transactions:entry:txn_00821745228551txn_0082m_017Streaming Subscription1320DISCOVER
7transactions:entry:txn_01161744810951txn_0116m_024Clothing350.8DISCOVER
8transactions:entry:txn_00861744670551txn_0086m_005Ride Share528.52VISA
\n", "
" ], "text/plain": [ " id timestamp transaction_id merchant_id \\\n", "0 transactions:entry:txn_0121 1746312151 txn_0121 m_012 \n", "1 transactions:entry:txn_0026 1746214951 txn_0026 m_016 \n", "2 transactions:entry:txn_0102 1746168151 txn_0102 m_021 \n", "3 transactions:entry:txn_0009 1745919751 txn_0009 m_022 \n", "4 transactions:entry:txn_0198 1745797351 txn_0198 m_005 \n", "5 transactions:entry:txn_0168 1745462551 txn_0168 m_014 \n", "6 transactions:entry:txn_0082 1745228551 txn_0082 m_017 \n", "7 transactions:entry:txn_0116 1744810951 txn_0116 m_024 \n", "8 transactions:entry:txn_0086 1744670551 txn_0086 m_005 \n", "\n", " item_name amount card_provider \n", "0 Furniture 1277.46 AMEX \n", "1 Clothing 1166.55 DISCOVER \n", "2 Gaming Console 697.55 VISA \n", "3 Ride Share 259.34 DISCOVER \n", "4 Furniture 1042.48 AMEX \n", "5 Groceries 1450.52 AMEX \n", "6 Streaming Subscription 1320 DISCOVER \n", "7 Clothing 350.8 DISCOVER \n", "8 Ride Share 528.52 VISA " ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Example: Get recent transaction history for a user\n", "user_id = \"u_016\"\n", "user_filter = Tag(\"user_id\") == user_id\n", "\n", "# Regular search query for transactions, sorted by timestamp\n", "query = FilterQuery(\n", " return_fields=[\"transaction_id\", \"timestamp\", \"merchant_id\", \"item_name\", \"amount\", \"card_provider\"],\n", " filter_expression=user_filter,\n", " num_results=10\n", ").sort_by(\"timestamp\", asc=False)\n", "\n", "results = index.query(query)\n", "pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
daydaily_transactionsdaily_total
017446705511528.52
117448109511350.8
2174522855111320
3174546255111450.52
4174579735111042.48
517459197511259.34
617461681511697.55
7174621495111166.55
8174631215111277.46
\n", "
" ], "text/plain": [ " day daily_transactions daily_total\n", "0 1744670551 1 528.52\n", "1 1744810951 1 350.8\n", "2 1745228551 1 1320\n", "3 1745462551 1 1450.52\n", "4 1745797351 1 1042.48\n", "5 1745919751 1 259.34\n", "6 1746168151 1 697.55\n", "7 1746214951 1 1166.55\n", "8 1746312151 1 1277.46" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calculate transaction frequency and spending patterns\n", "user_filter = Tag(\"user_id\") == user_id\n", "\n", "# Using Redis aggregation functions to work with dates\n", "agg_query = (\n", " AggregationQuery(str(user_filter))\n", " .load(\"@timestamp\")\n", " .apply(ts=\"format('%s', @timestamp)\")\n", " .apply(day=\"SUBSTR(@ts, 0, 10)\")\n", " .group_by(\n", " \"@day\",\n", " reducers.count().alias(\"daily_transactions\"),\n", " reducers.sum(\"amount\").alias(\"daily_total\")\n", " )\n", " .sort_by(\"@day\")\n", ")\n", "\n", "results = index.aggregate(agg_query)\n", "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", "\n", "pd.DataFrame(results)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's analyze transaction patterns by geographic location." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
merchant_idtransaction_counttransaction_totals
0m_0171352.43
1m_0111515.01
2m_0121588.93
3m_0222607.55
4m_0201835.78
5m_0212998.04
6m_00511000.78
7m_01921130.5
8m_00411420.3
9m_00931738.16
10m_00721788.59
11m_00251922.51
12m_01432060.54
13m_00332105.87
14m_01343129.86
15m_00843357.49
16m_02433394.42
17m_01654135.16
18m_00654463.77
19m_02344858.47
\n", "
" ], "text/plain": [ " merchant_id transaction_count transaction_totals\n", "0 m_017 1 352.43\n", "1 m_011 1 515.01\n", "2 m_012 1 588.93\n", "3 m_022 2 607.55\n", "4 m_020 1 835.78\n", "5 m_021 2 998.04\n", "6 m_005 1 1000.78\n", "7 m_019 2 1130.5\n", "8 m_004 1 1420.3\n", "9 m_009 3 1738.16\n", "10 m_007 2 1788.59\n", "11 m_002 5 1922.51\n", "12 m_014 3 2060.54\n", "13 m_003 3 2105.87\n", "14 m_013 4 3129.86\n", "15 m_008 4 3357.49\n", "16 m_024 3 3394.42\n", "17 m_016 5 4135.16\n", "18 m_006 5 4463.77\n", "19 m_023 4 4858.47" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Group transactions by latitude/longitude ranges to identify geographic clusters\n", "# For simplicity, we'll round lat/lon to the nearest whole number and group\n", "from redisvl.query.filter import GeoRadius, Geo\n", "\n", "\n", "geo_filter = Geo(\"location\") == GeoRadius(-71.730465, 27.584806, 1000, \"mi\")\n", "\n", "agg_query = (\n", " AggregationQuery(str(geo_filter))\n", " .group_by(\n", " \"@merchant_id\", \n", " reducers.count().alias(\"transaction_count\"),\n", " reducers.sum(\"amount\").alias(\"transaction_totals\")\n", " )\n", " .sort_by(\"@transaction_totals\", max=20)\n", ")\n", "\n", "results = index.aggregate(agg_query)\n", "results = [make_dict(row) for row in convert_bytes(results.rows)]\n", "\n", "pd.DataFrame(results)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cleaning Up Redis Resources\n", "\n", "When you're done, it's good practice to clean up your Redis resources." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Clean up by deleting the index\n", "# Uncomment the line below when you're ready to delete the index\n", "index.delete(drop=True)\n", "client.flushall()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "In this notebook, we've demonstrated how to:\n", "\n", "1. Load transaction data into Redis\n", "2. Create vector embeddings for semantic search\n", "3. Perform various search operations:\n", " - Exact match filtering\n", " - Vector similarity search\n", " - Full text search\n", " - Using search patterns for record linkage tasks\n", "4. Execute complex aggregation queries\n", " - Analyzing spending patterns by user\n", " - Looking at transaction volumes over time\n", " - Analyzing geographic transaction patterns\n", "\n", "These capabilities make Redis and RedisVL powerful tools for building real-time feature stores that can support fraud detection, personalization, and analytics applications." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.2" } }, "nbformat": 4, "nbformat_minor": 4 }