{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "# Fraud Detection in Financial Services with Redis & RedisVL\n", "\n", "## Let's Begin!\n", "\"Open\n", "\n", "Card fraud is fundamentally a *similarity* problem: a fraudulent transaction\n", "tends to look like other fraudulent transactions, and unlike the normal\n", "spending behavior of the account it hits. That makes it a natural fit for\n", "**vector search**.\n", "\n", "In this notebook we use Redis as a real-time fraud engine:\n", "\n", "1. Turn each transaction into a **feature vector** (amount, time, location,\n", " velocity, channel, …).\n", "2. Index those vectors in Redis with **RedisVL**.\n", "3. Score a new transaction three complementary ways:\n", " - **KNN fraud scoring** — how fraudulent do the most similar past\n", " transactions look?\n", " - **Anomaly detection** — how far is this transaction from the account's\n", " *normal* behavior?\n", " - **Velocity / rules** — combine vector similarity with metadata filters\n", " (card, merchant category, amount, time window) the way a real fraud\n", " system does.\n", "\n", "We use engineered numeric features (not text embeddings) because fraud signals\n", "are tabular — this also keeps the notebook fast, deterministic, and free of any\n", "API keys." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Packages" ] }, { "cell_type": "code", "execution_count": 1, "id": "5443d725", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m26.1.2\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q \"redisvl>=0.11.0\" pandas numpy scikit-learn" ] }, { "cell_type": "markdown", "id": "c8681b44", "metadata": {}, "source": [ "## Install Redis Stack\n", "\n", "This notebook stores and indexes transaction feature vectors in Redis, so we\n", "need a Redis instance with the search & query capability available.\n", "\n", "#### For Colab\n", "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." ] }, { "cell_type": "code", "execution_count": null, "id": "a12b5129", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "%%sh\n", "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", "sudo apt-get update > /dev/null 2>&1\n", "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", "redis-stack-server --daemonize yes" ] }, { "cell_type": "markdown", "id": "a6fd6377", "metadata": {}, "source": [ "#### For Alternative Environments\n", "There are many ways to get the necessary redis-stack instance running\n", "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your own version of Redis Enterprise running, that works too!\n", "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" ] }, { "cell_type": "markdown", "id": "a365e7f0", "metadata": {}, "source": [ "### Define the Redis Connection URL\n", "\n", "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." ] }, { "cell_type": "code", "execution_count": 2, "id": "ba20a6cd", "metadata": {}, "outputs": [], "source": [ "import os\n", "import warnings\n", "\n", "warnings.filterwarnings('ignore')\n", "\n", "# Replace values below with your own if using Redis Cloud instance\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", "\n", "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" ] }, { "cell_type": "markdown", "id": "80964cc4", "metadata": {}, "source": [ "### Create redis client" ] }, { "cell_type": "code", "execution_count": 3, "id": "8eaa67d1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redis import Redis\n", "\n", "client = Redis.from_url(REDIS_URL)\n", "client.ping()" ] }, { "cell_type": "markdown", "id": "03c461ff", "metadata": {}, "source": [ "## Generate a synthetic transaction dataset\n", "\n", "Real card data is sensitive and rarely shareable, so we simulate a labeled\n", "dataset that captures the signals fraud teams actually use. Each transaction\n", "has both **raw metadata** (card id, merchant category, amount, country,\n", "timestamp) and **behavioral features** used for the vector:\n", "\n", "| feature | fraud signal |\n", "|---|---|\n", "| `amount` | fraud skews toward large or oddly-round amounts |\n", "| `hour` | fraud clusters in the middle of the night |\n", "| `distance_from_home` | card-present fraud happens far from the cardholder |\n", "| `distance_from_last_txn` | impossible travel between consecutive swipes |\n", "| `ratio_to_median_amount` | spend far above the account's typical purchase |\n", "| `num_txn_last_hour` | velocity — many transactions in a short window |\n", "| `is_online` | online / card-not-present is higher risk |\n", "\n", "Roughly 3% of the transactions are fraudulent, similar to a stressed real-world\n", "mix." ] }, { "cell_type": "code", "execution_count": 4, "id": "cdeaedc3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "5012 transactions, 162 fraudulent (3.2%)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
amounthourdistance_from_homedistance_from_last_txnratio_to_median_amountnum_txn_last_houris_onlinemerchant_categorycountryis_fraudcard_idtxn_idtimestamp
021.461131.02.30.81310travelUS0card_719txn_01701728087
177.3115160.92.61.17510electronicsGB0card_669txn_11701272998
283.01941.148.80.96011online_retailBR0card_651txn_21701690135
353.291247.27.31.03110electronicsCA0card_258txn_31701737612
423.9988.311.20.95320travelFR0card_571txn_41701438900
\n", "
" ], "text/plain": [ " amount hour distance_from_home distance_from_last_txn \\\n", "0 21.46 11 31.0 2.3 \n", "1 77.31 15 160.9 2.6 \n", "2 83.01 9 41.1 48.8 \n", "3 53.29 12 47.2 7.3 \n", "4 23.99 8 8.3 11.2 \n", "\n", " ratio_to_median_amount num_txn_last_hour is_online merchant_category \\\n", "0 0.813 1 0 travel \n", "1 1.175 1 0 electronics \n", "2 0.960 1 1 online_retail \n", "3 1.031 1 0 electronics \n", "4 0.953 2 0 travel \n", "\n", " country is_fraud card_id txn_id timestamp \n", "0 US 0 card_719 txn_0 1701728087 \n", "1 GB 0 card_669 txn_1 1701272998 \n", "2 BR 0 card_651 txn_2 1701690135 \n", "3 CA 0 card_258 txn_3 1701737612 \n", "4 FR 0 card_571 txn_4 1701438900 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "rng = np.random.default_rng(42)\n", "\n", "N = 5000\n", "FRAUD_RATE = 0.03\n", "n_fraud = int(N * FRAUD_RATE)\n", "n_legit = N - n_fraud\n", "\n", "CATEGORIES = [\"grocery\", \"restaurant\", \"travel\", \"electronics\", \"fuel\", \"online_retail\", \"atm_withdrawal\"]\n", "COUNTRIES = [\"US\", \"CA\", \"GB\", \"DE\", \"FR\", \"NG\", \"RU\", \"BR\"]\n", "\n", "\n", "def make_transactions(n, fraud):\n", " if fraud:\n", " amount = rng.lognormal(mean=6.0, sigma=1.0, size=n) # larger spend\n", " hour = rng.choice(range(24), size=n, p=_night_heavy()) # late night\n", " distance_from_home = rng.exponential(scale=400, size=n) # far away\n", " distance_from_last = rng.exponential(scale=300, size=n) # impossible travel\n", " ratio_to_median = rng.lognormal(mean=1.2, sigma=0.6, size=n) # well above normal\n", " num_txn_last_hour = rng.poisson(lam=4.0, size=n) + 1 # bursty velocity\n", " is_online = rng.binomial(1, 0.7, size=n)\n", " country = rng.choice(COUNTRIES, size=n, p=[0.25,0.05,0.1,0.05,0.05,0.2,0.2,0.1])\n", " else:\n", " amount = rng.lognormal(mean=3.5, sigma=0.8, size=n)\n", " hour = rng.choice(range(24), size=n, p=_day_heavy())\n", " distance_from_home = rng.exponential(scale=25, size=n)\n", " distance_from_last = rng.exponential(scale=20, size=n)\n", " ratio_to_median = rng.lognormal(mean=0.0, sigma=0.3, size=n)\n", " num_txn_last_hour = rng.poisson(lam=0.5, size=n) + 1\n", " is_online = rng.binomial(1, 0.3, size=n)\n", " country = rng.choice(COUNTRIES, size=n, p=[0.6,0.1,0.1,0.05,0.05,0.03,0.02,0.05])\n", "\n", " return pd.DataFrame({\n", " \"amount\": np.round(amount, 2),\n", " \"hour\": hour,\n", " \"distance_from_home\": np.round(distance_from_home, 1),\n", " \"distance_from_last_txn\": np.round(distance_from_last, 1),\n", " \"ratio_to_median_amount\": np.round(ratio_to_median, 3),\n", " \"num_txn_last_hour\": num_txn_last_hour,\n", " \"is_online\": is_online,\n", " \"merchant_category\": rng.choice(CATEGORIES, size=n),\n", " \"country\": country,\n", " \"is_fraud\": int(fraud),\n", " })\n", "\n", "\n", "def _night_heavy():\n", " \"\"\" bias toward nighttime transactions \"\"\"\n", " w = np.array([3,3,3,3,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,3,3,3], dtype=float)\n", " return w / w.sum()\n", "\n", "\n", "def _day_heavy():\n", " \"\"\" bias toward daytime transactions \"\"\"\n", " w = np.array([1,1,1,1,1,1,2,3,4,4,4,5,5,4,4,4,4,4,5,4,3,2,1,1], dtype=float)\n", " return w / w.sum()\n", "\n", "\n", "df = pd.concat([make_transactions(n_legit, False), make_transactions(n_fraud, True)], ignore_index=True)\n", "df = df.sample(frac=1.0, random_state=1).reset_index(drop=True) # shuffle the legit and fraud rows together\n", "\n", "# create identifiers + a synthetic event time (seconds since an arbitrary epoch)\n", "df[\"card_id\"] = [\"card_\" + str(i) for i in rng.integers(0, 800, size=len(df))]\n", "df[\"txn_id\"] = [\"txn_\" + str(i) for i in range(len(df))]\n", "df[\"timestamp\"] = rng.integers(1_700_000_000, 1_700_000_000 + 60*60*24*30, size=len(df))\n", "\n", "# Inject one account-takeover burst: 12 transactions on a single card within\n", "# a 1-hour window. This is the classic \"velocity\" fraud pattern we detect later.\n", "BURST_CARD = \"card_burst\"\n", "BURST_START = 1_700_500_000\n", "burst = make_transactions(12, fraud=True)\n", "burst[\"card_id\"] = BURST_CARD\n", "burst[\"txn_id\"] = [\"txn_burst_\" + str(i) for i in range(len(burst))]\n", "burst[\"timestamp\"] = BURST_START + rng.integers(0, 60 * 60, size=len(burst)) # within one hour\n", "df = pd.concat([df, burst], ignore_index=True)\n", "\n", "print(f\"{len(df)} transactions, {df.is_fraud.sum()} fraudulent ({df.is_fraud.mean():.1%})\")\n", "df.head()" ] }, { "cell_type": "markdown", "id": "e6a1b653", "metadata": {}, "source": [ "### Build the feature vector\n", "\n", "We standardize the seven behavioral features (zero mean, unit variance) and pack\n", "them into one vector per transaction. Standardizing matters: without it,\n", "`amount` (hundreds) would dominate `is_online` (0/1) purely because of scale.\n", "\n", "We split the data into a **reference set** (what we index and search against) and\n", "a small **test set** of unseen transactions to score later. In production the\n", "reference set is your history of labeled transactions." ] }, { "cell_type": "code", "execution_count": 5, "id": "a0103aaf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "vector dim: 7 | reference: 4812 | test: 200\n" ] } ], "source": [ "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split\n", "\n", "FEATURES = [\n", " \"amount\", \"hour\", \"distance_from_home\", \"distance_from_last_txn\",\n", " \"ratio_to_median_amount\", \"num_txn_last_hour\", \"is_online\",\n", "]\n", "VECTOR_DIM = len(FEATURES)\n", "\n", "# keep the injected burst out of the test split so the velocity demo can find it\n", "splittable = df[df[\"card_id\"] != \"card_burst\"]\n", "ref_df, test_df = train_test_split(splittable, test_size=200, random_state=7, stratify=splittable[\"is_fraud\"])\n", "ref_df = pd.concat([ref_df, df[df[\"card_id\"] == \"card_burst\"]]).reset_index(drop=True)\n", "test_df = test_df.reset_index(drop=True)\n", "\n", "# fit the scaler ONLY on reference data, then apply to both\n", "scaler = StandardScaler().fit(ref_df[FEATURES])\n", "\n", "def to_vectors(frame):\n", " return scaler.transform(frame[FEATURES]).astype(np.float32)\n", "\n", "ref_vectors = to_vectors(ref_df)\n", "test_vectors = to_vectors(test_df)\n", "\n", "ref_df[\"vector\"] = list(ref_vectors)\n", "print(\"vector dim:\", VECTOR_DIM, \"| reference:\", len(ref_df), \"| test:\", len(test_df))" ] }, { "cell_type": "markdown", "id": "fe2df847", "metadata": {}, "source": [ "## Define the Redis index schema\n", "\n", "We index the raw transaction fields (so we can filter and investigate) plus the\n", "feature `vector`. We use an **HNSW** vector index with **L2 (Euclidean)**\n", "distance — for standardized feature vectors, L2 directly measures how different\n", "two transactions' behaviors are.\n", "\n", "We've chosen the **HNSW** approximate nearest neighbors algorithm instead of the\n", "exact **Flat** nearest neighbors algorithm because in real applications your system\n", "will likely have many more transactions than this notebook and speed is key for\n", "catching fraudulent charges as they happen. The **HNSW** algorithm isn't guaranteed\n", "to find the closest transactions, but it is able to scale to millions of transactions\n", "while maintaining a fast response rate." ] }, { "cell_type": "code", "execution_count": 6, "id": "0e093c72", "metadata": {}, "outputs": [], "source": [ "from redisvl.schema import IndexSchema\n", "from redisvl.index import SearchIndex\n", "\n", "index_name = \"transactions\"\n", "\n", "schema = IndexSchema.from_dict({\n", " \"index\": {\n", " \"name\": index_name,\n", " \"prefix\": index_name,\n", " \"storage_type\": \"hash\",\n", " },\n", " \"fields\": [\n", " {\"name\": \"txn_id\", \"type\": \"tag\"},\n", " {\"name\": \"card_id\", \"type\": \"tag\"},\n", " {\"name\": \"merchant_category\", \"type\": \"tag\", \"attrs\": {\"sortable\": True}},\n", " {\"name\": \"country\", \"type\": \"tag\", \"attrs\": {\"sortable\": True}},\n", " {\"name\": \"amount\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", " {\"name\": \"timestamp\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", " {\"name\": \"is_fraud\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", " {\n", " \"name\": \"vector\",\n", " \"type\": \"vector\",\n", " \"attrs\": {\n", " \"dims\": VECTOR_DIM,\n", " \"distance_metric\": \"l2\",\n", " \"algorithm\": \"hnsw\",\n", " \"datatype\": \"float32\",\n", " },\n", " },\n", " ],\n", "})\n", "\n", "index = SearchIndex(schema, client)\n", "index.create(overwrite=True, drop=True)" ] }, { "cell_type": "code", "execution_count": 7, "id": "76dc1917", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "Index Information:\n", "╭──────────────────┬──────────────────┬──────────────────┬──────────────────┬──────────────────╮\n", "│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │\n", "├──────────────────┼──────────────────┼──────────────────┼──────────────────┼──────────────────┤\n", "| transactions | HASH | ['transactions'] | [] | 0 |\n", "╰──────────────────┴──────────────────┴──────────────────┴──────────────────┴──────────────────╯\n", "Index Fields:\n", "╭───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────┬───────────────────╮\n", "│ Name │ Attribute │ Type │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │\n", "├───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┼───────────────────┤\n", "│ txn_id │ txn_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │ │ │ │ │\n", "│ card_id │ card_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │ │ │ │ │\n", "│ merchant_category │ merchant_category │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │ │ │ │ │\n", "│ country │ country │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │ │ │ │ │\n", "│ amount │ amount │ NUMERIC │ SORTABLE │ UNF │ │ │ │ │ │ │ │ │ │ │\n", "│ timestamp │ timestamp │ NUMERIC │ SORTABLE │ UNF │ │ │ │ │ │ │ │ │ │ │\n", "│ is_fraud │ is_fraud │ NUMERIC │ SORTABLE │ UNF │ │ │ │ │ │ │ │ │ │ │\n", "│ vector │ vector │ VECTOR │ algorithm │ HNSW │ data_type │ FLOAT32 │ dim │ 7 │ distance_metric │ L2 │ M │ 16 │ ef_construction │ 200 │\n", "╰───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────┴───────────────────╯\n" ] } ], "source": [ "!rvl index info -i transactions -u {REDIS_URL}" ] }, { "cell_type": "markdown", "id": "efbf595f", "metadata": {}, "source": [ "## Populate the index\n", "\n", "We load the reference transactions, embedding the float32 vector as bytes (the\n", "format Redis stores). The metadata travels alongside each vector so we can\n", "filter on it during search." ] }, { "cell_type": "code", "execution_count": 8, "id": "0c572b5d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "loaded 4812 transactions into the 'transactions' index\n" ] } ], "source": [ "def to_records(frame):\n", " records = []\n", " for _, row in frame.iterrows():\n", " records.append({\n", " \"txn_id\": row[\"txn_id\"],\n", " \"card_id\": row[\"card_id\"],\n", " \"merchant_category\": row[\"merchant_category\"],\n", " \"country\": row[\"country\"],\n", " \"amount\": float(row[\"amount\"]),\n", " \"timestamp\": int(row[\"timestamp\"]),\n", " \"is_fraud\": int(row[\"is_fraud\"]),\n", " \"vector\": row[\"vector\"].tobytes(),\n", " })\n", " return records\n", "\n", "keys = index.load(to_records(ref_df), id_field=\"txn_id\")\n", "print(f\"loaded {len(keys)} transactions into the '{index_name}' index\")" ] }, { "cell_type": "markdown", "id": "2b9323d1", "metadata": {}, "source": [ "## Technique 1 — KNN fraud scoring\n", "\n", "The core idea: to score a new transaction, find its **k nearest neighbors** in\n", "the indexed history and look at how many of them were fraud. A high\n", "neighborhood fraud rate is a strong, explainable signal.\n", "\n", "Let's grab one known-fraud and one known-legit transaction from the unseen test\n", "set and score each." ] }, { "cell_type": "code", "execution_count": 9, "id": "ad78b140", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "KNOWN FRAUD: neighborhood fraud rate = 100% (amount=$1464.11, category=restaurant)\n", "KNOWN LEGIT: neighborhood fraud rate = 0% (amount=$65.05, category=online_retail)\n" ] } ], "source": [ "from redisvl.query import VectorQuery\n", "\n", "K = 15\n", "\n", "def fraud_score(vector, k=K, filter_expression=None):\n", " q = VectorQuery(\n", " vector=vector,\n", " vector_field_name=\"vector\",\n", " num_results=k,\n", " return_fields=[\"txn_id\", \"card_id\", \"merchant_category\", \"amount\", \"is_fraud\"],\n", " return_score=True,\n", " filter_expression=filter_expression,\n", " )\n", " neighbors = index.query(q)\n", " fraud_fraction = np.mean([float(n[\"is_fraud\"]) for n in neighbors])\n", " return fraud_fraction, neighbors\n", "\n", "# one fraud + one legit example from the held-out test set\n", "fraud_i = test_df.index[test_df[\"is_fraud\"] == 1][0]\n", "legit_i = test_df.index[test_df[\"is_fraud\"] == 0][0]\n", "\n", "for label, i in [(\"KNOWN FRAUD\", fraud_i), (\"KNOWN LEGIT\", legit_i)]:\n", " score, neighbors = fraud_score(test_vectors[i])\n", " print(f\"{label}: neighborhood fraud rate = {score:.0%} (amount=${test_df.loc[i,'amount']:.2f}, \"\n", " f\"category={test_df.loc[i,'merchant_category']})\")" ] }, { "cell_type": "markdown", "id": "27257b3a", "metadata": {}, "source": [ "The fraudulent transaction sits in a neighborhood dense with other fraud; the\n", "legitimate one is surrounded by normal spend. Here are the actual neighbors\n", "returned for the fraudulent transaction — note the vector distance and the\n", "`is_fraud` flag on each:" ] }, { "cell_type": "code", "execution_count": 10, "id": "84c68494", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
txn_idcard_idmerchant_categoryamountis_fraudvector_distance
0txn_4325card_647fuel1287.19125.455909729
1txn_754card_731grocery1416.95128.6217079163
2txn_2175card_296travel1635.25128.7414302826
3txn_4213card_261fuel710.18129.5273799896
4txn_2865card_52restaurant1012.76130.6872444153
5txn_3738card_385electronics1198.33130.9978752136
6txn_3267card_101fuel838.94131.6496009827
7txn_436card_396grocery1076.41131.8088111877
8txn_3486card_144travel898.66134.2544250488
9txn_108card_514atm_withdrawal1782.87135.8748664856
10txn_1731card_155electronics1416.87136.1867599487
11txn_246card_236electronics1327.27137.1904792786
12txn_4875card_397online_retail970.9138.6752967834
13txn_burst_1card_bursttravel1664.99141.9085388184
14txn_burst_9card_burstfuel576.56144.3301773071
\n", "
" ], "text/plain": [ " txn_id card_id merchant_category amount is_fraud \\\n", "0 txn_4325 card_647 fuel 1287.19 1 \n", "1 txn_754 card_731 grocery 1416.95 1 \n", "2 txn_2175 card_296 travel 1635.25 1 \n", "3 txn_4213 card_261 fuel 710.18 1 \n", "4 txn_2865 card_52 restaurant 1012.76 1 \n", "5 txn_3738 card_385 electronics 1198.33 1 \n", "6 txn_3267 card_101 fuel 838.94 1 \n", "7 txn_436 card_396 grocery 1076.41 1 \n", "8 txn_3486 card_144 travel 898.66 1 \n", "9 txn_108 card_514 atm_withdrawal 1782.87 1 \n", "10 txn_1731 card_155 electronics 1416.87 1 \n", "11 txn_246 card_236 electronics 1327.27 1 \n", "12 txn_4875 card_397 online_retail 970.9 1 \n", "13 txn_burst_1 card_burst travel 1664.99 1 \n", "14 txn_burst_9 card_burst fuel 576.56 1 \n", "\n", " vector_distance \n", "0 25.455909729 \n", "1 28.6217079163 \n", "2 28.7414302826 \n", "3 29.5273799896 \n", "4 30.6872444153 \n", "5 30.9978752136 \n", "6 31.6496009827 \n", "7 31.8088111877 \n", "8 34.2544250488 \n", "9 35.8748664856 \n", "10 36.1867599487 \n", "11 37.1904792786 \n", "12 38.6752967834 \n", "13 41.9085388184 \n", "14 44.3301773071 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "score, neighbors = fraud_score(test_vectors[fraud_i])\n", "pd.DataFrame(neighbors)[[\"txn_id\", \"card_id\", \"merchant_category\", \"amount\", \"is_fraud\", \"vector_distance\"]]" ] }, { "cell_type": "markdown", "id": "615154c2", "metadata": {}, "source": [ "### Evaluate the scorer on the full test set\n", "\n", "Scoring all 200 unseen transactions lets us pick an alert threshold and see the\n", "precision/recall tradeoff — exactly the knob a fraud team tunes against their\n", "review capacity." ] }, { "cell_type": "code", "execution_count": 11, "id": "38833bd1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Alert threshold: neighborhood fraud rate >= 50%\n", "\n", "[[194 0]\n", " [ 1 5]]\n", "\n", " precision recall f1-score support\n", "\n", " legit 0.995 1.000 0.997 194\n", " fraud 1.000 0.833 0.909 6\n", "\n", " accuracy 0.995 200\n", " macro avg 0.997 0.917 0.953 200\n", "weighted avg 0.995 0.995 0.995 200\n", "\n" ] } ], "source": [ "from sklearn.metrics import classification_report, confusion_matrix\n", "\n", "scores = np.array([fraud_score(v)[0] for v in test_vectors])\n", "y_true = test_df[\"is_fraud\"].values\n", "\n", "THRESHOLD = 0.5 # alert if >= 50% of neighbors are fraud\n", "y_pred = (scores >= THRESHOLD).astype(int)\n", "\n", "print(f\"Alert threshold: neighborhood fraud rate >= {THRESHOLD:.0%}\\n\")\n", "print(confusion_matrix(y_true, y_pred))\n", "print()\n", "print(classification_report(y_true, y_pred, target_names=[\"legit\", \"fraud\"], digits=3))" ] }, { "cell_type": "markdown", "id": "0c52a454", "metadata": {}, "source": [ "## Technique 2 — anomaly detection against normal behavior\n", "\n", "KNN scoring needs labeled fraud examples. But genuinely novel fraud may not\n", "resemble any *past* fraud. A complementary, label-light approach is to measure how\n", "far a transaction is from the account's **normal** behavior. If the nearest\n", "*legitimate* transaction is still far away, the transaction is anomalous.\n", "\n", "We do this with a [RangeQuery](https://docs.redisvl.com/) filtered to\n", "`is_fraud == 0`. We're asking \"are there any normal transactions within distance R?\" If\n", "Ther's no match inside the radius ⇒ flagged as anomalous." ] }, { "cell_type": "code", "execution_count": 12, "id": "230e3b00", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "KNOWN FRAUD: nearest normal txn distance = none in radius -> anomaly=True\n", "KNOWN LEGIT: nearest normal txn distance = 0.06 -> anomaly=False\n" ] } ], "source": [ "from redisvl.query import RangeQuery\n", "from redisvl.query.filter import Num\n", "\n", "legit_only = Num(\"is_fraud\") == 0\n", "RADIUS = 3.0 # max L2 distance to be considered \"normal\"\n", "\n", "def anomaly_check(vector, radius=RADIUS):\n", " q = RangeQuery(\n", " vector=vector,\n", " vector_field_name=\"vector\",\n", " return_fields=[\"txn_id\", \"is_fraud\"],\n", " distance_threshold=radius,\n", " num_results=1,\n", " filter_expression=legit_only,\n", " )\n", " results = index.query(q)\n", " nearest = results[0][\"vector_distance\"] if results else None\n", " is_anomaly = nearest is None or float(nearest) > radius\n", " return is_anomaly, nearest\n", "\n", "for label, i in [(\"KNOWN FRAUD\", fraud_i), (\"KNOWN LEGIT\", legit_i)]:\n", " anomaly, nearest = anomaly_check(test_vectors[i])\n", " near_str = f\"{float(nearest):.2f}\" if nearest is not None else \"none in radius\"\n", " print(f\"{label}: nearest normal txn distance = {near_str} -> anomaly={anomaly}\")" ] }, { "cell_type": "markdown", "id": "61015a04", "metadata": {}, "source": [ "## Technique 3 — combine similarity with rules and velocity\n", "\n", "Production fraud systems blend the ML signal with hard business rules and\n", "filters. Because the metadata lives in the same index as the vectors, RedisVL\n", "lets us express these as **filtered vector queries** in a single round trip.\n", "\n", "**Example A — scoped KNN:** when investigating a suspicious online purchase,\n", "restrict the neighbor search to the *same merchant category* so the fraud score\n", "reflects peers in that category." ] }, { "cell_type": "code", "execution_count": 13, "id": "5c354d0c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "category = restaurant\n", " fraud score among ALL neighbors: 100%\n", " fraud score among 'restaurant' neighbors: 100%\n" ] } ], "source": [ "from redisvl.query.filter import Tag\n", "\n", "category = test_df.loc[fraud_i, \"merchant_category\"]\n", "\n", "scoped_score, _ = fraud_score(test_vectors[fraud_i], filter_expression=Tag(\"merchant_category\") == category)\n", "global_score, _ = fraud_score(test_vectors[fraud_i])\n", "print(f\"category = {category}\")\n", "print(f\" fraud score among ALL neighbors: {global_score:.0%}\")\n", "print(f\" fraud score among '{category}' neighbors: {scoped_score:.0%}\")" ] }, { "cell_type": "markdown", "id": "26500f3b", "metadata": {}, "source": [ "**Example B — velocity check.** A classic fraud rule: too many transactions on\n", "one card in a short window. Here we count a card's transactions in a 1-hour\n", "window using a pure metadata filter query (no vector needed) via `FilterQuery`." ] }, { "cell_type": "code", "execution_count": 14, "id": "22276200", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "card card_burst: 12 transactions in a 1-hour window -> likely account takeover\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
txn_idamountmerchant_categoryis_fraud
0txn_burst_0479.55travel1
1txn_burst_11664.99travel1
2txn_burst_2424.58electronics1
3txn_burst_3133.18electronics1
4txn_burst_4616.55fuel1
5txn_burst_5606.77online_retail1
6txn_burst_6333.53atm_withdrawal1
7txn_burst_71120.52electronics1
8txn_burst_840.89online_retail1
9txn_burst_9576.56fuel1
10txn_burst_10883.9grocery1
11txn_burst_1194.74online_retail1
\n", "
" ], "text/plain": [ " txn_id amount merchant_category is_fraud\n", "0 txn_burst_0 479.55 travel 1\n", "1 txn_burst_1 1664.99 travel 1\n", "2 txn_burst_2 424.58 electronics 1\n", "3 txn_burst_3 133.18 electronics 1\n", "4 txn_burst_4 616.55 fuel 1\n", "5 txn_burst_5 606.77 online_retail 1\n", "6 txn_burst_6 333.53 atm_withdrawal 1\n", "7 txn_burst_7 1120.52 electronics 1\n", "8 txn_burst_8 40.89 online_retail 1\n", "9 txn_burst_9 576.56 fuel 1\n", "10 txn_burst_10 883.9 grocery 1\n", "11 txn_burst_11 94.74 online_retail 1" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from redisvl.query import FilterQuery\n", "\n", "# our injected account-takeover card: a burst of transactions in one hour\n", "busy_card = \"card_burst\"\n", "window_start = int(ref_df.loc[ref_df[\"card_id\"] == busy_card, \"timestamp\"].min())\n", "window_end = window_start + 60 * 60 # one hour\n", "\n", "velocity_filter = (Tag(\"card_id\") == busy_card) & \\\n", " (Num(\"timestamp\") >= window_start) & (Num(\"timestamp\") <= window_end)\n", "\n", "vq = FilterQuery(\n", " return_fields=[\"txn_id\", \"amount\", \"merchant_category\", \"timestamp\", \"is_fraud\"],\n", " filter_expression=velocity_filter,\n", " num_results=100,\n", ")\n", "hits = index.query(vq)\n", "print(f\"card {busy_card}: {len(hits)} transactions in a 1-hour window -> likely account takeover\")\n", "pd.DataFrame(hits)[[\"txn_id\", \"amount\", \"merchant_category\", \"is_fraud\"]] if hits else \"no transactions in window\"" ] }, { "cell_type": "markdown", "id": "8e8700a7", "metadata": {}, "source": [ "**Example C — high-value online transactions from high-risk countries.** A\n", "pure filter query that surfaces transactions matching a risk policy, ready for\n", "manual review." ] }, { "cell_type": "code", "execution_count": 15, "id": "aa10f2de", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10 high-value transactions from high-risk countries flagged for review\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtxn_idcard_idamountcountrymerchant_categoryis_fraud
0transactions:txn_1731txn_1731card_1551416.87RUelectronics1
1transactions:txn_4875txn_4875card_397970.9NGonline_retail1
2transactions:txn_4128txn_4128card_391801.74NGelectronics1
3transactions:txn_848txn_848card_218754.6RUgrocery1
4transactions:txn_2959txn_2959card_6002133.8RUfuel1
5transactions:txn_4960txn_4960card_3191047.54RUonline_retail1
6transactions:txn_535txn_535card_478846.66NGrestaurant1
7transactions:txn_1823txn_1823card_5762.67RUonline_retail1
8transactions:txn_3938txn_3938card_5472172.73NGatm_withdrawal1
9transactions:txn_2865txn_2865card_521012.76NGrestaurant1
\n", "
" ], "text/plain": [ " id txn_id card_id amount country \\\n", "0 transactions:txn_1731 txn_1731 card_155 1416.87 RU \n", "1 transactions:txn_4875 txn_4875 card_397 970.9 NG \n", "2 transactions:txn_4128 txn_4128 card_39 1801.74 NG \n", "3 transactions:txn_848 txn_848 card_218 754.6 RU \n", "4 transactions:txn_2959 txn_2959 card_600 2133.8 RU \n", "5 transactions:txn_4960 txn_4960 card_319 1047.54 RU \n", "6 transactions:txn_535 txn_535 card_478 846.66 NG \n", "7 transactions:txn_1823 txn_1823 card_5 762.67 RU \n", "8 transactions:txn_3938 txn_3938 card_547 2172.73 NG \n", "9 transactions:txn_2865 txn_2865 card_52 1012.76 NG \n", "\n", " merchant_category is_fraud \n", "0 electronics 1 \n", "1 online_retail 1 \n", "2 electronics 1 \n", "3 grocery 1 \n", "4 fuel 1 \n", "5 online_retail 1 \n", "6 restaurant 1 \n", "7 online_retail 1 \n", "8 atm_withdrawal 1 \n", "9 restaurant 1 " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "review_q = FilterQuery(\n", " return_fields=[\"txn_id\", \"card_id\", \"amount\", \"country\", \"merchant_category\", \"is_fraud\"],\n", " filter_expression=(Num(\"amount\") >= 500) & ((Tag(\"country\") == \"NG\") | (Tag(\"country\") == \"RU\")),\n", " num_results=10,\n", ")\n", "flagged = index.query(review_q)\n", "print(f\"{len(flagged)} high-value transactions from high-risk countries flagged for review\")\n", "pd.DataFrame(flagged) if flagged else \"none flagged\"" ] }, { "cell_type": "markdown", "id": "2c4846df", "metadata": {}, "source": [ "## Putting it together — a real-time scoring function\n", "\n", "A single function that a transaction-processing service could call at authorization\n", "time. It returns a decision plus the *reasons*, which is essential for fraud\n", "analysts and for regulatory explainability. All of it is backed by Redis in\n", "milliseconds." ] }, { "cell_type": "code", "execution_count": 16, "id": "7b8f35a8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fraudulent example: {'decision': 'BLOCK', 'fraud_score': 1.0, 'reasons': ['100% of similar transactions were fraud', 'unlike any normal transaction (anomalous behavior)']}\n", "Legitimate example: {'decision': 'ALLOW', 'fraud_score': 0.0, 'reasons': []}\n", "Account takeover: {'decision': 'BLOCK', 'fraud_score': 1.0, 'reasons': ['100% of similar transactions were fraud', 'unlike any normal transaction (anomalous behavior)', 'high velocity: 12 transactions on this card']}\n" ] } ], "source": [ "def score_transaction(vector, card_id, timestamp=None):\n", " reasons = []\n", "\n", " # 1. supervised k-NN signal\n", " knn_score, _ = fraud_score(vector)\n", " if knn_score >= 0.5:\n", " reasons.append(f\"{knn_score:.0%} of similar transactions were fraud\")\n", "\n", " # 2. unsupervised anomaly signal\n", " anomaly, nearest = anomaly_check(vector)\n", " if anomaly:\n", " reasons.append(\"unlike any normal transaction (anomalous behavior)\")\n", "\n", " # 3. velocity rule\n", " if timestamp is not None:\n", " ts = int(timestamp)\n", " velocity_filter = (Tag(\"card_id\") == card_id) & \\\n", " (Num(\"timestamp\") >= ts - 60 * 60) & \\\n", " (Num(\"timestamp\") <= ts)\n", " recent = index.query(FilterQuery(\n", " return_fields=[\"txn_id\"],\n", " filter_expression=velocity_filter,\n", " num_results=100,\n", " ))\n", " if len(recent) >= 10:\n", " reasons.append(f\"high velocity: {len(recent)} transactions on this card\")\n", "\n", " decision = \"BLOCK\" if (knn_score >= 0.5 or anomaly) else \"ALLOW\"\n", " return {\"decision\": decision, \"fraud_score\": round(float(knn_score), 3), \"reasons\": reasons}\n", "\n", "# score the held-out fraud and legit examples, plus the account-takeover card\n", "burst_vec = ref_df.loc[ref_df[\"card_id\"] == \"card_burst\", \"vector\"].iloc[0]\n", "burst_ts = int(ref_df.loc[ref_df[\"card_id\"] == \"card_burst\", \"timestamp\"].max())\n", "print(\"Fraudulent example: \", score_transaction(test_vectors[fraud_i], test_df.loc[fraud_i, \"card_id\"], test_df.loc[fraud_i, \"timestamp\"]))\n", "print(\"Legitimate example: \", score_transaction(test_vectors[legit_i], test_df.loc[legit_i, \"card_id\"], test_df.loc[legit_i, \"timestamp\"]))\n", "print(\"Account takeover: \", score_transaction(burst_vec, \"card_burst\", burst_ts))" ] }, { "cell_type": "markdown", "id": "39e03833", "metadata": {}, "source": [ "## Why Redis for fraud detection?\n", "\n", "- **Latency** — authorization must complete in tens of milliseconds. Redis serves\n", " vector KNN, range, and filter queries from memory at that speed.\n", "- **One system, two jobs** — the same Redis index holds the feature vectors *and*\n", " the transaction metadata, so similarity search and business rules run together\n", " in a single query instead of joining across systems.\n", "- **Real-time updates** — new labeled transactions are `index.load`-ed and\n", " immediately searchable; the model's \"memory\" of fraud stays current without a\n", " retraining batch job.\n", "- **Velocity & history** — Redis's data structures (counters, sorted sets, TTLs)\n", " complement vector search for the time-window rules fraud systems rely on.\n", "\n", "### Next steps\n", "- Swap the synthetic generator for your own labeled transaction history.\n", "- Replace standardized features with a learned embedding (e.g. an autoencoder or\n", " a two-tower model) for richer similarity — see the\n", " [recommendation-systems recipes](../recommendation-systems/).\n", "- Add a [semantic cache](../semantic-cache/) or\n", " [feature store](../feature-store/) layer for the surrounding pipeline." ] }, { "cell_type": "markdown", "id": "2990efb9", "metadata": {}, "source": [ "### Clean up" ] }, { "cell_type": "code", "execution_count": 17, "id": "037aceee", "metadata": {}, "outputs": [], "source": [ "# clean up!\n", "index.delete()" ] } ], "metadata": { "kernelspec": { "display_name": "redis-ai-res", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }