{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "\n", "# Redis Online Feature Store with Featureform\n", "\n", "In this recipe, we will learn how to define, materialize, and serve machine learning features with [**Featureform**](https://docs.featureform.com/) using **Redis** as the low-latency [online (inference) store](https://redis.io/docs/latest/develop/ai/featureform/).\n", "\n", "## The problem: features are where ML projects break\n", "Most of the effort in a production ML system is not the model — it's the **features**. A feature like \"average transaction amount over the last 30 days\" has to be computed one way in a batch job to train the model, and a *different* way in application code to score a live request. When those two implementations drift apart you get **training-serving skew**: the model was trained on numbers it never actually sees in production, and accuracy quietly degrades. On top of that, the same feature gets re-implemented by every team that needs it, nobody can find what already exists, and there is no record of how any given value was produced.\n", "\n", "## What a feature store gives you\n", "A **feature store** is the interface between your raw data and your models. It lets you:\n", "- **Define a feature once** and serve the *identical* computation to both training (offline, high-throughput) and inference (online, low-latency) — killing training-serving skew.\n", "- **Reuse and discover** features across models and teams instead of rebuilding them.\n", "- **Version and audit** feature definitions alongside model code, so every prediction is traceable to the exact logic that produced it.\n", "\n", "## Why Featureform specifically\n", "[Featureform](https://docs.featureform.com/) is a **virtual** feature store — it does not copy your data into a new monolithic system. Instead it:\n", "- **Leaves your data where it is.** Register your existing warehouse, database, or stream as a *provider*; there is no migration.\n", "- **Treats features as code.** Transformations are plain Python/SQL functions, versioned in git and reviewed like any other code — not click-ops in a UI.\n", "- **Separates definition from infrastructure.** The same feature definition can be materialized to different online stores; you pick the right engine for each job.\n", "- **Tracks lineage.** Every feature, label, and training set is a named, versioned resource with a recorded path back to its source.\n", "\n", "## Why Redis as the online store\n", "Training reads happen in bulk and can be slow; **inference reads happen one entity at a time, on the critical path of a live request**, and must be fast. Redis is a natural fit as Featureform's inference store: in-memory, sub-millisecond point lookups, and horizontally scalable — so a fraud model can fetch a user's features and score a transaction well within a request budget. Featureform computes features in the offline store (here, Postgres) and **materializes** the results into Redis for serving.\n", "\n", "## What we'll build\n", "The canonical Featureform fraud-detection quickstart, wired to serve from Redis:\n", "- **Postgres** as the **offline store** — holds the raw transaction history and runs the feature transformations.\n", "- **Redis** as the **inference store** — materialized feature values are pushed here and served at sub-millisecond latency.\n", "\n", "We'll register both, define an `avg_transactions` feature and a `fraudulent` label keyed by user, build a training set, materialize to Redis, and serve a live feature lookup." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running this notebook\n", "\n", "> ⚠️ **This notebook runs locally with Docker — it will not run on Google Colab or in notebook CI**, because Featureform needs a running coordinator server plus provider containers (there is no Docker daemon on Colab).\n", "\n", "**Prerequisites**\n", "1. [Install Docker](https://docs.docker.com/get-docker/) and make sure the daemon is running (`docker ps`).\n", "2. Install the Featureform CLI and start the quickstart stack from a terminal:\n", "\n", " ```bash\n", " pip install featureform\n", " featureform deploy docker --quickstart\n", " ```\n", "\n", " This pulls and starts three containers: the **Featureform** coordinator (gRPC on `localhost:7878`, dashboard on `http://localhost`), a **Redis** container (online store, published on `localhost:6379`), and a **Postgres** container pre-loaded with an example `Transactions` table (offline store, `localhost:5432`).\n", "3. Run the cells below in order. Tear everything down at the end with `featureform stop docker`.\n", "\n", "The deploy step is also runnable from the notebook (next cell) if you prefer." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Environment Setup\n", "\n", "### Install Python Dependencies" ] }, { "cell_type": "code", "execution_count": 1, "id": "699107a9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q featureform redis pandas" ] }, { "cell_type": "markdown", "id": "eac16eea", "metadata": {}, "source": [ "### Start Featureform, Redis, and Postgres\n", "\n", "Skip this cell if you already ran `featureform deploy docker --quickstart` in a terminal (see above)." ] }, { "cell_type": "code", "execution_count": 2, "id": "9c253ef3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deploying Featureform on Docker\n", "Starting Docker deployment on Darwin 24.6.0\n", "Checking if featureform container exists...\n", "\tContainer featureform has status \"running\"\n", "\tContainer featureform is already running. Skipping...\n", "Checking if quickstart-postgres container exists...\n", "\tContainer quickstart-postgres has status \"running\"\n", "\tContainer quickstart-postgres is already running. Skipping...\n", "Checking if quickstart-redis container exists...\n", "\tContainer quickstart-redis has status \"running\"\n", "\tContainer quickstart-redis is already running. Skipping...\n", "\n", "Pulling Quickstart files\n", "\tPulling definitions.py\n", "\t\tdefinitions.py already exists. Skipping...\n", "\tPulling serving.py\n", "\t\tserving.py already exists. Skipping...\n", "\tPulling training.py\n", "\t\ttraining.py already exists. Skipping...\n", "\n", "Featureform is now running!\n", "To access the dashboard, visit http://localhost:80\n", "Run jupyter notebook in the quickstart directory to get started.\n" ] } ], "source": [ "# NBVAL_SKIP\n", "!featureform deploy docker --quickstart" ] }, { "cell_type": "markdown", "id": "55daf3c8", "metadata": {}, "source": [ "### Point the client at the Featureform server\n", "\n", "The Python client talks to the coordinator over gRPC on `localhost:7878`. `insecure=True` is required because the quickstart container serves an unencrypted endpoint." ] }, { "cell_type": "code", "execution_count": 3, "id": "f71a39a6", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "FEATUREFORM_HOST = os.getenv(\"FEATUREFORM_HOST\", \"localhost:7878\")" ] }, { "cell_type": "markdown", "id": "8d09e61e", "metadata": {}, "source": [ "## Register providers\n", "\n", "We register Redis as the inference store and Postgres as the offline store.\n", "\n", "The `host` is the address at which the **Featureform coordinator container** reaches each provider. The quickstart publishes Redis and Postgres on the host machine, so the coordinator reaches them via `host.docker.internal`.\n", "\n", "> **On Linux**, `host.docker.internal` may not resolve — use the Docker bridge IP `172.17.0.1` instead (see [featureform#1156](https://github.com/featureform/featureform/issues/1156)).\n", ">\n", "> **Using your own Redis?** Swap `REDIS_HOST` / `REDIS_PORT` / `REDIS_PASSWORD` below for your [Redis Cloud](https://redis.io/cloud/) or Redis Enterprise endpoint. Featureform will materialize and serve features from that instance instead." ] }, { "cell_type": "code", "execution_count": 4, "id": "90fa2f99", "metadata": {}, "outputs": [], "source": [ "import featureform as ff\n", "\n", "# Address the Featureform *coordinator container* uses to reach the providers.\n", "# Mac/Windows: \"host.docker.internal\". Linux: try \"172.17.0.1\".\n", "PROVIDER_HOST = os.getenv(\"PROVIDER_HOST\", \"host.docker.internal\")\n", "\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", PROVIDER_HOST)\n", "REDIS_PORT = int(os.getenv(\"REDIS_PORT\", \"6379\"))\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")\n", "\n", "redis = ff.register_redis(\n", " name=\"redis-quickstart\",\n", " description=\"Redis online (inference) store\",\n", " host=REDIS_HOST,\n", " port=REDIS_PORT,\n", " password=REDIS_PASSWORD,\n", " db=0,\n", ")\n", "\n", "postgres = ff.register_postgres(\n", " name=\"postgres-quickstart\",\n", " description=\"Postgres offline store with example transaction data\",\n", " host=PROVIDER_HOST,\n", " port=\"5432\",\n", " user=\"postgres\",\n", " password=\"password\",\n", " database=\"postgres\",\n", ")" ] }, { "cell_type": "markdown", "id": "c6ae7853", "metadata": {}, "source": [ "## Register the source data\n", "\n", "The Postgres quickstart image ships a `Transactions` table. We register it as a source so features can be derived from it." ] }, { "cell_type": "code", "execution_count": 5, "id": "831fd6a8", "metadata": {}, "outputs": [], "source": [ "transactions = postgres.register_table(\n", " name=\"transactions\",\n", " variant=\"quickstart\",\n", " table=\"transactions\",\n", ")" ] }, { "cell_type": "markdown", "id": "05075d29", "metadata": {}, "source": [ "## Define a feature transformation\n", "\n", "Features in Featureform are just transformations over registered sources. Here we compute each user's **average transaction amount** with a SQL transformation that runs in the offline store (Postgres). The `{{transactions.quickstart}}` placeholder references the source we just registered.\n", "\n", "*Why this matters:* this decorated function **is** the single definition of the feature. It runs in Postgres to build training data and its output is materialized to Redis for serving. It's one piece of code, so the two can never drift. It's versioned in git and reviewable like any other function, and the `variant=\"quickstart\"` tag lets you evolve the logic later without breaking models pinned to the old version." ] }, { "cell_type": "code", "execution_count": 6, "id": "bdd4f1b3", "metadata": {}, "outputs": [], "source": [ "@postgres.sql_transformation(variant=\"quickstart\")\n", "def average_user_transaction():\n", " \"\"\"Average transaction amount per user, computed in the offline store.\"\"\"\n", " return (\n", " \"SELECT CustomerID as user_id, avg(TransactionAmount) as avg_transaction_amt \"\n", " \"FROM {{transactions.quickstart}} GROUP BY user_id\"\n", " )" ] }, { "cell_type": "markdown", "id": "b20169e3", "metadata": {}, "source": [ "## Define the entity, feature, and label\n", "\n", "These three resource types are the core of how Featureform models data, and they play distinct roles:\n", "\n", "- **Entity** — *what a row is about.* Here the entity is a **user**, declared with `@ff.entity`. It's the join key: every feature and label below is keyed by user, so Featureform knows how to line them up. At serving time you look features up by an entity key (`{\"user\": \"C1214240\"}`).\n", "- **Feature** — *a model input.* `avg_transactions` is a per-user value derived from the `average_user_transaction` transformation. Because it has `inference_store=redis`, its values are **materialized into Redis** and served online, one entity at a time, on the request path.\n", "- **Label** — *the prediction target.* `fraudulent` (from the `isfraud` column) is the ground truth the model learns to predict. Labels are used **only offline** to build training sets; they are never materialized to the online store, because at inference time the answer is exactly what you're trying to produce.\n", "\n", "**How they interact:** the entity is the glue — features and the label are both keyed by user, and a *training set* (next step) joins them on that key into `(features, label)` rows. **How they differ:** features are model *inputs* served online for live scoring; the label is the *output* used only for offline training; the entity is neither — it's the identity that ties them together." ] }, { "cell_type": "code", "execution_count": 7, "id": "c812d380", "metadata": {}, "outputs": [], "source": [ "@ff.entity\n", "class User:\n", " avg_transactions = ff.Feature(\n", " average_user_transaction[[\"user_id\", \"avg_transaction_amt\"]],\n", " variant=\"quickstart\",\n", " type=ff.Float32,\n", " inference_store=redis,\n", " )\n", " fraudulent = ff.Label(\n", " transactions[[\"customerid\", \"isfraud\"]],\n", " variant=\"quickstart\",\n", " type=ff.Bool,\n", " )" ] }, { "cell_type": "markdown", "id": "a94b2b0e", "metadata": {}, "source": [ "## Register a training set\n", "\n", "A training set joins features to a label on the entity key, giving one reproducible source of truth for model training — built from the *same* feature definitions that serve online, so there's no training-serving skew." ] }, { "cell_type": "code", "execution_count": 8, "id": "cf94df0b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "TrainingSetVariant(name='fraud_training', owner='default_owner', label=('fraudulent', 'quickstart'), features=[('avg_transactions', 'quickstart')], description='', variant='quickstart', feature_lags=[], tags=[], properties={}, created=None, schedule='', schedule_obj=None, provider='', status='NO_STATUS', error=None, server_status=None, resource_snowflake_config=None, type=)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ff.register_training_set(\n", " name=\"fraud_training\",\n", " variant=\"quickstart\",\n", " label=(\"fraudulent\", \"quickstart\"),\n", " features=[(\"avg_transactions\", \"quickstart\")],\n", ")" ] }, { "cell_type": "markdown", "id": "fd3fb5dd", "metadata": {}, "source": [ "## Apply the definitions\n", "\n", "`client.apply()` registers everything with the coordinator and kicks off materialization: the SQL transformation runs in Postgres and the resulting feature values are pushed into Redis. `asynchronous=False` blocks until materialization finishes." ] }, { "cell_type": "code", "execution_count": 9, "id": "5594dba1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Applying Run: 2026-07-22t13-39-20\n", "Creating User default_owner \n", "Creating Provider redis-quickstart \n", "Creating Provider postgres-quickstart \n", "Creating Source Variant transactions quickstart\n", "Creating Source Variant average_user_transaction quickstart\n", "Creating Entity user \n", "Creating Feature Variant avg_transactions quickstart\n", "Creating Label Variant fraudulent quickstart\n", "Creating Trainingset Variant fraud_training quickstart\n", "\n" ] }, { "data": { "text/html": [ "
UserWarning: install \"ipywidgets\" for Jupyter support\n",
       "
\n" ], "text/plain": [ "UserWarning: install \"ipywidgets\" for Jupyter support\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "client = ff.Client(host=FEATUREFORM_HOST, insecure=True)\n",
    "client.apply(asynchronous=False, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve features from the Redis online store\n",
    "\n",
    "Now for the payoff: request a feature for a single entity key. This read is served straight from Redis, so it returns in milliseconds — the pattern you'd put behind a real-time fraud model. The entity dict is keyed by the lowercased entity class name (`user`).\n",
    "\n",
    "*Why this matters:* this is the exact same feature, by name and variant, that the training set below is built from — but fetched from Redis in the time budget of a live request. Define once, serve everywhere: the model scores production traffic on precisely the values it was trained on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "avg_transactions for user C1214240: [319.0]\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "avg_txn = client.features(\n",
    "    [(\"avg_transactions\", \"quickstart\")],\n",
    "    {\"user\": \"C1214240\"},\n",
    ")\n",
    "print(\"avg_transactions for user C1214240:\", avg_txn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark the online read\n",
    "\n",
    "Serving from Redis is the whole point of an online store. A single feature lookup should land in the low-millisecond range end-to-end (gRPC round-trip + Redis read)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.27 ms ± 179 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "%timeit client.features([(\"avg_transactions\", \"quickstart\")], {\"user\": \"C1214240\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a training set from the same definitions\n",
    "\n",
    "The offline side reuses the exact feature/label definitions. The dataset is iterable and streams rows of `(features, label)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[650.]] -> [False]\n",
      "[[234.]] -> [ True]\n",
      "[[1.]] -> [ True]\n",
      "[[370.]] -> [False]\n",
      "[[47.]] -> [ True]\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "dataset = client.training_set(\"fraud_training\", \"quickstart\")\n",
    "\n",
    "for i, row in enumerate(dataset):\n",
    "    print(row.features(), \"->\", row.label())\n",
    "    if i >= 4:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the feature values in Redis\n",
    "\n",
    "Featureform materializes features into the online store, so we can connect to the quickstart Redis (published on `localhost:6379`) and confirm the keys landed. Values are stored under Featureform-encoded keys, so they won't be human-readable, but you'll see them populated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 keys in Redis; first 10:\n",
      "b'{\"Prefix\":\"Featureform_table__\",\"Feature\":\"avg_transactions\",\"Variant\":\"quickstart\"}'\n",
      "b'Featureform_table____tables'\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "from redis import Redis\n",
    "\n",
    "# The quickstart Redis is published on the host at localhost:6379.\n",
    "redis_client = Redis(host=\"localhost\", port=6379, password=\"\")\n",
    "keys = redis_client.keys()\n",
    "print(f\"{len(keys)} keys in Redis; first 10:\")\n",
    "for k in keys[:10]:\n",
    "    print(k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "Stop and remove the quickstart containers when you're done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
      "I0000 00:00:1784752778.502483 1749902 fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tearing down Featureform on Docker\n",
      "Stopping containers...\n",
      "\tStopping featureform container\n",
      "\tStopping quickstart-postgres container\n",
      "\tStopping quickstart-redis container\n",
      "Container quickstart-clickhouse not found. Skipping...\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "!featureform stop docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "\n",
    "- [Redis Feature Form docs](https://redis.io/docs/latest/develop/ai/featureform/)\n",
    "- [Featureform documentation](https://docs.featureform.com/)\n",
    "- [Feast + Redis credit scoring recipe](./00_feast_credit_score.ipynb) — the other feature-store pattern in this repo"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "redis-ai-res",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}