{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "\n", "# Featureform Feature Versioning with Variants\n", "\n", "In this recipe we register **two variants of the same feature** in [**Featureform**](https://docs.featureform.com/), serve each independently from **Redis**, and see how variants let you evolve feature logic without breaking models pinned to the old definition.\n", "\n", "## Why variants\n", "Feature logic changes — you tighten a filter, switch an aggregation, fix a bug. If you edit a feature in place, every model that used the old values silently starts seeing new ones, and past predictions become impossible to reproduce. Featureform's answer is the **variant**: a named version of a feature. Old and new coexist; each model **pins** the variant it was trained on; the dashboard records both.\n", "\n", "## What we'll build\n", "One feature, `avg_transactions`, in two variants:\n", "- **`v1`** — average of *all* transactions.\n", "- **`v2`** — average of only *high-value* transactions (a later refinement).\n", "\n", "Both are materialized to Redis and served side by side." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The stack — all local, no Spark\n", "\n", "- **ClickHouse** — offline store; runs a SQL transformation per variant.\n", "- **Redis** — online store; serves both variants.\n", "- **Featureform** coordinator.\n", "\n", "> ⚠️ **Needs local Docker; will not run on Colab or in CI.** The cells below start everything this recipe needs: the two provider containers on a private Docker network, plus the Featureform coordinator (gRPC `localhost:7878`, dashboard `http://localhost` — [install docs](https://docs.featureform.com/deployment/quickstart-docker))." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Start ClickHouse and Redis\n", "\n", "Both run on a private Docker network (`ff-net`) so the Featureform coordinator can reach them by container name — no host-port collisions. Only ClickHouse's HTTP port is published (on `18123`) so the data-load cell below can connect." ] }, { "cell_type": "code", "execution_count": 1, "id": "e0283434", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "57276e8a3ffb5f1dc7447ade156557d7cae1caf41a048c40592888e3fba24f74\n", "46f387815f93d3f15ca50945ba5b6c8073d560fef1886e11e616a726c8ef6e80\n", "3e44c948dae889136acf704fa158640f9ff65d188d7f1bfb68d0186110565387\n" ] } ], "source": [ "# NBVAL_SKIP\n", "# Launch this notebook's own containers on a private Docker network (ff-net) so the\n", "# Featureform coordinator reaches them by name — no host-port collisions. Names are\n", "# project-scoped (ff-clickhouse/ff-redis) so they won't clash with other containers on\n", "# your machine. Only ClickHouse's HTTP port is published (on 18123, to avoid the common\n", "# 8123) so this notebook can load data; the native port and Redis stay inside ff-net.\n", "# ClickHouse is pinned to 24.10 — the native-protocol version the coordinator speaks.\n", "!docker rm -f ff-clickhouse ff-redis 2>/dev/null\n", "!docker network create ff-net 2>/dev/null || true\n", "!docker run -d --network ff-net --name ff-clickhouse -p 18123:8123 -e CLICKHOUSE_PASSWORD=featureform clickhouse/clickhouse-server:24.10\n", "!docker run -d --network ff-net --name ff-redis redis:8" ] }, { "cell_type": "markdown", "id": "c66479e7", "metadata": {}, "source": [ "## Environment Setup\n", "\n", "### Install Python Dependencies" ] }, { "cell_type": "code", "execution_count": 2, "id": "8f771d2d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q featureform redis clickhouse-connect numpy" ] }, { "cell_type": "markdown", "id": "e7e67ddf", "metadata": {}, "source": [ "### Start the Featureform coordinator\n", "\n", "Skip this cell if you already ran `featureform deploy docker` in a terminal. It starts the coordinator (gRPC `localhost:7878`, dashboard `http://localhost`) and attaches it to `ff-net` so it can reach ClickHouse and Redis by container name." ] }, { "cell_type": "code", "execution_count": 3, "id": "5d4abfb1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deploying Featureform on Docker\n", "Starting Docker deployment on Darwin 24.6.0\n", "Checking if featureform container exists...\n", "\tContainer featureform not found. Creating new container...\n", "\t'featureform' container started\n", "\n", "Featureform is now running!\n", "To access the dashboard, visit http://localhost:80\n", "To apply definition files, run `featureform apply --host http://localhost:7878 --insecure`\n", "coordinator attached to ff-net\n" ] } ], "source": [ "# NBVAL_SKIP\n", "import sys\n", "# Start the Featureform coordinator (gRPC on localhost:7878, dashboard on http://localhost).\n", "# Invoke via the kernel's own interpreter so it works even if the `featureform` console\n", "# script isn't on PATH (common in Jupyter/VSCode after %pip install). Then attach the\n", "# coordinator to ff-net, retrying until confirmed, so it resolves ClickHouse/Redis by name.\n", "!{sys.executable} -m featureform deploy docker\n", "!for i in $(seq 10); do docker network connect ff-net featureform 2>/dev/null; docker inspect featureform --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' 2>/dev/null | grep -q ff-net && echo \"coordinator attached to ff-net\" && break; sleep 1; done" ] }, { "cell_type": "markdown", "id": "92222a05", "metadata": {}, "source": [ "### Configure connections" ] }, { "cell_type": "code", "execution_count": 4, "id": "0d27cc27", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "# This recipe is *about* variants, so disable Featureform's \"equivalent variant\" auto-reuse.\n", "# With it on (the default, FF_GET_EQUIVALENT_VARIANTS), applying two feature variants that\n", "# differ only in their source transformation makes the coordinator judge v2 \"equivalent\" to\n", "# v1 and collapse it (\"equivalent feature variant already exists, going to use its variant:\n", "# v1\") — so v2 never registers and serving (\"avg_transactions\", \"v2\") fails with a metadata\n", "# NotFound. Turning it off registers each variant on its own.\n", "os.environ[\"FF_GET_EQUIVALENT_VARIANTS\"] = \"false\"\n", "\n", "# Featureform coordinator (gRPC)\n", "FEATUREFORM_HOST = os.getenv(\"FEATUREFORM_HOST\", \"localhost:7878\")\n", "\n", "# The coordinator reaches the providers by container name over the shared ff-net network.\n", "# ClickHouse offline store (recent images require a password for network access)\n", "CLICKHOUSE_HOST = os.getenv(\"CLICKHOUSE_HOST\", \"ff-clickhouse\")\n", "CLICKHOUSE_NATIVE_PORT = int(os.getenv(\"CLICKHOUSE_NATIVE_PORT\", \"9000\"))\n", "CLICKHOUSE_HTTP_PORT = int(os.getenv(\"CLICKHOUSE_HTTP_PORT\", \"18123\")) # published to host for the data-load cell\n", "CLICKHOUSE_USER = os.getenv(\"CLICKHOUSE_USER\", \"default\")\n", "CLICKHOUSE_PASSWORD = os.getenv(\"CLICKHOUSE_PASSWORD\", \"featureform\")\n", "CLICKHOUSE_DATABASE = os.getenv(\"CLICKHOUSE_DATABASE\", \"default\")\n", "\n", "# Redis online store\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"ff-redis\")\n", "REDIS_PORT = int(os.getenv(\"REDIS_PORT\", \"6379\"))\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")" ] }, { "cell_type": "markdown", "id": "3bff7488", "metadata": {}, "source": [ "### Create a sample transactions table in ClickHouse" ] }, { "cell_type": "code", "execution_count": 5, "id": "ca16983f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rows: 500\n" ] } ], "source": [ "# NBVAL_SKIP\n", "import time\n", "import clickhouse_connect\n", "import numpy as np\n", "\n", "# ClickHouse may still be starting up — retry until it accepts connections.\n", "for _ in range(30):\n", " try:\n", " ch = clickhouse_connect.get_client(\n", " host=\"localhost\", port=CLICKHOUSE_HTTP_PORT,\n", " username=CLICKHOUSE_USER, password=CLICKHOUSE_PASSWORD,\n", " )\n", " ch.command(\"SELECT 1\")\n", " break\n", " except Exception:\n", " time.sleep(1)\n", "else:\n", " raise RuntimeError(f\"ClickHouse not reachable on localhost:{CLICKHOUSE_HTTP_PORT} — is the container running?\")\n", "\n", "ch.command(\"DROP TABLE IF EXISTS transactions\")\n", "ch.command(\n", " \"\"\"\n", " CREATE TABLE transactions (\n", " TransactionID String,\n", " CustomerID String,\n", " TransactionAmount Float64\n", " ) ENGINE = MergeTree ORDER BY CustomerID\n", " \"\"\"\n", ")\n", "rng = np.random.default_rng(42)\n", "rows = [[f\"T{i:05d}\", f\"C{int(rng.integers(1000, 1050)):04d}\", round(float(rng.gamma(2.0, 50.0)), 2)]\n", " for i in range(500)]\n", "ch.insert(\"transactions\", rows, column_names=[\"TransactionID\", \"CustomerID\", \"TransactionAmount\"])\n", "print(\"rows:\", ch.command(\"SELECT count() FROM transactions\"))" ] }, { "cell_type": "markdown", "id": "859c08d6", "metadata": {}, "source": [ "## Register providers and source" ] }, { "cell_type": "code", "execution_count": 6, "id": "a171ec57", "metadata": {}, "outputs": [], "source": [ "import featureform as ff\n", "\n", "clickhouse = ff.register_clickhouse(\n", " name=\"clickhouse-quickstart\",\n", " description=\"ClickHouse offline store with transaction history\",\n", " host=CLICKHOUSE_HOST,\n", " port=CLICKHOUSE_NATIVE_PORT,\n", " user=CLICKHOUSE_USER,\n", " password=CLICKHOUSE_PASSWORD,\n", " database=CLICKHOUSE_DATABASE,\n", ")\n", "\n", "redis = ff.register_redis(\n", " name=\"redis-quickstart\",\n", " description=\"Redis online (inference) store\",\n", " host=REDIS_HOST,\n", " port=REDIS_PORT,\n", " password=REDIS_PASSWORD,\n", " db=0,\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "id": "2bc0fec1", "metadata": {}, "outputs": [], "source": [ "transactions = clickhouse.register_table(\n", " name=\"transactions\", variant=\"quickstart\", table=\"transactions\",\n", ")" ] }, { "cell_type": "markdown", "id": "72a8199d", "metadata": {}, "source": [ "## Two transformations — the old logic and the refinement\n", "\n", "Each variant is backed by its own transformation. `v1` averages every transaction; `v2` averages only transactions above 100. Both output the same shape (`user_id`, `avg_transaction_amt`) so they can be variants of one feature." ] }, { "cell_type": "code", "execution_count": 8, "id": "0ae085a4", "metadata": {}, "outputs": [], "source": [ "@clickhouse.sql_transformation(variant=\"v1\")\n", "def average_user_transaction_all():\n", " \"\"\"v1: average of all transactions.\"\"\"\n", " return (\n", " \"SELECT CustomerID AS user_id, avg(TransactionAmount) AS avg_transaction_amt \"\n", " \"FROM {{transactions.quickstart}} GROUP BY CustomerID\"\n", " )\n", "\n", "@clickhouse.sql_transformation(variant=\"v2\")\n", "def average_user_transaction_highvalue():\n", " \"\"\"v2: average of only high-value (>100) transactions.\"\"\"\n", " return (\n", " \"SELECT CustomerID AS user_id, avg(TransactionAmount) AS avg_transaction_amt \"\n", " \"FROM {{transactions.quickstart}} WHERE TransactionAmount > 100 GROUP BY CustomerID\"\n", " )" ] }, { "cell_type": "markdown", "id": "0ffbac70", "metadata": {}, "source": [ "## Declare the feature in two variants\n", "\n", "`ff.Variants` groups multiple versions under a single feature name. Each entry points at its own transformation and carries its own `variant` tag. Both materialize to Redis, and neither can clobber the other." ] }, { "cell_type": "code", "execution_count": 9, "id": "402ca19b", "metadata": {}, "outputs": [], "source": [ "@ff.entity\n", "class User:\n", " avg_transactions = ff.Variants({\n", " \"v1\": ff.Feature(\n", " average_user_transaction_all[[\"user_id\", \"avg_transaction_amt\"]],\n", " variant=\"v1\",\n", " type=ff.Float32,\n", " inference_store=redis,\n", " ),\n", " \"v2\": ff.Feature(\n", " average_user_transaction_highvalue[[\"user_id\", \"avg_transaction_amt\"]],\n", " variant=\"v2\",\n", " type=ff.Float32,\n", " inference_store=redis,\n", " ),\n", " })" ] }, { "cell_type": "markdown", "id": "82a10503", "metadata": {}, "source": [ "## Apply\n", "\n", "Both transformations run and both variants materialize into Redis." ] }, { "cell_type": "code", "execution_count": 10, "id": "d53dcb13", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Applying Run: sleepy_almeida\n", "Creating User default_owner \n", "Creating Provider clickhouse-quickstart \n", "Creating Provider redis-quickstart \n", "Creating Source Variant transactions quickstart\n", "Creating Source Variant average_user_transaction_all v1\n", "Creating Source Variant average_user_transaction_highvalue v2\n", "Creating Entity user \n", "Creating Feature Variant avg_transactions v1\n", "Creating Feature Variant avg_transactions v2\n", "\n" ] }, { "data": { "text/html": [ "
UserWarning: install \"ipywidgets\" for Jupyter support\n",
       "
\n" ], "text/plain": [ "UserWarning: install \"ipywidgets\" for Jupyter support\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "client = ff.Client(host=FEATUREFORM_HOST, insecure=True)\n",
    "client.apply(asynchronous=False, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cca0a25",
   "metadata": {},
   "source": [
    "## Serve each variant — pinned by name\n",
    "\n",
    "A caller asks for `avg_transactions` **and a specific variant**. Same feature name, two definitions, two values. A model trained on `v1` keeps requesting `v1` and is completely unaffected by the later `v2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6e83d02b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No resources to apply\n",
      "user C1047\n",
      "  avg_transactions v1 (all txns):        [92.00199890136719]\n",
      "  avg_transactions v2 (high-value only): [152.7857208251953]\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "user_id = client.dataframe(average_user_transaction_all)[\"user_id\"].iloc[0]\n",
    "\n",
    "v1 = client.features([(\"avg_transactions\", \"v1\")], {\"user\": user_id})\n",
    "v2 = client.features([(\"avg_transactions\", \"v2\")], {\"user\": user_id})\n",
    "print(f\"user {user_id}\")\n",
    "print(f\"  avg_transactions v1 (all txns):        {v1}\")\n",
    "print(f\"  avg_transactions v2 (high-value only): {v2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9279918c",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "Stop and remove the containers when you're done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "39571a9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tearing down Featureform on Docker\n",
      "Stopping containers...\n",
      "\tStopping featureform container\n",
      "Container quickstart-clickhouse not found. Skipping...\n",
      "ff-clickhouse\n",
      "ff-redis\n",
      "featureform\n",
      "ff-net\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "import sys\n",
    "# Tear down everything this notebook started. Remove containers (including the coordinator)\n",
    "# before the network; the coordinator's endpoint releases asynchronously, so retry the delete.\n",
    "!{sys.executable} -m featureform stop docker\n",
    "!docker rm -f ff-clickhouse ff-redis featureform 2>/dev/null\n",
    "!for i in $(seq 5); do docker network rm ff-net 2>/dev/null && break || sleep 1; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "\n",
    "- [Featureform variants & versioning](https://docs.featureform.com/)\n",
    "- [Featureform transformations & lineage recipe](./03_featureform_transformations_lineage.ipynb)\n",
    "- [Featureform + Redis fraud detection recipe](./02_featureform_fraud_detection.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "redis-ai-res",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}