{ "cells": [ { "cell_type": "markdown", "id": "intro", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "\n", "# Featureform Embeddings & Vector Search on Redis\n", "\n", "In this recipe we register **embedding features** in [**Featureform**](https://docs.featureform.com/), materialize them to **Redis as a vector index**, and run **semantic (nearest-neighbor) search** against them with `client.nearest()`.\n", "\n", "## Why embeddings belong in a feature store\n", "An embedding is just a feature whose value is a vector. Treating it as a first-class Featureform feature buys you the same guarantees as any other feature: it's **defined once**, **versioned**, and **served from a low-latency online store** — here, Redis, which doubles as the vector index for similarity search. The model that produced the embedding, the source rows, and the serving index all stay linked.\n", "\n", "## What we'll build\n", "A tiny **semantic product search**:\n", "1. Embed product descriptions with a sentence-transformer model.\n", "2. Load the vectors into ClickHouse and register them as an `ff.Embedding` feature, materialized to **Redis**.\n", "3. Embed a free-text query and ask Redis for the nearest products with `client.nearest()`.\n", "\n", "> ℹ️ **Why the embeddings are precomputed in Python:** Featureform runs SQL transformations in the offline store, and SQL can't call a transformer model. So we compute vectors in the notebook and load them into ClickHouse. (Computing embeddings *inside* a transformation would require a Spark/Kubernetes provider.)" ] }, { "cell_type": "markdown", "id": "stack", "metadata": {}, "source": [ "## The stack — all local, no Spark\n", "\n", "- **ClickHouse** — offline store; holds the source rows and their precomputed vectors.\n", "- **Redis** — online store **and vector index**; serves nearest-neighbor queries.\n", "- **Featureform** coordinator — registers resources and materializes the vectors into Redis.\n", "\n", "> ⚠️ **Needs local Docker; will not run on Colab or in CI.** The next cell starts all three containers itself (coordinator on gRPC `localhost:7878`, dashboard `http://localhost`) and the cleanup cell removes them — you don't need anything running beforehand." ] }, { "cell_type": "markdown", "id": "start-hdr", "metadata": {}, "source": [ "### Start the local stack (Featureform coordinator, ClickHouse, Redis)" ] }, { "cell_type": "code", "execution_count": 1, "id": "start-stack", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bd36809e85fc2cb808e82395e7eb3de65b7871b8c0a69954c282caae399d71ca\n", "0fb6c973fd01870ada20c05c01e0fc6717f6290ffc217d3b09787d2e1b084063\n", "d5aea8ee352ea46cd9e369c307322bfa1356929962d0c9c85f68907be72f6755\n", "9b0c5a27d92ad988d452e6be488b9f20aaedcc9b77bf73b86ad8dd928cded741\n", "featureform + clickhouse + redis ready\n" ] } ], "source": [ "# NBVAL_SKIP\n", "# Launch the full local stack this notebook needs on a private Docker network, so\n", "# the coordinator reaches ClickHouse/Redis by container name and we don't have to\n", "# publish (and risk host-port collisions on) their internal ports. Only the ports\n", "# the host itself uses are published: ClickHouse HTTP 8123 (data load) and the\n", "# coordinator's gRPC 7878 + dashboard 80. Remove any existing containers/network\n", "# first so a re-run always gets a fresh registry and store. (Other systems may\n", "# start/stop these containers; we own their lifecycle here.)\n", "!docker rm -f featureform clickhouse redis 2>/dev/null\n", "!docker network rm ff-net 2>/dev/null\n", "!docker network create ff-net\n", "!docker run -d --name clickhouse --network ff-net -p 8123:8123 -e CLICKHOUSE_PASSWORD=featureform clickhouse/clickhouse-server:latest\n", "!docker run -d --name redis --network ff-net redis:8\n", "!docker run -d --name featureform --network ff-net -p 80:80 -p 7878:7878 featureformcom/featureform:latest\n", "\n", "# Wait until everything is ready. ClickHouse needs a few seconds to apply the\n", "# password (it rejects auth during that window) and the coordinator validates the\n", "# ClickHouse provider on apply(), so both must be reachable before we register.\n", "import subprocess, time\n", "\n", "def _ready():\n", " ch = subprocess.run([\"docker\", \"exec\", \"clickhouse\", \"clickhouse-client\",\n", " \"--password\", \"featureform\", \"-q\", \"SELECT 1\"],\n", " capture_output=True, text=True)\n", " rd = subprocess.run([\"docker\", \"exec\", \"redis\", \"redis-cli\", \"ping\"],\n", " capture_output=True, text=True)\n", " ff = subprocess.run([\"docker\", \"inspect\", \"--format\",\n", " \"{{.State.Health.Status}}\", \"featureform\"],\n", " capture_output=True, text=True)\n", " return (ch.stdout.strip() == \"1\" and rd.stdout.strip() == \"PONG\"\n", " and ff.stdout.strip() == \"healthy\")\n", "\n", "for _ in range(150):\n", " if _ready():\n", " print(\"featureform + clickhouse + redis ready\")\n", " break\n", " time.sleep(2)\n", "else:\n", " raise RuntimeError(\"stack did not become ready in time; check `docker ps -a`\")" ] }, { "cell_type": "markdown", "id": "setup-hdr", "metadata": {}, "source": [ "## Environment Setup\n", "\n", "### Install Python Dependencies" ] }, { "cell_type": "code", "execution_count": 2, "id": "pip", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q featureform redis clickhouse-connect sentence-transformers pandas" ] }, { "cell_type": "markdown", "id": "conf-hdr", "metadata": {}, "source": [ "### Configure connections\n", "\n", "The coordinator container reaches ClickHouse and Redis by **container name** over the shared `ff-net` network. This notebook (running on the host) talks to ClickHouse's published HTTP port and the coordinator's published gRPC port via `localhost`." ] }, { "cell_type": "code", "execution_count": 3, "id": "conf", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "# Featureform coordinator (gRPC), reached from this notebook on the host.\n", "FEATUREFORM_HOST = os.getenv(\"FEATUREFORM_HOST\", \"localhost:7878\")\n", "\n", "# The coordinator reaches the providers by container name over the shared Docker\n", "# network (ff-net), so these are container names + internal ports, not host ports.\n", "CLICKHOUSE_HOST = os.getenv(\"CLICKHOUSE_HOST\", \"clickhouse\")\n", "CLICKHOUSE_NATIVE_PORT = int(os.getenv(\"CLICKHOUSE_NATIVE_PORT\", \"9000\"))\n", "CLICKHOUSE_USER = os.getenv(\"CLICKHOUSE_USER\", \"default\")\n", "CLICKHOUSE_PASSWORD = os.getenv(\"CLICKHOUSE_PASSWORD\", \"featureform\")\n", "CLICKHOUSE_DATABASE = os.getenv(\"CLICKHOUSE_DATABASE\", \"default\")\n", "\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"redis\")\n", "REDIS_PORT = int(os.getenv(\"REDIS_PORT\", \"6379\"))\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")" ] }, { "cell_type": "markdown", "id": "embed-hdr", "metadata": {}, "source": [ "### Embed products and load them into ClickHouse\n", "\n", "We embed each product description with `all-MiniLM-L6-v2` (384-dimensional vectors) and store the vectors in a ClickHouse `Array(Float32)` column. `DIMS` must match both the model and the `ff.Embedding` definition later." ] }, { "cell_type": "code", "execution_count": 4, "id": "embed", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/justin.cechmanek/.pyenv/versions/redis-ai-res/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "loaded 8 products, 384-dim embeddings\n" ] } ], "source": [ "# NBVAL_SKIP\n", "import clickhouse_connect\n", "from sentence_transformers import SentenceTransformer\n", "\n", "PRODUCTS = [\n", " (\"p01\", \"Wireless noise-cancelling over-ear headphones\"),\n", " (\"p02\", \"Bluetooth portable speaker, waterproof\"),\n", " (\"p03\", \"Ergonomic mechanical keyboard with RGB backlight\"),\n", " (\"p04\", \"4K ultra-wide gaming monitor, 144Hz\"),\n", " (\"p05\", \"Stainless steel insulated water bottle\"),\n", " (\"p06\", \"Cast iron skillet, pre-seasoned\"),\n", " (\"p07\", \"Trail running shoes with grip sole\"),\n", " (\"p08\", \"Merino wool hiking socks, 3-pack\"),\n", "]\n", "\n", "model = SentenceTransformer(\"all-MiniLM-L6-v2\")\n", "DIMS = model.get_sentence_embedding_dimension() # 384\n", "\n", "ids = [p[0] for p in PRODUCTS]\n", "names = [p[1] for p in PRODUCTS]\n", "vectors = model.encode(names).tolist()\n", "\n", "# ClickHouse HTTP port 8123 is published to the host, so we load from localhost.\n", "ch = clickhouse_connect.get_client(host=\"localhost\", port=8123,\n", " username=CLICKHOUSE_USER, password=CLICKHOUSE_PASSWORD)\n", "ch.command(\"DROP TABLE IF EXISTS products\")\n", "ch.command(\n", " \"\"\"\n", " CREATE TABLE products (\n", " id String,\n", " name String,\n", " embedding Array(Float32)\n", " ) ENGINE = MergeTree ORDER BY id\n", " \"\"\"\n", ")\n", "ch.insert(\"products\", list(zip(ids, names, vectors)),\n", " column_names=[\"id\", \"name\", \"embedding\"])\n", "print(f\"loaded {len(ids)} products, {DIMS}-dim embeddings\")" ] }, { "cell_type": "markdown", "id": "prov-hdr", "metadata": {}, "source": [ "## Register the providers" ] }, { "cell_type": "code", "execution_count": 5, "id": "prov", "metadata": {}, "outputs": [], "source": [ "import featureform as ff\n", "\n", "# Workaround for bugs in featureform 1.15.8 (latest release) that break the\n", "# embedding/vector-search path only:\n", "# 1. featureform/types.py uses `pb` but never imports it -> NameError on apply()\n", "# 2. VectorType.from_proto references an undefined `protoVal` -> NameError\n", "# 3. client.nearest() calls impl._nearest, but the method is named `nearest`\n", "import featureform.types as _ff_types\n", "from featureform.proto import metadata_pb2 as _ff_pb\n", "from featureform.enums import ScalarType as _ff_ScalarType\n", "from featureform.serving import HostedClientImpl as _ff_hosted\n", "_ff_types.pb = _ff_pb\n", "_ff_hosted._nearest = _ff_hosted.nearest\n", "_ff_types.VectorType.from_proto = classmethod(\n", " lambda cls, v: cls(_ff_ScalarType.from_proto(v.scalar), v.dimension, v.is_embedding)\n", ")\n", "\n", "clickhouse = ff.register_clickhouse(\n", " name=\"clickhouse-quickstart\",\n", " description=\"ClickHouse offline store holding product vectors\",\n", " host=CLICKHOUSE_HOST,\n", " port=CLICKHOUSE_NATIVE_PORT,\n", " user=CLICKHOUSE_USER,\n", " password=CLICKHOUSE_PASSWORD,\n", " database=CLICKHOUSE_DATABASE,\n", ")\n", "\n", "redis = ff.register_redis(\n", " name=\"redis-quickstart\",\n", " description=\"Redis online (inference) store\",\n", " host=REDIS_HOST,\n", " port=REDIS_PORT,\n", " password=REDIS_PASSWORD,\n", " db=0,\n", ")" ] }, { "cell_type": "markdown", "id": "src-hdr", "metadata": {}, "source": [ "## Register the source and the embedding feature\n", "\n", "We register the `products` table, then declare an `ff.Embedding` over its vector column. `vector_db=redis` tells Featureform to materialize the vectors into Redis and build a vector index there; `dims` must match the model. `@ff.entity` keys the embedding by product." ] }, { "cell_type": "code", "execution_count": 6, "id": "src", "metadata": {}, "outputs": [], "source": [ "products = clickhouse.register_table(\n", " name=\"products\",\n", " variant=\"quickstart\",\n", " table=\"products\",\n", ")\n", "\n", "@ff.entity\n", "class Product:\n", " product_embedding = ff.Embedding(\n", " products[[\"id\", \"embedding\"]],\n", " dims=DIMS,\n", " vector_db=redis,\n", " variant=\"quickstart\",\n", " description=\"Sentence-transformer embedding of the product description\",\n", " )" ] }, { "cell_type": "markdown", "id": "apply-hdr", "metadata": {}, "source": [ "## Apply\n", "\n", "`client.apply()` registers everything and materializes the vectors into Redis, building the searchable index." ] }, { "cell_type": "code", "execution_count": 7, "id": "apply", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
UserWarning: install \"ipywidgets\" for Jupyter support\n",
       "
\n" ], "text/plain": [ "UserWarning: install \"ipywidgets\" for Jupyter support\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Applying Run: 2026-07-24t15-46-15\n", "Creating User default_owner \n", "Creating Provider clickhouse-quickstart \n", "Creating Provider redis-quickstart \n", "Creating Source Variant products quickstart\n", "Creating Entity product \n", "Creating Feature Variant product_embedding quickstart\n", "\n" ] }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "client = ff.Client(host=FEATUREFORM_HOST, insecure=True)\n",
    "client.apply(asynchronous=False, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "search-hdr",
   "metadata": {},
   "source": [
    "## Semantic search from Redis\n",
    "\n",
    "Embed a free-text query with the **same model**, then ask Redis for the nearest product embeddings. `client.nearest()` returns the entity keys (product ids) of the closest vectors — served from the Redis index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "search",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query: 'something to listen to music outdoors'\n",
      "\n",
      "  p02: Bluetooth portable speaker, waterproof\n",
      "  p01: Wireless noise-cancelling over-ear headphones\n",
      "  p03: Ergonomic mechanical keyboard with RGB backlight\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "import time\n",
    "\n",
    "query = \"something to listen to music outdoors\"\n",
    "query_vec = model.encode(query).tolist()\n",
    "\n",
    "# Materialization into Redis is eventually consistent: apply() can return just\n",
    "# before the vectors are queryable. Retry briefly until the index answers.\n",
    "for attempt in range(10):\n",
    "    try:\n",
    "        neighbors = client.nearest((\"product_embedding\", \"quickstart\"), query_vec, k=3)\n",
    "        break\n",
    "    except Exception:\n",
    "        if attempt == 9:\n",
    "            raise\n",
    "        time.sleep(2)\n",
    "\n",
    "name_by_id = dict(zip(ids, names))\n",
    "print(f\"query: {query!r}\\n\")\n",
    "for pid in neighbors:\n",
    "    print(f\"  {pid}: {name_by_id.get(pid, '?')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recap",
   "metadata": {},
   "source": [
    "### What just happened\n",
    "\n",
    "The nearest neighbors came back from **Redis**, not from re-scanning the source. The embedding is a normal Featureform feature — versioned and defined once — that happens to be served through a vector index. The dashboard at **http://localhost** shows it alongside every other feature, with its source lineage intact."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cleanup-hdr",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "Remove the coordinator and both stores. Because the setup cell tears these down and recreates them, you can re-run this notebook top-to-bottom any time and get a clean, fully materialized run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cleanup",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "featureform\n",
      "clickhouse\n",
      "redis\n",
      "ff-net\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_SKIP\n",
    "# Remove everything this notebook started (coordinator, both stores, network).\n",
    "!docker rm -f featureform clickhouse redis\n",
    "!docker network rm ff-net"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "more",
   "metadata": {},
   "source": [
    "## Learn more\n",
    "\n",
    "- [Featureform embeddings & vector search](https://docs.featureform.com/)\n",
    "- [Featureform + Redis fraud detection recipe](./02_featureform_fraud_detection.ipynb)\n",
    "- [RedisVL vector search recipes](../vector-search/) — using Redis as a vector database directly"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}