{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "# Migrating from HNSW to SVS-VAMANA\n", "\n", "## Let's Begin!\n", "\"Open\n", "\n", "This notebook demonstrates how to migrate existing HNSW vector indices to SVS-VAMANA for improved memory efficiency while maintaining search quality.\n", "\n", "## What You'll Learn\n", "\n", "- How to assess your current HNSW index for migration\n", "- Step-by-step migration from HNSW to SVS-VAMANA\n", "- Memory usage comparison and cost analysis\n", "- Search quality validation between HNSW and SVS-VAMANA\n", "- Performance benchmarking and recall comparison\n", "- Migration decision framework for production systems\n", "\n", "## Prerequisites\n", "\n", "- Redis 8.2.0+ (with the search module active)\n", "- Existing HNSW index with substantial data (1000+ documents recommended)\n", "- High-dimensional vectors (768+ dimensions for best compression benefits)\n", "\n", "## HNSW vs SVS-VAMANA\n", "\n", "**HNSW (Hierarchical Navigable Small World):**\n", "- Excellent search quality and recall\n", "- Fast query performance\n", "- Higher memory usage (stores full-precision vectors)\n", "- Good for applications prioritizing search quality\n", "\n", "**SVS-VAMANA:**\n", "- Competitive search quality with compression\n", "- Significant memory savings (50-75% reduction)\n", "- Built-in vector compression (LeanVec, quantization)\n", "- Ideal for large-scale deployments with cost constraints" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ๐Ÿ“‹ HNSW to SVS-VAMANA Migration Checklist\n", "\n", "**PRE-MIGRATION:**\n", "- โ˜ Backup existing HNSW index data\n", "- โ˜ Test migration on staging environment\n", "- โ˜ Validate search quality with real queries\n", "- โ˜ Measure baseline HNSW performance metrics\n", "- โ˜ Plan rollback strategy\n", "- โ˜ Document current HNSW parameters (M, EF_construction, EF_runtime)\n", "\n", "**MIGRATION:**\n", "- โ˜ Create SVS-VAMANA index with tested configuration\n", "- โ˜ Migrate data in batches during low-traffic periods\n", "- โ˜ Monitor memory usage and indexing progress\n", "- โ˜ Validate data integrity after migration\n", "- โ˜ Test search functionality thoroughly\n", "- โ˜ Compare recall metrics with baseline\n", "\n", "**POST-MIGRATION:**\n", "- โ˜ Monitor search performance and quality\n", "- โ˜ Track memory usage and cost savings\n", "- โ˜ Update application configuration\n", "- โ˜ Document new SVS-VAMANA settings\n", "- โ˜ Clean up old HNSW index after validation period\n", "- โ˜ Update monitoring and alerting thresholds\n", "\n", "**๐Ÿ’ก HNSW-SPECIFIC TIPS:**\n", "- HNSW indices are more complex to rebuild than FLAT\n", "- Consider the impact on applications using EF_runtime tuning\n", "- SVS-VAMANA may have different optimal query parameters\n", "- Test with your specific HNSW configuration (M, EF values)\n", "- Monitor for 48-72 hours before removing HNSW index\n", "- Keep compression settings documented for future reference" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ๐Ÿ“ฆ Installation & Setup\n", "\n", "This notebook uses **RedisVL vectorizers** for generating embeddings and **Redis Stack** for vector search.\n", "\n", "**Requirements:**\n", "- Redis 8.2.0+ (with the search module active, for SVS-VAMANA support)\n", "- redisvl>=0.11.0 (required for SVS-VAMANA migration features and vectorizers)\n", "- redis-py>=6.4.0 (required for compatibility with RedisVL 0.11.0+)\n", "- numpy (for vector operations)\n", "\n", "**โš ๏ธ Important:** If you encounter Redis connection errors, upgrade redis-py: `pip install -U \"redis>=6.4.0\"`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Install Packages" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/Users/nitin.kanukolanu/workspace/redis-vl-python/.venv/bin/python3: No module named pip\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": "%pip install \"redisvl>=0.11.0\" \"redis>=6.4.0\" \"numpy>=1.21.0\" \"sentence-transformers>=2.2.0\"" }, { "metadata": {}, "cell_type": "markdown", "source": "### Download Sample Data" }, { "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:18.087794Z", "start_time": "2025-12-09T15:18:14.222219Z" } }, "cell_type": "code", "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cloning into 'temp_repo'...\r\n", "remote: Enumerating objects: 2879, done.\u001B[K\r\n", "remote: Counting objects: 100% (1068/1068), done.\u001B[K\r\n", "remote: Compressing objects: 100% (492/492), done.\u001B[K\r\n", "remote: Total 2879 (delta 784), reused 733 (delta 574), pack-reused 1811 (from 2)\u001B[K\r\n", "Receiving objects: 100% (2879/2879), 68.57 MiB | 34.97 MiB/s, done.\r\n", "Resolving deltas: 100% (1649/1649), done.\r\n", "mv: rename temp_repo/python-recipes/vector-search/resources to ./resources: Directory not empty\r\n" ] } ], "execution_count": 3, "source": [ "# NBVAL_SKIP\n", "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n", "!mv temp_repo/python-recipes/vector-search/resources .\n", "!rm -rf temp_repo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Install Redis\n", "\n", "Later in this tutorial, Redis will be used to store, index, and query vector\n", "embeddings and full text fields. **We need to have a Redis\n", "instance available.**\n", "\n", "#### Local Redis\n", "Use the shell script below to download, extract, and install [Redis](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/apt/) directly from the Redis package archive for a Linux environment." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "%%sh\n", "sudo apt-get install lsb-release curl gpg\n", "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", "sudo chmod 644 /usr/share/keyrings/redis-archive-keyring.gpg\n", "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", "sudo apt-get update\n", "sudo apt-get install redis\n", "\n", "redis-server --version\n", "redis-server --daemonize yes --loadmodule /usr/lib/redis/modules/redisearch.so" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Alternative Redis Access (Cloud, Docker, other)\n", "There are many ways to get the necessary redis-stack instance running\n", "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", "own version of Redis Enterprise running, that works too!\n", "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", "3. With docker: `docker run -d --name redis -p 6379:6379 redis:latest`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define the Redis Connection URL\n", "\n", "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:10.599659Z", "start_time": "2025-12-09T15:18:10.050008Z" } }, "source": [ "import os\n", "import json\n", "import numpy as np\n", "import time\n", "\n", "# Redis and RedisVL imports\n", "import redis\n", "from redisvl.index import SearchIndex\n", "from redisvl.query import VectorQuery\n", "from redisvl.redis.utils import array_to_buffer, buffer_to_array\n", "from redisvl.utils import CompressionAdvisor\n", "from redisvl.redis.connection import supports_svs\n", "\n", "# RedisVL Vectorizer imports\n", "from redisvl.utils.vectorize import HFTextVectorizer\n", "\n", "# Replace values below with your own if using Redis Cloud instance\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", "\n", "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"\n", "\n", "print(\"๐Ÿ“š Libraries imported successfully!\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“š Libraries imported successfully!\n" ] } ], "execution_count": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Verify Redis and SVS Support\n", "\n", "First, let's ensure Redis Stack is running and supports SVS-VAMANA." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:12.750534Z", "start_time": "2025-12-09T15:18:12.741314Z" } }, "source": [ "# Test Redis connection and SVS support\n", "try:\n", " client = redis.Redis.from_url(REDIS_URL)\n", " client.ping()\n", " print(\"โœ… Redis connection successful\")\n", " \n", " # Check Redis version\n", " redis_info = client.info()\n", " redis_version = redis_info['redis_version']\n", " print(f\"๐Ÿ“Š Redis version: {redis_version}\")\n", " \n", " # Check SVS support\n", " if supports_svs(client):\n", " print(\"โœ… SVS-VAMANA supported\")\n", " else:\n", " print(\"โŒ SVS-VAMANA not supported\")\n", " print(\"Please ensure you're using Redis Stack 8.2.0+ with RediSearch 2.8.10+\")\n", " \n", "except Exception as e:\n", " print(f\"โŒ Redis connection failed: {e}\")\n", " print(\"Please ensure Redis Stack is running on localhost:6379\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "โœ… Redis connection successful\n", "๐Ÿ“Š Redis version: 8.4.0\n", "โœ… SVS-VAMANA supported\n" ] } ], "execution_count": 2 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Load Sample Data\n", "\n", "We'll use the movie dataset to demonstrate the migration process." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:24.996005Z", "start_time": "2025-12-09T15:18:24.992212Z" } }, "source": [ "# Load the movies dataset\n", "with open('resources/movies.json', 'r') as f:\n", " movies_data = json.load(f)\n", "\n", "print(\n", " f\"๐Ÿ“ฝ๏ธ Loaded {len(movies_data)} movie records\",\n", " f\"Sample movie: {movies_data[0]['title']}\",\n", " f\"Genres available: {set(movie['genre'] for movie in movies_data)}\",\n", " sep=\"\\n\"\n", ")\n", "\n", "# Configuration for demonstration \n", "dims = 768 # Using all-mpnet-base-v2 model (768 dimensions)\n", "num_docs = len(movies_data) # Use actual dataset size\n", "\n", "print(\n", " f\"\\n๐Ÿ”ง Configuration:\",\n", " f\"Vector dimensions: {dims}\",\n", " f\"Dataset size: {num_docs} movie documents\",\n", " f\"Vectorizer: RedisVL HFTextVectorizer\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“ฝ๏ธ Loaded 20 movie records\n", "Sample movie: Explosive Pursuit\n", "Genres available: {'action', 'comedy'}\n", "\n", "๐Ÿ”ง Configuration:\n", "Vector dimensions: 768\n", "Dataset size: 20 movie documents\n", "Vectorizer: RedisVL HFTextVectorizer\n" ] } ], "execution_count": 4 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3: Create HNSW Index\n", "\n", "First, we'll create an HNSW index with typical production settings." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:27.075125Z", "start_time": "2025-12-09T15:18:27.065560Z" } }, "source": [ "# Create HNSW schema with production-like settings\n", "hnsw_schema = {\n", " \"index\": {\n", " \"name\": \"hnsw_demo_index\",\n", " \"prefix\": \"demo:hnsw:\",\n", " },\n", " \"fields\": [\n", " {\"name\": \"movie_id\", \"type\": \"tag\"},\n", " {\"name\": \"title\", \"type\": \"text\"},\n", " {\"name\": \"genre\", \"type\": \"tag\"},\n", " {\"name\": \"rating\", \"type\": \"numeric\"},\n", " {\"name\": \"description\", \"type\": \"text\"},\n", " {\n", " \"name\": \"embedding\",\n", " \"type\": \"vector\",\n", " \"attrs\": {\n", " \"dims\": dims,\n", " \"algorithm\": \"hnsw\",\n", " \"datatype\": \"float32\",\n", " \"distance_metric\": \"cosine\",\n", " \"m\": 16, # Number of bi-directional links for each node\n", " \"ef_construction\": 200, # Size of dynamic candidate list\n", " \"ef_runtime\": 10 # Size of dynamic candidate list during search\n", " }\n", " }\n", " ]\n", "}\n", "\n", "print(\"Creating HNSW index with optimized settings...\")\n", "hnsw_index = SearchIndex.from_dict(hnsw_schema, redis_url=REDIS_URL)\n", "hnsw_index.create(overwrite=True)\n", "print(f\"โœ… Created HNSW index: {hnsw_index.name}\")\n", "\n", "# Display HNSW configuration\n", "print(\n", " \"\\n๐Ÿ”ง HNSW Configuration:\",\n", " f\"M (connections per node): 16\",\n", " f\"EF Construction: 200\",\n", " f\"EF Runtime: 10\",\n", " f\"Distance metric: cosine\",\n", " f\"Data type: float32\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Creating HNSW index with optimized settings...\n", "โœ… Created HNSW index: hnsw_demo_index\n", "\n", "๐Ÿ”ง HNSW Configuration:\n", "M (connections per node): 16\n", "EF Construction: 200\n", "EF Runtime: 10\n", "Distance metric: cosine\n", "Data type: float32\n" ] } ], "execution_count": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 4: Generate Embeddings and Load HNSW Index\n", "\n", "Generate embeddings for movie descriptions and populate the HNSW index." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:38.464659Z", "start_time": "2025-12-09T15:18:28.435730Z" } }, "source": [ "# Generate embeddings using RedisVL vectorizers\n", "print(\"๐Ÿ”„ Generating embeddings for movie descriptions...\")\n", "\n", "descriptions = [movie['description'] for movie in movies_data]\n", "\n", "# Use RedisVL HFTextVectorizer\n", "print(\"๐Ÿš€ Using RedisVL HFTextVectorizer...\")\n", "vectorizer = HFTextVectorizer(\n", " model=\"sentence-transformers/all-mpnet-base-v2\", # 768 dimensions\n", ")\n", "\n", "# Generate embeddings using RedisVL vectorizer\n", "embeddings = vectorizer.embed_many(descriptions)\n", "embeddings = np.array(embeddings, dtype=np.float32)\n", "\n", "print(f\"โœ… Generated {len(embeddings)} real embeddings using RedisVL HFTextVectorizer\")\n", "print(f\"๐Ÿ“Š Embedding shape: {embeddings.shape}\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ”„ Generating embeddings for movie descriptions...\n", "๐Ÿš€ Using RedisVL HFTextVectorizer...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/vishal.bala/PycharmProjects/redis-ai-resources/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "16:18:34 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n", "16:18:34 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-mpnet-base-v2\n", "โœ… Generated 20 real embeddings using RedisVL HFTextVectorizer\n", "๐Ÿ“Š Embedding shape: (20, 768)\n" ] } ], "execution_count": 6 }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:40.269494Z", "start_time": "2025-12-09T15:18:40.266771Z" } }, "source": [ "# Prepare data for loading into HNSW index\n", "sample_data = []\n", "for i, movie in enumerate(movies_data):\n", " sample_data.append({\n", " 'movie_id': str(movie['id']),\n", " 'title': movie['title'],\n", " 'genre': movie['genre'],\n", " 'rating': movie['rating'],\n", " 'description': movie['description'],\n", " 'embedding': array_to_buffer(embeddings[i].astype(np.float32), dtype='float32')\n", " })\n", "\n", "print(f\"๐Ÿ“ฆ Prepared {len(sample_data)} documents for indexing\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“ฆ Prepared 20 documents for indexing\n" ] } ], "execution_count": 7 }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:46.060722Z", "start_time": "2025-12-09T15:18:41.031015Z" } }, "source": [ "# Load data into HNSW index\n", "print(\"๐Ÿ“ฅ Loading data into HNSW index...\")\n", "batch_size = 100 # Process in batches\n", "\n", "for i in range(0, len(sample_data), batch_size):\n", " batch = sample_data[i:i+batch_size]\n", " hnsw_index.load(batch)\n", " print(f\" Loaded {min(i+batch_size, len(sample_data))}/{len(sample_data)} documents\")\n", "\n", "# Wait for indexing to complete\n", "print(\"โณ Waiting for HNSW indexing to complete...\")\n", "time.sleep(5) # HNSW indexing takes longer than FLAT\n", "\n", "hnsw_info = hnsw_index.info()\n", "print(\n", " f\"\\nโœ… HNSW index loaded with {hnsw_info['num_docs']} documents\",\n", " f\"Index size: {hnsw_info.get('vector_index_sz_mb', 'N/A')} MB\",\n", " f\"Indexing time: ~5 seconds (HNSW graph construction)\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“ฅ Loading data into HNSW index...\n", " Loaded 20/20 documents\n", "โณ Waiting for HNSW indexing to complete...\n", "\n", "โœ… HNSW index loaded with 20 documents\n", "Index size: 3.2259750366210938 MB\n", "Indexing time: ~5 seconds (HNSW graph construction)\n" ] } ], "execution_count": 8 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 5: Get Compression Recommendation\n", "\n", "Use the CompressionAdvisor to get optimal SVS-VAMANA settings for our data." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:47.623552Z", "start_time": "2025-12-09T15:18:47.619953Z" } }, "source": [ "# Get compression recommendation\n", "print(\"๐Ÿ” Analyzing data for optimal compression settings...\")\n", "\n", "# Get recommendations for different priorities\n", "memory_config = CompressionAdvisor.recommend(dims=dims, priority=\"memory\")\n", "balanced_config = CompressionAdvisor.recommend(dims=dims, priority=\"balanced\")\n", "performance_config = CompressionAdvisor.recommend(dims=dims, priority=\"performance\")\n", "\n", "print(\n", " \"\\n๐Ÿ“Š Compression Recommendations:\",\n", " \"\",\n", " \"๐Ÿ—œ๏ธ Memory Priority:\",\n", " f\" Algorithm: {memory_config.algorithm}\",\n", " f\" Compression: {memory_config.compression if hasattr(memory_config, 'compression') else 'None'}\",\n", " f\" Datatype: {memory_config.datatype}\",\n", " f\" Dimensions: {dims} โ†’ {memory_config.reduce if hasattr(memory_config, 'reduce') else dims}\",\n", " \"\",\n", " \"โš–๏ธ Balanced Priority:\",\n", " f\" Algorithm: {balanced_config.algorithm}\",\n", " f\" Compression: {balanced_config.compression if hasattr(balanced_config, 'compression') else 'None'}\",\n", " f\" Datatype: {balanced_config.datatype}\",\n", " f\" Dimensions: {dims} โ†’ {balanced_config.reduce if hasattr(balanced_config, 'reduce') else dims}\",\n", " \"\",\n", " \"โšก Performance Priority:\",\n", " f\" Algorithm: {performance_config.algorithm}\",\n", " f\" Compression: {performance_config.compression if hasattr(performance_config, 'compression') else 'None'}\",\n", " f\" Datatype: {performance_config.datatype}\",\n", " f\" Dimensions: {dims} โ†’ {performance_config.reduce if hasattr(performance_config, 'reduce') else dims}\",\n", " sep=\"\\n\"\n", ")\n", "\n", "# Select configuration (using memory priority for maximum savings)\n", "selected_config = memory_config\n", "# Use reduce if it exists and is not None, otherwise use original dims\n", "target_dims = selected_config.reduce if (hasattr(selected_config, 'reduce') and selected_config.reduce is not None) else dims\n", "target_dtype = selected_config.datatype\n", "\n", "print(\n", " f\"\\nโœ… Selected configuration: Memory Priority\",\n", " f\"Expected memory reduction: ~{((dims - target_dims) / dims * 100):.1f}% from dimension reduction\",\n", " f\"Additional savings from {selected_config.datatype} compression\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ” Analyzing data for optimal compression settings...\n", "\n", "๐Ÿ“Š Compression Recommendations:\n", "\n", "๐Ÿ—œ๏ธ Memory Priority:\n", " Algorithm: svs-vamana\n", " Compression: LVQ4\n", " Datatype: float32\n", " Dimensions: 768 โ†’ None\n", "\n", "โš–๏ธ Balanced Priority:\n", " Algorithm: svs-vamana\n", " Compression: LVQ4x4\n", " Datatype: float32\n", " Dimensions: 768 โ†’ None\n", "\n", "โšก Performance Priority:\n", " Algorithm: svs-vamana\n", " Compression: LVQ4x4\n", " Datatype: float32\n", " Dimensions: 768 โ†’ None\n", "\n", "โœ… Selected configuration: Memory Priority\n", "Expected memory reduction: ~0.0% from dimension reduction\n", "Additional savings from float32 compression\n" ] } ], "execution_count": 9 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 6: Create SVS-VAMANA Index\n", "\n", "Create the SVS-VAMANA index with the recommended compression settings." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:49.967891Z", "start_time": "2025-12-09T15:18:49.957951Z" } }, "source": [ "# Create SVS-VAMANA schema with compression\n", "svs_schema = {\n", " \"index\": {\n", " \"name\": \"svs_demo_index\",\n", " \"prefix\": \"demo:svs:\",\n", " },\n", " \"fields\": [\n", " {\"name\": \"movie_id\", \"type\": \"tag\"},\n", " {\"name\": \"title\", \"type\": \"text\"},\n", " {\"name\": \"genre\", \"type\": \"tag\"},\n", " {\"name\": \"rating\", \"type\": \"numeric\"},\n", " {\"name\": \"description\", \"type\": \"text\"},\n", " {\n", " \"name\": \"embedding\",\n", " \"type\": \"vector\",\n", " \"attrs\": {\n", " \"dims\": target_dims, # Use reduced dimensions\n", " \"algorithm\": \"svs-vamana\",\n", " \"datatype\": selected_config.datatype,\n", " \"distance_metric\": \"cosine\"\n", " # Note: Don't include the full selected_config to avoid dims/reduce conflict\n", " }\n", " }\n", " ]\n", "}\n", "\n", "print(\"Creating SVS-VAMANA index with compression...\")\n", "svs_index = SearchIndex.from_dict(svs_schema, redis_url=REDIS_URL)\n", "svs_index.create(overwrite=True)\n", "print(\n", " f\"โœ… Created SVS-VAMANA index: {svs_index.name}\",\n", " f\"Compression: {selected_config.compression if hasattr(selected_config, 'compression') else 'None'}\",\n", " f\"Datatype: {selected_config.datatype}\",\n", " f\"Dimensions: {dims} โ†’ {target_dims}\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Creating SVS-VAMANA index with compression...\n", "โœ… Created SVS-VAMANA index: svs_demo_index\n", "Compression: LVQ4\n", "Datatype: float32\n", "Dimensions: 768 โ†’ 768\n" ] } ], "execution_count": 10 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 7: Migrate Data from HNSW to SVS-VAMANA\n", "\n", "Extract data from the HNSW index and migrate it to SVS-VAMANA with compression." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:51.877234Z", "start_time": "2025-12-09T15:18:51.862453Z" } }, "source": [ "# Extract data from HNSW index\n", "print(\"๐Ÿ”„ Extracting data from HNSW index...\")\n", "\n", "client = redis.Redis.from_url(REDIS_URL)\n", "keys = client.keys(\"demo:hnsw:*\")\n", "print(f\"Found {len(keys)} documents to migrate\")\n", "\n", "# Process and transform data for SVS index\n", "svs_data = []\n", "\n", "for key in keys:\n", " doc_data = client.hgetall(key)\n", " \n", " if b'embedding' in doc_data:\n", " # Extract original vector from HNSW index\n", " original_vector = np.array(buffer_to_array(doc_data[b'embedding'], dtype='float32'))\n", " \n", " # Apply dimensionality reduction if needed (LeanVec)\n", " if target_dims < dims:\n", " vector = original_vector[:target_dims]\n", " else:\n", " vector = original_vector\n", " \n", " # Convert to target datatype\n", " if target_dtype == 'float16':\n", " vector = vector.astype(np.float16)\n", " \n", " svs_data.append({\n", " \"movie_id\": doc_data[b'movie_id'].decode(),\n", " \"title\": doc_data[b'title'].decode(),\n", " \"genre\": doc_data[b'genre'].decode(),\n", " \"rating\": int(doc_data[b'rating'].decode()),\n", " \"description\": doc_data[b'description'].decode(),\n", " \"embedding\": array_to_buffer(vector, dtype=target_dtype)\n", " })\n", "\n", "print(f\"Prepared {len(svs_data)} documents for migration\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ”„ Extracting data from HNSW index...\n", "Found 20 documents to migrate\n", "Prepared 20 documents for migration\n" ] } ], "execution_count": 11 }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:57.887145Z", "start_time": "2025-12-09T15:18:52.866035Z" } }, "source": [ "# Load data into SVS index\n", "print(\"๐Ÿ“ฅ Loading data into SVS-VAMANA index...\")\n", "batch_size = 100 # Define batch size for migration\n", "\n", "if len(svs_data) > 0:\n", " for i in range(0, len(svs_data), batch_size):\n", " batch = svs_data[i:i+batch_size]\n", " svs_index.load(batch)\n", " print(f\" Migrated {min(i+batch_size, len(svs_data))}/{len(svs_data)} documents\")\n", "\n", " # Wait for indexing to complete\n", " print(\"โณ Waiting for SVS-VAMANA indexing to complete...\")\n", " time.sleep(5)\n", "\n", " svs_info = svs_index.info()\n", " print(\n", " f\"\\nโœ… Migration complete! SVS index has {svs_info['num_docs']} documents\",\n", " f\"Index size: {svs_info.get('vector_index_sz_mb', 'N/A')} MB\",\n", " sep=\"\\n\"\n", " )\n", "else:\n", " print(\"โš ๏ธ No data to migrate. Make sure the HNSW index was populated first.\")\n", " print(\" Run the previous cells to load data into the HNSW index.\")\n", " svs_info = svs_index.info()" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“ฅ Loading data into SVS-VAMANA index...\n", " Migrated 20/20 documents\n", "โณ Waiting for SVS-VAMANA indexing to complete...\n", "\n", "โœ… Migration complete! SVS index has 20 documents\n", "Index size: 3.0177383422851563 MB\n" ] } ], "execution_count": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 8: Compare Memory Usage\n", "\n", "Analyze the memory savings achieved through the HNSW to SVS-VAMANA migration." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:18:59.379258Z", "start_time": "2025-12-09T15:18:59.376792Z" } }, "source": [ "# Helper function to extract memory info\n", "def get_memory_mb(index_info):\n", " \"\"\"Extract memory usage in MB from index info\"\"\"\n", " memory = index_info.get('vector_index_sz_mb', 0)\n", " if isinstance(memory, str):\n", " try:\n", " return float(memory)\n", " except ValueError:\n", " return 0.0\n", " return float(memory)\n", "\n", "# Get memory usage\n", "hnsw_memory = get_memory_mb(hnsw_info)\n", "svs_memory = get_memory_mb(svs_info)\n", "\n", "print(\n", " \"๐Ÿ“Š Memory Usage Comparison\",\n", " \"=\" * 40,\n", " f\"Original HNSW index: {hnsw_memory:.2f} MB\",\n", " f\"SVS-VAMANA index: {svs_memory:.2f} MB\",\n", " \"\",\n", " sep=\"\\n\"\n", ")\n", "\n", "if hnsw_memory > 0:\n", " if svs_memory > 0:\n", " savings = ((hnsw_memory - svs_memory) / hnsw_memory) * 100\n", " print(\n", " f\"๐Ÿ’ฐ Memory savings: {savings:.1f}%\",\n", " f\"Absolute reduction: {hnsw_memory - svs_memory:.2f} MB\",\n", " sep=\"\\n\"\n", " )\n", " else:\n", " print(\"โณ SVS index still indexing - memory comparison pending\")\n", "else:\n", " print(\"โš ๏ธ Memory information not available\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“Š Memory Usage Comparison\n", "========================================\n", "Original HNSW index: 3.23 MB\n", "SVS-VAMANA index: 3.02 MB\n", "\n", "๐Ÿ’ฐ Memory savings: 6.5%\n", "Absolute reduction: 0.21 MB\n" ] } ], "execution_count": 13 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 9: Validate Search Quality\n", "\n", "Compare search quality between HNSW and SVS-VAMANA to ensure the migration maintains acceptable recall." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:19:03.384495Z", "start_time": "2025-12-09T15:19:03.379214Z" } }, "source": [ "# Generate test queries\n", "print(\"๐Ÿ” Generating test queries for quality validation...\")\n", "\n", "np.random.seed(123) # For reproducible test queries\n", "num_test_queries = 10\n", "test_queries = []\n", "\n", "for i in range(num_test_queries):\n", " # Create test query vectors\n", " query_vec = np.random.random(dims).astype(np.float32)\n", " query_vec = query_vec / np.linalg.norm(query_vec) # Normalize\n", " test_queries.append(query_vec)\n", "\n", "print(f\"Generated {len(test_queries)} test queries\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ” Generating test queries for quality validation...\n", "Generated 10 test queries\n" ] } ], "execution_count": 14 }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:19:04.652101Z", "start_time": "2025-12-09T15:19:04.641255Z" } }, "source": [ "# Test HNSW search quality\n", "print(\"๐Ÿ” Testing HNSW search quality...\")\n", "\n", "hnsw_results = []\n", "hnsw_start = time.time()\n", "\n", "for query_vec in test_queries:\n", " query = VectorQuery(\n", " vector=query_vec,\n", " vector_field_name=\"embedding\",\n", " return_fields=[\"movie_id\", \"title\", \"genre\"],\n", " dtype=\"float32\",\n", " num_results=10\n", " )\n", " results = hnsw_index.query(query)\n", " hnsw_results.append([doc[\"movie_id\"] for doc in results])\n", "\n", "hnsw_time = time.time() - hnsw_start\n", "print(f\"HNSW search completed in {hnsw_time:.3f} seconds\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ” Testing HNSW search quality...\n", "HNSW search completed in 0.008 seconds\n" ] } ], "execution_count": 15 }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:19:05.586458Z", "start_time": "2025-12-09T15:19:05.574807Z" } }, "source": [ "# Test SVS-VAMANA search quality\n", "print(\"๐Ÿ” Testing SVS-VAMANA search quality...\")\n", "\n", "svs_results = []\n", "svs_start = time.time()\n", "\n", "for i, query_vec in enumerate(test_queries):\n", " # Adjust query vector for SVS index (handle dimensionality reduction)\n", " if target_dims < dims:\n", " svs_query_vec = query_vec[:target_dims]\n", " else:\n", " svs_query_vec = query_vec\n", " \n", " if target_dtype == 'float16':\n", " svs_query_vec = svs_query_vec.astype(np.float16)\n", " \n", " query = VectorQuery(\n", " vector=svs_query_vec,\n", " vector_field_name=\"embedding\",\n", " return_fields=[\"movie_id\", \"title\", \"genre\"],\n", " dtype=target_dtype,\n", " num_results=10\n", " )\n", " results = svs_index.query(query)\n", " svs_results.append([doc[\"movie_id\"] for doc in results])\n", "\n", "svs_time = time.time() - svs_start\n", "print(f\"SVS-VAMANA search completed in {svs_time:.3f} seconds\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ” Testing SVS-VAMANA search quality...\n", "SVS-VAMANA search completed in 0.009 seconds\n" ] } ], "execution_count": 16 }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:19:06.627904Z", "start_time": "2025-12-09T15:19:06.623676Z" } }, "source": [ "# Calculate recall and performance metrics\n", "def calculate_recall(reference_results, test_results, k=10):\n", " \"\"\"Calculate recall@k between two result sets\"\"\"\n", " if not reference_results or not test_results:\n", " return 0.0\n", " \n", " total_recall = 0.0\n", " for ref, test in zip(reference_results, test_results):\n", " ref_set = set(ref[:k])\n", " test_set = set(test[:k])\n", " if len(ref_set) > 0:\n", " recall = len(ref_set.intersection(test_set)) / len(ref_set)\n", " total_recall += recall\n", " \n", " return total_recall / len(reference_results)\n", "\n", "# Calculate metrics\n", "recall_at_5 = calculate_recall(hnsw_results, svs_results, k=5)\n", "recall_at_10 = calculate_recall(hnsw_results, svs_results, k=10)\n", "\n", "print(\n", " \"๐Ÿ“Š Search Quality Comparison\",\n", " \"=\" * 40,\n", " \"HNSW (baseline): 100% recall (exact graph-based search)\",\n", " f\"SVS-VAMANA Recall@5: {recall_at_5*100:.1f}% (vs HNSW baseline)\",\n", " f\"SVS-VAMANA Recall@10: {recall_at_10*100:.1f}% (vs HNSW baseline)\",\n", " \"\",\n", " \"โฑ๏ธ Performance Comparison:\",\n", " f\"HNSW query time: {hnsw_time:.3f}s ({hnsw_time/num_test_queries*1000:.1f}ms per query)\",\n", " f\"SVS-VAMANA query time: {svs_time:.3f}s ({svs_time/num_test_queries*1000:.1f}ms per query)\",\n", " f\"Speed difference: {((hnsw_time - svs_time) / hnsw_time * 100):+.1f}%\",\n", " sep=\"\\n\"\n", ")\n", "\n", "# Quality assessment\n", "if recall_at_10 >= 0.95:\n", " quality_assessment = \"๐ŸŸข Excellent - Minimal quality loss\"\n", "elif recall_at_10 >= 0.90:\n", " quality_assessment = \"๐ŸŸก Good - Acceptable quality for most applications\"\n", "elif recall_at_10 >= 0.80:\n", " quality_assessment = \"๐ŸŸ  Fair - Consider if quality requirements are flexible\"\n", "else:\n", " quality_assessment = \"๐Ÿ”ด Poor - Migration not recommended\"\n", "\n", "print(f\"\\n๐ŸŽฏ Quality Assessment: {quality_assessment}\")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“Š Search Quality Comparison\n", "========================================\n", "HNSW (baseline): 100% recall (exact graph-based search)\n", "SVS-VAMANA Recall@5: 100.0% (vs HNSW baseline)\n", "SVS-VAMANA Recall@10: 100.0% (vs HNSW baseline)\n", "\n", "โฑ๏ธ Performance Comparison:\n", "HNSW query time: 0.008s (0.8ms per query)\n", "SVS-VAMANA query time: 0.009s (0.9ms per query)\n", "Speed difference: -10.7%\n", "\n", "๐ŸŽฏ Quality Assessment: ๐ŸŸข Excellent - Minimal quality loss\n" ] } ], "execution_count": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 10: Migration Decision Framework\n", "\n", "Based on the analysis, determine if migration is recommended." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:19:19.464223Z", "start_time": "2025-12-09T15:19:19.460488Z" } }, "source": [ "# Migration decision logic\n", "memory_savings_threshold = 5 # Minimum % memory savings\n", "recall_threshold = 0.85 # Minimum 85% recall@10\n", "\n", "memory_savings_pct = ((hnsw_memory - svs_memory) / hnsw_memory * 100) if hnsw_memory > 0 and svs_memory > 0 else 0\n", "meets_memory_threshold = memory_savings_pct >= memory_savings_threshold\n", "meets_quality_threshold = recall_at_10 >= recall_threshold\n", "\n", "print(\n", " \"๐Ÿค” Migration Decision Analysis\",\n", " \"=\" * 40,\n", " \"\",\n", " \"๐Ÿ“Š Criteria Evaluation:\",\n", " f\"Memory savings: {memory_savings_pct:.1f}% {'โœ…' if meets_memory_threshold else 'โŒ'} (threshold: {memory_savings_threshold}%)\",\n", " f\"Search quality: {recall_at_10:.3f} {'โœ…' if meets_quality_threshold else 'โŒ'} (threshold: {recall_threshold})\",\n", " \"\",\n", " sep=\"\\n\"\n", ")\n", "\n", "if meets_memory_threshold and meets_quality_threshold:\n", " recommendation = \"๐ŸŸข RECOMMENDED\"\n", " reasoning = \"Migration provides significant memory savings while maintaining good search quality.\"\n", "elif meets_memory_threshold and not meets_quality_threshold:\n", " recommendation = \"๐ŸŸก CONDITIONAL\"\n", " reasoning = \"Good memory savings but reduced search quality. Consider if your application can tolerate lower recall.\"\n", "elif not meets_memory_threshold and meets_quality_threshold:\n", " recommendation = \"๐ŸŸ  LIMITED BENEFIT\"\n", " reasoning = \"Search quality is maintained but memory savings are minimal. Migration may not be worth the effort.\"\n", "else:\n", " recommendation = \"๐Ÿ”ด NOT RECOMMENDED\"\n", " reasoning = \"Insufficient memory savings and/or poor search quality. Consider alternative optimization strategies.\"\n", "\n", "print(\n", " f\"๐ŸŽฏ Migration Recommendation: {recommendation}\",\n", " f\"๐Ÿ’ญ Reasoning: {reasoning}\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿค” Migration Decision Analysis\n", "========================================\n", "\n", "๐Ÿ“Š Criteria Evaluation:\n", "Memory savings: 6.5% โœ… (threshold: 5%)\n", "Search quality: 1.000 โœ… (threshold: 0.85)\n", "\n", "๐ŸŽฏ Migration Recommendation: ๐ŸŸข RECOMMENDED\n", "๐Ÿ’ญ Reasoning: Migration provides significant memory savings while maintaining good search quality.\n" ] } ], "execution_count": 18 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 12: Cleanup\n", "\n", "Clean up the demonstration indices." ] }, { "cell_type": "code", "metadata": { "ExecuteTime": { "end_time": "2025-12-09T15:19:25.487110Z", "start_time": "2025-12-09T15:19:25.480324Z" } }, "source": [ "print(\"๐Ÿงน Cleaning up demonstration indices...\")\n", "\n", "# Clean up HNSW index\n", "try:\n", " hnsw_index.delete(drop=True)\n", " print(\"โœ… Deleted HNSW demonstration index\")\n", "except Exception as e:\n", " print(f\"โš ๏ธ Failed to delete HNSW index: {e}\")\n", "\n", "# Clean up SVS index\n", "try:\n", " svs_index.delete(drop=True)\n", " print(\"โœ… Deleted SVS-VAMANA demonstration index\")\n", "except Exception as e:\n", " print(f\"โš ๏ธ Failed to delete SVS index: {e}\")\n", "\n", "print(\n", " \"\\n๐ŸŽ‰ HNSW to SVS-VAMANA migration demonstration complete!\",\n", " \"\\nNext steps:\",\n", " \"1. Apply learnings to your production HNSW indices\",\n", " \"2. Test with your actual query patterns and data\",\n", " \"3. Monitor performance in your environment\",\n", " \"4. Consider gradual rollout strategy\",\n", " \"5. Evaluate impact on applications using HNSW-specific features\",\n", " sep=\"\\n\"\n", ")" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿงน Cleaning up demonstration indices...\n", "โœ… Deleted HNSW demonstration index\n", "โœ… Deleted SVS-VAMANA demonstration index\n", "\n", "๐ŸŽ‰ HNSW to SVS-VAMANA migration demonstration complete!\n", "\n", "Next steps:\n", "1. Apply learnings to your production HNSW indices\n", "2. Test with your actual query patterns and data\n", "3. Monitor performance in your environment\n", "4. Consider gradual rollout strategy\n", "5. Evaluate impact on applications using HNSW-specific features\n" ] } ], "execution_count": 19 } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 4 }