{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n",
    "# Migrating from FLAT to SVS-VAMANA\n",
    "\n",
    "## Let's Begin!\n",
    "<a href=\"https://colab.research.google.com/github/redis-developer/redis-ai-resources/blob/main/python-recipes/vector-search/07_flat_to_svs_vamana_migration.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "This notebook demonstrates how to migrate existing FLAT vector indices to SVS-VAMANA for improved memory efficiency and cost savings.\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "- How to assess your current FLAT index for migration\n",
    "- Step-by-step migration from FLAT to SVS-VAMANA\n",
    "- Memory usage comparison and cost analysis\n",
    "- Search quality validation\n",
    "- Performance benchmarking\n",
    "- Migration decision framework\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "- Redis 8.2.0+ (with the search module active)\n",
    "- Existing vector index with substantial data (1000+ documents recommended)\n",
    "- Vector embeddings (768 dimensions using sentence-transformers/all-mpnet-base-v2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📋 FLAT to SVS-VAMANA Migration Checklist\n",
    "\n",
    "**PRE-MIGRATION:**\n",
    "- ☐ Backup existing FLAT index data\n",
    "- ☐ Test migration on staging environment\n",
    "- ☐ Validate search quality with real queries\n",
    "- ☐ Measure baseline FLAT performance metrics\n",
    "- ☐ Plan rollback strategy\n",
    "- ☐ Document current FLAT index configuration\n",
    "\n",
    "**MIGRATION:**\n",
    "- ☐ Create SVS-VAMANA index with tested configuration\n",
    "- ☐ Migrate data in batches during low-traffic periods\n",
    "- ☐ Monitor memory usage and indexing progress\n",
    "- ☐ Validate data integrity after migration\n",
    "- ☐ Test search functionality thoroughly\n",
    "- ☐ Compare recall metrics with baseline\n",
    "\n",
    "**POST-MIGRATION:**\n",
    "- ☐ Monitor search performance and quality\n",
    "- ☐ Track memory usage and cost savings\n",
    "- ☐ Update application configuration\n",
    "- ☐ Document new SVS-VAMANA settings\n",
    "- ☐ Clean up old FLAT index after validation period\n",
    "- ☐ Update monitoring and alerting thresholds\n",
    "\n",
    "**💡 FLAT-SPECIFIC TIPS:**\n",
    "- FLAT indices are simpler to migrate than HNSW (no graph structure)\n",
    "- FLAT provides 100% recall, so focus on acceptable recall threshold for SVS-VAMANA\n",
    "- SVS-VAMANA will be faster than FLAT for large datasets\n",
    "- Memory savings are most significant with FLAT migrations\n",
    "- Consider using compression for maximum memory reduction\n",
    "- Test query performance improvements with your dataset size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📦 Installation & Setup\n",
    "\n",
    "This notebook uses **RedisVL vectorizers** for generating embeddings and **Redis Stack** for vector search.\n",
    "\n",
    "**Requirements:**\n",
    "- Redis 8.2.0+ (with the search module active, for SVS-VAMANA support)\n",
    "- redisvl>=0.11.0 (required for SVS-VAMANA migration features and vectorizers)\n",
    "- redis-py>=6.4.0 (required for compatibility with RedisVL 0.11.0+)\n",
    "- numpy (for vector operations)\n",
    "\n",
    "**⚠️ Important:** If you encounter Redis connection errors, upgrade redis-py: `pip install -U \"redis>=6.4.0\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install \"redisvl>=0.11.0\" \"redis>=6.4.0\" \"numpy>=1.21.0\" \"sentence-transformers>=2.2.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n",
    "!mv temp_repo/python-recipes/vector-search/resources .\n",
    "!rm -rf temp_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Redis\n",
    "\n",
    "Later in this tutorial, Redis will be used to store, index, and query vector\n",
    "embeddings and full text fields. **We need to have a Redis\n",
    "instance available.**\n",
    "\n",
    "#### Local Redis\n",
    "Use the shell script below to download, extract, and install [Redis](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/apt/) directly from the Redis package archive for a Linux environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NBVAL_SKIP\n",
    "%%sh\n",
    "sudo apt-get install lsb-release curl gpg\n",
    "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "sudo chmod 644 /usr/share/keyrings/redis-archive-keyring.gpg\n",
    "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
    "sudo apt-get update\n",
    "sudo apt-get install redis\n",
    "\n",
    "redis-server --version\n",
    "redis-server --daemonize yes --loadmodule /usr/lib/redis/modules/redisearch.so"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Alternative Redis Access (Cloud, Docker, other)\n",
    "There are many ways to get the necessary redis-stack instance running\n",
    "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
    "own version of Redis Enterprise running, that works too!\n",
    "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
    "3. With docker: `docker run -d --name redis -p 6379:6379 redis:latest`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the Redis Connection URL\n",
    "\n",
    "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:22:30.002812Z",
     "start_time": "2025-12-09T15:22:29.999594Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Required imports from redis-vl\n",
    "import numpy as np\n",
    "import time\n",
    "from redisvl.index import SearchIndex\n",
    "from redisvl.query import VectorQuery\n",
    "from redisvl.redis.utils import array_to_buffer, buffer_to_array\n",
    "from redisvl.utils import CompressionAdvisor\n",
    "from redisvl.redis.connection import supports_svs\n",
    "import redis\n",
    "\n",
    "# RedisVL Vectorizer imports\n",
    "from redisvl.utils.vectorize import HFTextVectorizer\n",
    "\n",
    "# Replace values below with your own if using Redis Cloud instance\n",
    "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
    "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\")      # ex: 18374\n",
    "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\")  # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
    "\n",
    "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
    "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Verify SVS-VAMANA Support\n",
    "\n",
    "First, let's ensure your Redis environment supports SVS-VAMANA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:22:30.963607Z",
     "start_time": "2025-12-09T15:22:30.953423Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Redis connection successful\n",
      "✅ SVS-VAMANA supported\n",
      "   Ready for migration!\n"
     ]
    }
   ],
   "source": [
    "# Check Redis connection and SVS support\n",
    "try:\n",
    "    client = redis.Redis.from_url(REDIS_URL)\n",
    "    client.ping()\n",
    "    print(\"✅ Redis connection successful\")\n",
    "\n",
    "    if supports_svs(client):\n",
    "        print(\"✅ SVS-VAMANA supported\")\n",
    "        print(\"   Ready for migration!\")\n",
    "    else:\n",
    "        print(\"❌ SVS-VAMANA not supported\")\n",
    "        print(\"   Requires Redis >= 8.2.0 with RediSearch >= 2.8.10\")\n",
    "        print(\"   Please upgrade Redis Stack before proceeding\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"❌ Redis connection failed: {e}\")\n",
    "    print(\"   Please ensure Redis is running and accessible\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Assess Your Current Index\n",
    "\n",
    "For this demonstration, we'll create a sample FLAT index. In practice, you would analyze your existing index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:22:34.601711Z",
     "start_time": "2025-12-09T15:22:34.597914Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📥 Loading sample movie data...\n",
      "Loaded 20 movie records\n",
      "Sample movie: Explosive Pursuit - A daring cop chases a notorious criminal across the city in a high-stakes game of cat and mouse.\n"
     ]
    }
   ],
   "source": [
    "# Download sample data from redis-ai-resources\n",
    "print(\"📥 Loading sample movie data...\")\n",
    "import os\n",
    "import json\n",
    "\n",
    "# Load the movies dataset\n",
    "url = \"resources/movies.json\"\n",
    "with open(\"resources/movies.json\", \"r\") as f:\n",
    "    movies_data = json.load(f)\n",
    "\n",
    "print(f\"Loaded {len(movies_data)} movie records\")\n",
    "print(f\"Sample movie: {movies_data[0]['title']} - {movies_data[0]['description']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:22:53.897543Z",
     "start_time": "2025-12-09T15:22:53.894843Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📊 Migration Assessment\n",
      "Vector dimensions: 768\n",
      "Dataset size: 20 movie documents\n",
      "Data includes: title, genre, rating, description\n",
      "Vectorizer: RedisVL HFTextVectorizer\n"
     ]
    }
   ],
   "source": [
    "# Configuration for demonstration  \n",
    "dims = 768  # Using all-mpnet-base-v2 model (768 dimensions)\n",
    "\n",
    "num_docs = len(movies_data)  # Use actual dataset size\n",
    "\n",
    "print(\n",
    "    \"📊 Migration Assessment\",\n",
    "    f\"Vector dimensions: {dims}\",\n",
    "    f\"Dataset size: {num_docs} movie documents\",\n",
    "    \"Data includes: title, genre, rating, description\",\n",
    "    f\"Vectorizer: RedisVL HFTextVectorizer\",\n",
    "    sep=\"\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Next, let's configure a smaple FLAT index. Notice the algorithm value, dims value, and datatype value under fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:00.480896Z",
     "start_time": "2025-12-09T15:23:00.471064Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating sample FLAT index...\n",
      "✅ Created FLAT index: migration_demo_flat\n"
     ]
    }
   ],
   "source": [
    "flat_schema = {\n",
    "    \"index\": {\n",
    "        \"name\": \"migration_demo_flat\",\n",
    "        \"prefix\": \"demo:flat:\",\n",
    "    },\n",
    "    \"fields\": [\n",
    "        {\"name\": \"movie_id\", \"type\": \"tag\"},\n",
    "        {\"name\": \"title\", \"type\": \"text\"},\n",
    "        {\"name\": \"genre\", \"type\": \"tag\"},\n",
    "        {\"name\": \"rating\", \"type\": \"numeric\"},\n",
    "        {\"name\": \"description\", \"type\": \"text\"},\n",
    "        {\n",
    "            \"name\": \"embedding\",\n",
    "            \"type\": \"vector\",\n",
    "            \"attrs\": {\n",
    "                \"dims\": dims,\n",
    "                \"algorithm\": \"flat\",\n",
    "                \"datatype\": \"float32\",\n",
    "                \"distance_metric\": \"cosine\"\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Create and populate FLAT index\n",
    "print(\"Creating sample FLAT index...\")\n",
    "flat_index = SearchIndex.from_dict(flat_schema, redis_url=REDIS_URL)\n",
    "flat_index.create(overwrite=True)\n",
    "print(f\"✅ Created FLAT index: {flat_index.name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Generate embeddings for movie descriptions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:10.277500Z",
     "start_time": "2025-12-09T15:23:01.825003Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🔄 Generating embeddings for movie descriptions...\n",
      "🚀 Using RedisVL HFTextVectorizer...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/vishal.bala/PycharmProjects/redis-ai-resources/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16:23:06 sentence_transformers.SentenceTransformer INFO   Use pytorch device_name: mps\n",
      "16:23:06 sentence_transformers.SentenceTransformer INFO   Load pretrained SentenceTransformer: sentence-transformers/all-mpnet-base-v2\n",
      "✅ Generated 20 real embeddings using RedisVL HFTextVectorizer\n"
     ]
    }
   ],
   "source": [
    "# Generate embeddings using RedisVL vectorizers\n",
    "print(\"🔄 Generating embeddings for movie descriptions...\")\n",
    "embedding_model=\"sentence-transformers/all-mpnet-base-v2\"\n",
    "descriptions = [movie['description'] for movie in movies_data]\n",
    "\n",
    "# Use RedisVL HFTextVectorizer\n",
    "print(\"🚀 Using RedisVL HFTextVectorizer...\")\n",
    "vectorizer = HFTextVectorizer(\n",
    "    model=embedding_model  # 768 dimensions\n",
    ")\n",
    "\n",
    "# Generate embeddings using RedisVL vectorizer\n",
    "embeddings = vectorizer.embed_many(descriptions)\n",
    "embeddings = np.array(embeddings, dtype=np.float32)\n",
    "\n",
    "print(f\"✅ Generated {len(embeddings)} real embeddings using RedisVL HFTextVectorizer\")\n",
    "\n",
    "# Prepare data for loading\n",
    "sample_data = []\n",
    "for i, movie in enumerate(movies_data):\n",
    "    sample_data.append({\n",
    "        'movie_id': str(movie['id']),\n",
    "        'title': movie['title'],\n",
    "        'genre': movie['genre'],\n",
    "        'rating': movie['rating'],\n",
    "        'description': movie['description'],\n",
    "        'embedding': array_to_buffer(embeddings[i].astype(np.float32), dtype='float32')\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:14.840375Z",
     "start_time": "2025-12-09T15:23:11.817076Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📥 Loading data into FLAT index...\n",
      "  Loaded 20/20 documents\n",
      "Waiting for indexing to complete...\n",
      "\n",
      "✅ FLAT index loaded with 20 documents\n",
      "Index size: 3.0168838500976563 MB\n"
     ]
    }
   ],
   "source": [
    "# Load data into FLAT index\n",
    "print(\"📥 Loading data into FLAT index...\")\n",
    "batch_size = 100  # Process in batches\n",
    "\n",
    "for i in range(0, len(sample_data), batch_size):\n",
    "    batch = sample_data[i:i+batch_size]\n",
    "    flat_index.load(batch)\n",
    "    print(f\"  Loaded {min(i+batch_size, len(sample_data))}/{len(sample_data)} documents\")\n",
    "\n",
    "# Wait for indexing to complete\n",
    "print(\"Waiting for indexing to complete...\")\n",
    "time.sleep(3)\n",
    "\n",
    "flat_info = flat_index.info()\n",
    "print(f\"\\n✅ FLAT index loaded with {flat_info['num_docs']} documents\")\n",
    "print(f\"Index size: {flat_info.get('vector_index_sz_mb', 'N/A')} MB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Get Compression Recommendation\n",
    "\n",
    "The CompressionAdvisor analyzes your vector dimensions and provides optimal compression settings for SVS-VAMANA vector indices. It eliminates the guesswork from parameter tuning by providing intelligent recommendations based on your vector characteristics and performance priorities.\n",
    "\n",
    "## Configuration Strategy\n",
    "**High-Dimensional Vectors (≥1024 dims)**: Uses **LeanVec4x8** compression with dimensionality reduction. Memory priority reduces dimensions by 50%, speed priority by\n",
    "25%, balanced by 50%. Achieves 60-80% memory savings.\n",
    "\n",
    "**Lower-Dimensional Vectors (<1024 dims)**: Uses **LVQ compression** without dimensionality reduction. Memory priority uses LVQ4 (4 bits), speed uses LVQ4x8 (12 bits),\n",
    "balanced uses LVQ4x4 (8 bits). Achieves 60-87% memory savings.\n",
    "\n",
    "**Our Configuration (768 dims)**: Will use **LVQ compression** as we're below the 1024 dimension threshold. This provides excellent compression without dimensionality reduction.\n",
    "\n",
    "## Available Compression Types\n",
    "- **LVQ4/LVQ4x4/LVQ4x8**: 4/8/12 bits per dimension\n",
    "- **LeanVec4x8/LeanVec8x8**: 12/16 bits + dimensionality reduction for high-dim vectors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:17.484764Z",
     "start_time": "2025-12-09T15:23:17.481838Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🔍 Analyzing compression options...\n",
      "\n",
      "MEMORY priority:\n",
      "  Algorithm: svs-vamana\n",
      "  Compression: LVQ4\n",
      "  Datatype: float32\n",
      "\n",
      "BALANCED priority:\n",
      "  Algorithm: svs-vamana\n",
      "  Compression: LVQ4x4\n",
      "  Datatype: float32\n",
      "\n",
      "PERFORMANCE priority:\n",
      "  Algorithm: svs-vamana\n",
      "  Compression: LVQ4x4\n",
      "  Datatype: float32\n",
      "\n",
      "📋 Selected configuration: LVQ4 with float32\n",
      "Expected memory savings: Significant for 768-dimensional vectors\n"
     ]
    }
   ],
   "source": [
    "# Get compression recommendation\n",
    "print(\"🔍 Analyzing compression options...\")\n",
    "print()\n",
    "\n",
    "# Try different priorities to show options\n",
    "priorities = [\"memory\", \"balanced\", \"performance\"]\n",
    "configs = {}\n",
    "\n",
    "for priority in priorities:\n",
    "    config = CompressionAdvisor.recommend(dims=dims, priority=priority)\n",
    "    configs[priority] = config\n",
    "    print(f\"{priority.upper()} priority:\")\n",
    "    print(f\"  Algorithm: {config.algorithm}\")\n",
    "    print(f\"  Compression: {config.compression if hasattr(config, 'compression') else 'None'}\")\n",
    "    print(f\"  Datatype: {config.datatype}\")\n",
    "    if hasattr(config, 'reduce') and config.reduce:\n",
    "        reduction = ((dims - config.reduce) / dims) * 100\n",
    "        print(f\"  Dimensionality: {dims} → {config.reduce} ({reduction:.1f}% reduction)\")\n",
    "    print()\n",
    "\n",
    "# Select memory-optimized configuration for migration\n",
    "selected_config = configs[\"memory\"]\n",
    "print(f\"📋 Selected configuration: {selected_config.compression if hasattr(selected_config, 'compression') else 'None'} with {selected_config.datatype}\")\n",
    "print(f\"Expected memory savings: Significant for {dims}-dimensional vectors\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Create SVS-VAMANA Index\n",
    "\n",
    "Now we'll create the new SVS-VAMANA index with the recommended compression settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:19.618376Z",
     "start_time": "2025-12-09T15:23:19.608513Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating SVS-VAMANA index with compression...\n",
      "✅ Created SVS-VAMANA index: migration_demo_svs\n",
      "Compression: LVQ4\n",
      "Datatype: float32\n"
     ]
    }
   ],
   "source": [
    "# Fallback configuration if not defined (for CI/CD compatibility)\n",
    "if 'selected_config' not in locals():\n",
    "    from redisvl.utils import CompressionAdvisor\n",
    "    selected_config = CompressionAdvisor.recommend(dims=dims, priority=\"memory\")\n",
    "\n",
    "# Create SVS-VAMANA schema with compression\n",
    "svs_schema = {\n",
    "    \"index\": {\n",
    "        \"name\": \"migration_demo_svs\",\n",
    "        \"prefix\": \"demo:svs:\",\n",
    "    },\n",
    "    \"fields\": [\n",
    "        {\"name\": \"movie_id\", \"type\": \"tag\"},\n",
    "        {\"name\": \"title\", \"type\": \"text\"},\n",
    "        {\"name\": \"genre\", \"type\": \"tag\"},\n",
    "        {\"name\": \"rating\", \"type\": \"numeric\"},\n",
    "        {\"name\": \"description\", \"type\": \"text\"},\n",
    "        {\n",
    "            \"name\": \"embedding\",\n",
    "            \"type\": \"vector\",\n",
    "            \"attrs\": {\n",
    "                \"dims\": selected_config.reduce if (hasattr(selected_config, 'reduce') and selected_config.reduce is not None) else dims,\n",
    "                \"algorithm\": \"svs-vamana\",\n",
    "                \"datatype\": selected_config.datatype,\n",
    "                \"distance_metric\": \"cosine\"\n",
    "                # Note: Don't include the full selected_config to avoid dims/reduce conflict\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "print(\"Creating SVS-VAMANA index with compression...\")\n",
    "svs_index = SearchIndex.from_dict(svs_schema, redis_url=REDIS_URL)\n",
    "svs_index.create(overwrite=True)\n",
    "print(f\"✅ Created SVS-VAMANA index: {svs_index.name}\")\n",
    "print(f\"Compression: {selected_config.compression if hasattr(selected_config, 'compression') else 'None'}\")\n",
    "print(f\"Datatype: {selected_config.datatype}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Migrate Data\n",
    "\n",
    "Extract data from the original index and load it into the SVS-VAMANA index with compression applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:21.292984Z",
     "start_time": "2025-12-09T15:23:21.290954Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🔄 Migrating data to SVS-VAMANA...\n",
      "Target dimensions: 768 (from 768)\n",
      "Target datatype: float32\n"
     ]
    }
   ],
   "source": [
    "print(\"🔄 Migrating data to SVS-VAMANA...\")\n",
    "\n",
    "# Fallback configuration if not defined (for CI/CD compatibility)\n",
    "if 'selected_config' not in locals():\n",
    "    from redisvl.utils import CompressionAdvisor\n",
    "    selected_config = CompressionAdvisor.recommend(dims=dims, priority=\"memory\")\n",
    "\n",
    "# Determine target vector dimensions (may be reduced by LeanVec)\n",
    "# Use reduce if it exists and is not None, otherwise use original dims\n",
    "target_dims = selected_config.reduce if (hasattr(selected_config, 'reduce') and selected_config.reduce is not None) else dims\n",
    "target_dtype = selected_config.datatype\n",
    "\n",
    "print(f\"Target dimensions: {target_dims} (from {dims})\")\n",
    "print(f\"Target datatype: {target_dtype}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:22.893857Z",
     "start_time": "2025-12-09T15:23:22.878963Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data from original index...\n",
      "Found 20 documents to migrate\n",
      "Prepared 20 documents for migration\n"
     ]
    }
   ],
   "source": [
    "# Extract data from FLAT index\n",
    "print(\"Extracting data from original index...\")\n",
    "keys = client.keys(\"demo:flat:*\")\n",
    "print(f\"Found {len(keys)} documents to migrate\")\n",
    "\n",
    "# Process and transform data for SVS index\n",
    "svs_data = []\n",
    "for i, key in enumerate(keys):\n",
    "    doc_data = client.hgetall(key)\n",
    "    \n",
    "    if b'embedding' in doc_data:\n",
    "        # Extract original vector\n",
    "        original_vector = np.array(buffer_to_array(doc_data[b'embedding'], dtype='float32'))\n",
    "        \n",
    "        # Apply dimensionality reduction if needed (LeanVec)\n",
    "        if target_dims < dims:\n",
    "            vector = original_vector[:target_dims]\n",
    "        else:\n",
    "            vector = original_vector\n",
    "        \n",
    "        # Convert to target datatype\n",
    "        if target_dtype == 'float16':\n",
    "            vector = vector.astype(np.float16)\n",
    "        \n",
    "        svs_data.append({\n",
    "            \"movie_id\": doc_data[b'movie_id'].decode(),\n",
    "            \"title\": doc_data[b'title'].decode(),\n",
    "            \"genre\": doc_data[b'genre'].decode(),\n",
    "            \"rating\": int(doc_data[b'rating'].decode()),\n",
    "            \"description\": doc_data[b'description'].decode(),\n",
    "            \"embedding\": array_to_buffer(vector, dtype=target_dtype)\n",
    "        })\n",
    "    \n",
    "    if (i + 1) % 500 == 0:\n",
    "        print(f\"  Processed {i + 1}/{len(keys)} documents\")\n",
    "\n",
    "print(f\"Prepared {len(svs_data)} documents for migration\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:28.514695Z",
     "start_time": "2025-12-09T15:23:23.494711Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data into SVS-VAMANA index...\n",
      "  Migrated 20/20 documents\n",
      "Waiting for indexing to complete...\n",
      "\n",
      "✅ Migration complete! SVS index has 20 documents\n"
     ]
    }
   ],
   "source": [
    "# Load data into SVS index\n",
    "print(\"Loading data into SVS-VAMANA index...\")\n",
    "batch_size = 100  # Define batch size for migration\n",
    "\n",
    "if len(svs_data) > 0:\n",
    "    for i in range(0, len(svs_data), batch_size):\n",
    "        batch = svs_data[i:i+batch_size]\n",
    "        svs_index.load(batch)\n",
    "        print(f\"  Migrated {min(i+batch_size, len(svs_data))}/{len(svs_data)} documents\")\n",
    "\n",
    "    # Wait for indexing to complete\n",
    "    print(\"Waiting for indexing to complete...\")\n",
    "    time.sleep(5)\n",
    "\n",
    "    svs_info = svs_index.info()\n",
    "    print(f\"\\n✅ Migration complete! SVS index has {svs_info['num_docs']} documents\")\n",
    "else:\n",
    "    print(\"⚠️  No data to migrate. Make sure the FLAT index was populated first.\")\n",
    "    print(\"   Run the previous cells to load data into the FLAT index.\")\n",
    "    svs_info = svs_index.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Compare Memory Usage\n",
    "\n",
    "Let's analyze the memory savings achieved through compression. This is just an example on the small sample data. Use a larger dataset before deciding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:28.554508Z",
     "start_time": "2025-12-09T15:23:28.551629Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📊 Memory Usage Comparison\n",
      "========================================\n",
      "Original FLAT index:    3.02 MB\n",
      "SVS-VAMANA index:       3.02 MB\n",
      "\n",
      "💰 Memory savings: -0.0%\n",
      "Absolute reduction: -0.00 MB\n"
     ]
    }
   ],
   "source": [
    "# Helper function to extract memory info\n",
    "def get_memory_mb(index_info):\n",
    "    \"\"\"Extract memory usage in MB from index info\"\"\"\n",
    "    memory = index_info.get('vector_index_sz_mb', 0)\n",
    "    if isinstance(memory, str):\n",
    "        try:\n",
    "            return float(memory)\n",
    "        except ValueError:\n",
    "            return 0.0\n",
    "    return float(memory)\n",
    "\n",
    "# Get memory usage\n",
    "flat_memory = get_memory_mb(flat_info)\n",
    "svs_memory = get_memory_mb(svs_info)\n",
    "\n",
    "print(\n",
    "    \"📊 Memory Usage Comparison\",\n",
    "    \"=\" * 40,\n",
    "    f\"Original FLAT index:    {flat_memory:.2f} MB\",\n",
    "    f\"SVS-VAMANA index:       {svs_memory:.2f} MB\",\n",
    "    \"\",\n",
    "    sep=\"\\n\"\n",
    ")\n",
    "\n",
    "if flat_memory > 0:\n",
    "    if svs_memory > 0:\n",
    "        savings = ((flat_memory - svs_memory) / flat_memory) * 100\n",
    "        print(\n",
    "            f\"💰 Memory savings: {savings:.1f}%\",\n",
    "            f\"Absolute reduction: {flat_memory - svs_memory:.2f} MB\",\n",
    "            sep=\"\\n\"\n",
    "        )\n",
    "    else:\n",
    "        print(\"⏳ SVS index still indexing - memory comparison pending\")\n",
    "else:\n",
    "    print(\"⚠️  Memory information not available\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Validate Search Quality\n",
    "\n",
    "Compare search quality and performance between FLAT and SVS-VAMANA indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:28.585499Z",
     "start_time": "2025-12-09T15:23:28.564735Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📊 Search Quality Comparison\n",
      "========================================\n",
      "FLAT (baseline):        100% recall (brute-force exact search)\n",
      "SVS-VAMANA Recall@5:    100.0% (vs FLAT baseline)\n",
      "SVS-VAMANA Recall@10:   100.0% (vs FLAT baseline)\n",
      "\n",
      "⏱️ Performance Comparison:\n",
      "FLAT query time:        0.009s (0.9ms per query)\n",
      "SVS-VAMANA query time:  0.007s (0.7ms per query)\n",
      "Speed difference:       +21.1%\n",
      "\n",
      "🎯 Quality Assessment: 🟢 Excellent - Minimal quality loss\n"
     ]
    }
   ],
   "source": [
    "# Calculate recall and performance metrics\n",
    "def calculate_recall(reference_results, test_results, k=10):\n",
    "    \"\"\"Calculate recall@k between two result sets\"\"\"\n",
    "    if not reference_results or not test_results:\n",
    "        return 0.0\n",
    "    \n",
    "    ref_ids = set([doc['movie_id'] for doc in reference_results[:k]])\n",
    "    test_ids = set([doc['movie_id'] for doc in test_results[:k]])\n",
    "    \n",
    "    if not ref_ids:\n",
    "        return 0.0\n",
    "    \n",
    "    return len(ref_ids.intersection(test_ids)) / len(ref_ids)\n",
    "\n",
    "# Create test queries\n",
    "num_test_queries = 10\n",
    "test_queries = []\n",
    "\n",
    "for i in range(num_test_queries):\n",
    "    query_vec = np.random.random(dims).astype(np.float32)\n",
    "    query_vec = query_vec / np.linalg.norm(query_vec)\n",
    "    test_queries.append(query_vec)\n",
    "\n",
    "# Test FLAT index (ground truth)\n",
    "flat_results_list = []\n",
    "flat_start = time.time()\n",
    "\n",
    "for query_vec in test_queries:\n",
    "    query = VectorQuery(\n",
    "        vector=query_vec,\n",
    "        vector_field_name=\"embedding\",\n",
    "        return_fields=[\"movie_id\", \"title\", \"genre\"],\n",
    "        dtype=\"float32\",\n",
    "        num_results=10\n",
    "    )\n",
    "    results = flat_index.query(query)\n",
    "    flat_results_list.append(results)\n",
    "\n",
    "flat_time = time.time() - flat_start\n",
    "\n",
    "# Test SVS-VAMANA index\n",
    "svs_results_list = []\n",
    "svs_start = time.time()\n",
    "\n",
    "for query_vec in test_queries:\n",
    "    # Adjust query vector for SVS index (handle dimensionality reduction)\n",
    "    if target_dims < dims:\n",
    "        svs_query_vec = query_vec[:target_dims]\n",
    "    else:\n",
    "        svs_query_vec = query_vec\n",
    "    \n",
    "    if target_dtype == 'float16':\n",
    "        svs_query_vec = svs_query_vec.astype(np.float16)\n",
    "    \n",
    "    query = VectorQuery(\n",
    "        vector=svs_query_vec,\n",
    "        vector_field_name=\"embedding\",\n",
    "        return_fields=[\"movie_id\", \"title\", \"genre\"],\n",
    "        dtype=target_dtype,\n",
    "        num_results=10\n",
    "    )\n",
    "    results = svs_index.query(query)\n",
    "    svs_results_list.append(results)\n",
    "\n",
    "svs_time = time.time() - svs_start\n",
    "\n",
    "# Calculate recall metrics\n",
    "recall_at_5 = np.mean([calculate_recall(flat_res, svs_res, k=5) \n",
    "                       for flat_res, svs_res in zip(flat_results_list, svs_results_list)])\n",
    "recall_at_10 = np.mean([calculate_recall(flat_res, svs_res, k=10) \n",
    "                        for flat_res, svs_res in zip(flat_results_list, svs_results_list)])\n",
    "\n",
    "print(\n",
    "    \"📊 Search Quality Comparison\",\n",
    "    \"=\" * 40,\n",
    "    \"FLAT (baseline):        100% recall (brute-force exact search)\",\n",
    "    f\"SVS-VAMANA Recall@5:    {recall_at_5*100:.1f}% (vs FLAT baseline)\",\n",
    "    f\"SVS-VAMANA Recall@10:   {recall_at_10*100:.1f}% (vs FLAT baseline)\",\n",
    "    \"\",\n",
    "    \"⏱️ Performance Comparison:\",\n",
    "    f\"FLAT query time:        {flat_time:.3f}s ({flat_time/num_test_queries*1000:.1f}ms per query)\",\n",
    "    f\"SVS-VAMANA query time:  {svs_time:.3f}s ({svs_time/num_test_queries*1000:.1f}ms per query)\",\n",
    "    f\"Speed difference:       {((flat_time - svs_time) / flat_time * 100):+.1f}%\",\n",
    "    sep=\"\\n\"\n",
    ")\n",
    "\n",
    "# Quality assessment\n",
    "if recall_at_10 >= 0.95:\n",
    "    quality_assessment = \"🟢 Excellent - Minimal quality loss\"\n",
    "elif recall_at_10 >= 0.90:\n",
    "    quality_assessment = \"🟡 Good - Acceptable quality for most applications\"\n",
    "elif recall_at_10 >= 0.80:\n",
    "    quality_assessment = \"🟠 Fair - Consider if quality requirements are flexible\"\n",
    "else:\n",
    "    quality_assessment = \"🔴 Poor - Migration not recommended\"\n",
    "\n",
    "print(f\"\\n🎯 Quality Assessment: {quality_assessment}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Migration Decision Framework\n",
    "\n",
    "Analyze the migration results and provide a recommendation based on memory savings and search quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:32.801584Z",
     "start_time": "2025-12-09T15:23:32.798152Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🤔 Migration Decision Analysis\n",
      "========================================\n",
      "\n",
      "📊 Criteria Evaluation:\n",
      "Memory savings: -0.0% ❌ (threshold: 5%)\n",
      "Search quality: 1.000 ✅ (threshold: 0.85)\n",
      "\n",
      "🎯 Migration Recommendation: 🟠 LIMITED BENEFIT\n",
      "💭 Reasoning: Search quality is maintained but memory savings are minimal. Migration may not be worth the effort.\n"
     ]
    }
   ],
   "source": [
    "# Migration decision logic\n",
    "memory_savings_threshold = 5  # Minimum % memory savings\n",
    "recall_threshold = 0.85  # Minimum 85% recall@10\n",
    "\n",
    "memory_savings_pct = ((flat_memory - svs_memory) / flat_memory * 100) if flat_memory > 0 and svs_memory > 0 else 0\n",
    "meets_memory_threshold = memory_savings_pct >= memory_savings_threshold\n",
    "meets_quality_threshold = recall_at_10 >= recall_threshold\n",
    "\n",
    "print(\n",
    "    \"🤔 Migration Decision Analysis\",\n",
    "    \"=\" * 40,\n",
    "    \"\",\n",
    "    \"📊 Criteria Evaluation:\",\n",
    "    f\"Memory savings: {memory_savings_pct:.1f}% {'✅' if meets_memory_threshold else '❌'} (threshold: {memory_savings_threshold}%)\",\n",
    "    f\"Search quality: {recall_at_10:.3f} {'✅' if meets_quality_threshold else '❌'} (threshold: {recall_threshold})\",\n",
    "    \"\",\n",
    "    sep=\"\\n\"\n",
    ")\n",
    "\n",
    "if meets_memory_threshold and meets_quality_threshold:\n",
    "    recommendation = \"🟢 RECOMMENDED\"\n",
    "    reasoning = \"Migration provides significant memory savings while maintaining good search quality.\"\n",
    "elif meets_memory_threshold and not meets_quality_threshold:\n",
    "    recommendation = \"🟡 CONDITIONAL\"\n",
    "    reasoning = \"Good memory savings but reduced search quality. Consider if your application can tolerate lower recall.\"\n",
    "elif not meets_memory_threshold and meets_quality_threshold:\n",
    "    recommendation = \"🟠 LIMITED BENEFIT\"\n",
    "    reasoning = \"Search quality is maintained but memory savings are minimal. Migration may not be worth the effort.\"\n",
    "else:\n",
    "    recommendation = \"🔴 NOT RECOMMENDED\"\n",
    "    reasoning = \"Insufficient memory savings and/or poor search quality. Consider alternative optimization strategies.\"\n",
    "\n",
    "print(\n",
    "    f\"🎯 Migration Recommendation: {recommendation}\",\n",
    "    f\"💭 Reasoning: {reasoning}\",\n",
    "    sep=\"\\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Cleanup\n",
    "\n",
    "Clean up the demonstration indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-12-09T15:23:38.291Z",
     "start_time": "2025-12-09T15:23:38.285026Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧹 Cleaning up demonstration indices...\n",
      "✅ Deleted FLAT demonstration index\n",
      "✅ Deleted SVS-VAMANA demonstration index\n",
      "\n",
      "🎉 Migration demonstration complete!\n",
      "\n",
      "Next steps:\n",
      "1. Apply learnings to your production data\n",
      "2. Test with your actual query patterns\n",
      "3. Monitor performance in your environment\n",
      "4. Consider gradual rollout strategy\n"
     ]
    }
   ],
   "source": [
    "print(\"🧹 Cleaning up demonstration indices...\")\n",
    "\n",
    "# Clean up FLAT index\n",
    "try:\n",
    "    flat_index.delete(drop=True)\n",
    "    print(\"✅ Deleted FLAT demonstration index\")\n",
    "except Exception as e:\n",
    "    print(f\"⚠️  Failed to delete FLAT index: {e}\")\n",
    "\n",
    "# Clean up SVS index\n",
    "try:\n",
    "    svs_index.delete(drop=True)\n",
    "    print(\"✅ Deleted SVS-VAMANA demonstration index\")\n",
    "except Exception as e:\n",
    "    print(f\"⚠️  Failed to delete SVS index: {e}\")\n",
    "\n",
    "print(\n",
    "    \"\\n🎉 Migration demonstration complete!\",\n",
    "    \"\\nNext steps:\",\n",
    "    \"1. Apply learnings to your production data\",\n",
    "    \"2. Test with your actual query patterns\",\n",
    "    \"3. Monitor performance in your environment\",\n",
    "    \"4. Consider gradual rollout strategy\",\n",
    "    sep=\"\\n\"\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}