{ "cells": [ { "cell_type": "markdown", "id": "cbba56a9", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "# Routing Optimization\n", "\n", "Implementing a semantic router is a great light weight way to add branching logic to your application without taking on additional LLM calls. However, it can be tough to determine the optimal distance threshold values for your routes to maximize performance. This guide will walk through:\n", "\n", "- how to configure a semantic router\n", "- how to optimize the distance thresholds for the routes\n", "- a comparison between performing similar logic with an LLM versus a router\n", "\n", "## Let's Begin!\n", "\"Open\n" ] }, { "cell_type": "markdown", "id": "19bdc2a5-2192-4f5f-bd6e-7c956fd0e230", "metadata": {}, "source": [ "# Setup\n", "\n", "## Install Packages" ] }, { "cell_type": "code", "execution_count": null, "id": "c620286e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n", "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -q sentence-transformers ranx \"redisvl>=0.6.0\" \"redis-retrieval-optimizer>=0.4.2\"" ] }, { "cell_type": "markdown", "id": "c1250544", "metadata": {}, "source": [ "### Grab data (if colab)" ] }, { "cell_type": "code", "execution_count": null, "id": "76c1f678", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n", "!mv temp_repo/python-recipes/semantic-router/resources .\n", "!rm -rf temp_repo" ] }, { "cell_type": "markdown", "id": "323aec7f", "metadata": {}, "source": [ "## Run a Redis instance\n", "\n", "#### For Colab\n", "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." ] }, { "cell_type": "code", "execution_count": null, "id": "2cb85a99", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "%%sh\n", "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", "sudo apt-get update > /dev/null 2>&1\n", "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", "redis-stack-server --daemonize yes" ] }, { "cell_type": "markdown", "id": "7c5dbaaf", "metadata": {}, "source": [ "#### For Alternative Environments\n", "There are many ways to get the necessary redis-stack instance running\n", "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", "own version of Redis Enterprise running, that works too!\n", "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" ] }, { "cell_type": "markdown", "id": "1d4499ae", "metadata": {}, "source": [ "### Define the Redis Connection URL\n", "\n", "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." ] }, { "cell_type": "code", "execution_count": 3, "id": "aefda1d1", "metadata": {}, "outputs": [], "source": [ "import os\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "# Replace values below with your own if using Redis Cloud instance\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", "\n", "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" ] }, { "cell_type": "markdown", "id": "10f4cb85", "metadata": {}, "source": [ "# Routing with multiple routes\n", "\n", "## Define the Routes\n", "\n", "Below we define 3 different routes. One for `faq` (frequently asked questions), one for `general`, and\n", "another for `blocked`. Now for this example, the goal here is\n", "surely topic \"classification\". But you can create routes and references for\n", "almost anything.\n", "\n", "Each route has a set of references that cover the \"semantic surface area\" of the\n", "route. The incoming query from a user needs to be semantically similar to one or\n", "more of the references in order to \"match\" on the route. Note that each route can have it's own distinct `distance_threshold` that defines what is considered a match for the particular query. " ] }, { "cell_type": "code", "execution_count": 4, "id": "60ad280c", "metadata": {}, "outputs": [], "source": [ "from redisvl.extensions.router import Route\n", "\n", "faq = Route(\n", " name=\"faq\",\n", " references=[\n", " \"How do I reset my password?\",\n", " \"Where can I view my order history?\",\n", " \"How do I update my shipping address?\",\n", " \"Where are my saved payment methods?\",\n", " \"How do I change my email preferences?\",\n", " \"How can I see my loyalty points balance?\",\n", " \"Where do I find my digital receipts?\",\n", " \"How do I enable two-factor authentication?\",\n", " \"Can I change my username or email?\",\n", " \"How do I manage my account settings?\"\n", " ],\n", " metadata={\"category\": \"account_management\", \"priority\": 1},\n", " distance_threshold=0.5\n", ")\n", "\n", "general = Route(\n", " name=\"general\",\n", " references=[\n", " \"I received the wrong item in my order, can you help?\",\n", " \"Can you recommend products that match my specific needs?\",\n", " \"The assembly instructions for my furniture are unclear\",\n", " \"I need help finding a product with particular specifications\",\n", " \"My order arrived damaged, what are my options?\",\n", " \"Can you help me design a room with your products?\",\n", " \"I'm looking for custom sizing options for this product\",\n", " \"The item I received doesn't match the online description\",\n", " \"I need advice on which model would work best for my situation\",\n", " \"Can you help troubleshoot an issue with my recent purchase?\"\n", " ],\n", " metadata={\"category\": \"customer_service\", \"priority\": 2},\n", " distance_threshold=0.5\n", ")\n", "\n", "blocked = Route(\n", " name=\"blocked\",\n", " references=[\n", " \"What is your company's stance on the recent election?\",\n", " \"Do you support liberal or conservative policies?\",\n", " \"Can you tell me another customer's address?\",\n", " \"What's your CEO's opinion on gun control?\",\n", " \"I need personal information about one of your employees\",\n", " \"How does your company vote on political issues?\",\n", " \"Can you provide me with someone's credit card details?\",\n", " \"What's your position on immigration reform?\",\n", " \"I want to know where a specific customer lives\",\n", " \"Does your company donate to political campaigns?\"\n", " ],\n", " metadata={\"category\": \"prohibited\", \"priority\": 3},\n", " distance_threshold=0.5\n", ")" ] }, { "cell_type": "markdown", "id": "9cdbcbff", "metadata": {}, "source": [ "## Initialize the SemanticRouter\n", "\n", "Like before the ``SemanticRouter`` class will automatically create an index within Redis upon initialization for the route references." ] }, { "cell_type": "code", "execution_count": 5, "id": "e80aaf84", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "13:22:06 datasets INFO PyTorch version 2.3.0 available.\n", "13:22:06 sentence_transformers.SentenceTransformer INFO Use pytorch device_name: mps\n", "13:22:06 sentence_transformers.SentenceTransformer INFO Load pretrained SentenceTransformer: sentence-transformers/all-mpnet-base-v2\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6345d6b8899347ec9c3eac71442f2bd1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:00 used claude sonnet 3.7 for generation of resource\n", "\n", "```txt\n", "You are a test data creation helper. \n", "\n", "Create test data of the form:\n", "\n", "{\n", " \"query\": \"query about a topic\",\n", " \"query_match\": \"topic-the-query-matches\"\n", "}\n", "\n", "The 3 available topics are: faq, general, and blocked. Generate many examples that map to these topics such that we can train a model to find the best thresholds for this classification task. Also make sure to include some examples that don't map to any of the topics to check the null case for these leave the query_match field empty.\n", "```\n", "\n", "The output of this call was saved to `./resources/test_data.json`" ] }, { "cell_type": "code", "execution_count": 8, "id": "3c03a117", "metadata": {}, "outputs": [], "source": [ "import json\n", "\n", "with open(\"resources/ecom_train_data.json\", \"r\") as f:\n", " train_data = json.load(f)" ] }, { "cell_type": "markdown", "id": "1d0c5c2a", "metadata": {}, "source": [ "## Run optimization with router\n", "\n", "Using the `RouterThresholdOptimizer` from the `redis-retrieval-optimizer` library." ] }, { "cell_type": "code", "execution_count": 9, "id": "83d2a15c", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a7825e73ad0647f0a84d5f7f4db318e1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:00 str:\n", " prompt = f\"\"\"\n", " You are a classification bot. Your job is to classify the following query as either faq, general, blocked, or none. Return only the string label or an empty string if no match.\n", "\n", " general is defined as request requiring customer service.\n", " faq is defined as a request for commonly asked account questions.\n", " blocked is defined as a request for prohibited information.\n", "\n", " query: \"{question}\"\n", " \"\"\"\n", " response = client.responses.create(\n", " model=\"gpt-4o-mini\",\n", " input=prompt,\n", " )\n", " return response" ] }, { "cell_type": "code", "execution_count": 10, "id": "feb25546", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "13:23:11 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n" ] }, { "data": { "text/plain": [ "'faq'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with open(\"resources/ecom_test_data.json\", \"r\") as f:\n", " test_data = json.load(f)\n", "\n", "\n", "res = ask_openai(test_data[0][\"query\"])\n", "res.output_text" ] }, { "cell_type": "code", "execution_count": 12, "id": "5ee72be1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'input_tokens': 99,\n", " 'input_tokens_details': {'cached_tokens': 0},\n", " 'output_tokens': 2,\n", " 'output_tokens_details': {'reasoning_tokens': 0},\n", " 'total_tokens': 101}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res.usage.model_dump()" ] }, { "cell_type": "code", "execution_count": 13, "id": "e5c921b2", "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "INPUT_TOKEN_PRICE = (0.15 / 1_000_000)\n", "OUTPUT_TOKEN_PRICE = (0.60 / 1_000_000)\n", "\n", "def calc_cost_rough(openai_response):\n", " return openai_response.usage.input_tokens * INPUT_TOKEN_PRICE + openai_response.usage.output_tokens * OUTPUT_TOKEN_PRICE\n", "\n", "def test_classifier(classifier, test_data, is_router=False):\n", " correct = 0\n", " times = []\n", " costs = []\n", "\n", " for data in test_data:\n", " start = time.time()\n", " if is_router:\n", " prediction = classifier(data[\"query\"]).name\n", " else:\n", " openai_response = ask_openai(data[\"query\"])\n", " prediction = openai_response.output_text\n", " costs.append(calc_cost_rough(openai_response))\n", " \n", " if not prediction or prediction.lower() == \"none\":\n", " prediction = \"\"\n", "\n", " times.append(time.time() - start)\n", " print(f\"Expected | Observed: {data['query_match']} | {prediction.lower()}\")\n", " if prediction.lower() == data[\"query_match\"]:\n", " correct += 1\n", "\n", " accuracy = correct / len(test_data)\n", " avg_time = np.mean(times)\n", " cost = np.sum(costs) if costs else 0\n", " return accuracy, avg_time, round(cost, 4)" ] }, { "cell_type": "code", "execution_count": 14, "id": "5c6024e8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "13:23:43 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: faq | faq\n", "13:23:43 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: faq | faq\n", "13:23:44 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: faq | faq\n", "13:23:44 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: faq | faq\n", "13:23:45 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: faq | general\n", "13:23:45 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: faq | faq\n", "13:23:46 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: general | general\n", "13:23:46 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: general | general\n", "13:23:47 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: general | general\n", "13:23:47 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: general | general\n", "13:23:48 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: general | general\n", "13:23:48 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: general | general\n", "13:23:49 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | \n", "13:23:49 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | blocked\n", "13:23:50 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | blocked\n", "13:23:50 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | general\n", "13:23:51 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | blocked\n", "13:23:52 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | blocked\n", "13:23:52 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | \n", "13:23:53 httpx INFO HTTP Request: POST https://api.openai.com/v1/responses \"HTTP/1.1 200 OK\"\n", "Expected | Observed: blocked | blocked\n" ] } ], "source": [ "llm_accuracy, llm_avg_time, llm_cost = test_classifier(ask_openai, test_data)" ] }, { "cell_type": "code", "execution_count": 15, "id": "c3362a1b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(0.8, 0.5609435558319091, 0.0003)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "llm_accuracy, llm_avg_time, llm_cost" ] }, { "cell_type": "code", "execution_count": 16, "id": "40ddc05d", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "65740a8a0b094a68aea0d31fd3c6d87a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Batches: 0%| | 0/1 [00:00