{ "cells": [ { "cell_type": "markdown", "id": "478e8a79", "metadata": {}, "source": [ "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", "\n", "# Advanced Semantic Routing - Improving Routing Performance\n", "In many experiments, we use benchmark datasets to demonstrate the classification capabilities of the semantic router. The semantic router performs well for such well-curated sets because the examples are well aligned and routes are clearly defined.\n", "\n", "However, in practice, route definitions are often hand-authored and only weakly specified. Using human expertise for route definitions upstream might result in poor quality if:\n", "- examples are written from intuition (and not from real users)\n", "- examples are sparse\n", "- route boundaries are fuzzy\n", "- bias is introduced due to human decision (different intent definition between different people)\n", "\n", "In this notebook, we will explore various strategies for improving router performance in some cases where route quality is not guaranteed. These strategies include:\n", "- simply using another embedding model\n", "- using the LLM upstream for augmenting the set of route references\n", "- using the LLM as a fallback classifier when the performance of the semantic router starts to degrade (albeit with some tradeoffs).\n", "\n", "## Before we get started\n", "To understand more about semantic routing, please refer to the following: \n", "- [RedisVL Semantic Routing](https://docs.redisvl.com/en/latest/user_guide/08_semantic_router.html) - A guide on how to use the SemanticRouter from RedisVL\n", "- [Source Code (GitHub)](https://github.com/redis/redis-vl-python#semantic-routing) - Source code for the implementation.\n", "- [Embeddings](https://redis.io/blog/why-vector-embeddings-are-here-to-stay/) - A blog post about embeddings and what they are used for.\n", "\n", "\n", "## Let's Begin!\n", "\"Open" ] }, { "cell_type": "markdown", "id": "3eedeb21", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "markdown", "id": "7a5cf694", "metadata": {}, "source": [ "First, let's download the required packages and set our API keys:" ] }, { "cell_type": "code", "execution_count": null, "id": "cbf41041", "metadata": {}, "outputs": [], "source": [ "%pip install -q \"redisvl[sentence-transformers]>=0.6.0\" datasets openai" ] }, { "cell_type": "markdown", "id": "2b5c9779", "metadata": {}, "source": [ "### Install Redis " ] }, { "cell_type": "markdown", "id": "7d65f06c", "metadata": {}, "source": [ "#### For Colab\n", "Use the shell script below to download, extract, and install Redis Stack directly from the Redis package archive." ] }, { "cell_type": "code", "execution_count": null, "id": "04b9cd71", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "# Colab only: install and start Redis Stack.\n", "%%sh\n", "sudo apt-get install -y -qq lsb-release curl gpg > /dev/null\n", "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", "sudo chmod 644 /usr/share/keyrings/redis-archive-keyring.gpg\n", "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", "sudo apt-get update -qq > /dev/null\n", "sudo apt-get install -y -qq redis > /dev/null\n", "\n", "redis-server --version\n", "redis-server --daemonize yes --loadmodule /usr/lib/redis/modules/redisearch.so" ] }, { "cell_type": "markdown", "id": "e8a7e3d0", "metadata": {}, "source": [ "#### For Alternative Environments\n", "There are many ways to get the necessary redis-stack instance running\n", "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n", "own version of Redis Enterprise running, that works too!\n", "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" ] }, { "cell_type": "markdown", "id": "ceb0af3d", "metadata": {}, "source": [ "### Define the Redis Connection URL\n", "\n", "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." ] }, { "cell_type": "code", "execution_count": 3, "id": "e082ae46", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import os\n", "from redis import Redis\n", "\n", "# Replace values below with your own if using Redis Cloud instance\n", "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", "\n", "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"\n", "\n", "client = Redis.from_url(REDIS_URL)\n", "client.ping()\n", "client.flushall()" ] }, { "cell_type": "markdown", "id": "194cfe65", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 4, "id": "3d9f259e", "metadata": {}, "outputs": [], "source": [ "import asyncio\n", "import json\n", "import getpass\n", "import random\n", "from collections import defaultdict\n", "from pprint import pprint, pformat\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "from openai import AsyncOpenAI, OpenAI\n", "from redisvl.extensions.router import Route, SemanticRouter\n", "from redisvl.utils.vectorize import HFTextVectorizer\n", "\n", "random.seed(123)" ] }, { "cell_type": "markdown", "id": "a50f0f87", "metadata": {}, "source": [ "## API keys" ] }, { "cell_type": "code", "execution_count": 5, "id": "36cafb9e", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "# OpenAI API key is required for LLM calls\n", "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\") or getpass.getpass(\"OpenAI API key: \")\n", "openai_client = AsyncOpenAI(api_key=OPENAI_API_KEY)" ] }, { "cell_type": "markdown", "id": "a238a050", "metadata": {}, "source": [ "## Data setup" ] }, { "cell_type": "markdown", "id": "4e8646ad", "metadata": {}, "source": [ "We get our sample data from the [CLINC dataset](https://huggingface.co/datasets/DeepPavlov/clinc150). \n", "In our setup, we simulate a real-life scenario where we have identified a bunch of routes for intent classification, but we are limited by a lack of high-quality references for each route. \n", "\n", "In this scenario, we attempt to perform intent classification on a simulated banking chatbot, using intents from the CLINC dataset related to `balance`, `transactions`, `transfer`, `pay_bill`, `bill_due`, `card_declined`, `report_fraud`, `freeze_account`, `credit_score`, `direct_deposit`. The dataset represents actual queries that we might expect from a deployed real-life production chatbot." ] }, { "cell_type": "code", "execution_count": 6, "id": "60f9aaeb", "metadata": {}, "outputs": [], "source": [ "from datasets import load_dataset\n", "data = load_dataset(\"clinc_oos\", \"plus\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "9e7e2b94", "metadata": {}, "outputs": [], "source": [ "intents = [\n", " \"balance\",\n", " \"transactions\",\n", " \"transfer\",\n", " \"pay_bill\",\n", " \"bill_due\",\n", " \"card_declined\",\n", " \"report_fraud\",\n", " \"freeze_account\",\n", " \"credit_score\",\n", " \"direct_deposit\",\n", "]" ] }, { "cell_type": "code", "execution_count": 8, "id": "818c0747", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'train': {'rows': 1000,\n", " 'sample_texts': ['i need $20000 transferred from my savings to my checking',\n", " 'complete a transaction from savings to checking of $20000',\n", " 'transfer $20000 from my savings account to checking account'],\n", " 'sample_intents': ['transfer', 'transfer', 'transfer']},\n", " 'validation': {'rows': 200,\n", " 'sample_texts': ['transfer ten dollars from my wells fargo account to my bank of america account',\n", " 'take one hundred and fifty bucks from my wells fargo checking account and put it in my wells fargo savings account',\n", " 'put one hundred and seventy five bucks in my wells fargo checking account from my citibank savings account'],\n", " 'sample_intents': ['transfer', 'transfer', 'transfer']},\n", " 'test': {'rows': 300,\n", " 'sample_texts': ['can you please provide me with assistance in moving money from one account to another',\n", " 'i would like help moving money between accounts',\n", " 'can you assist me in moving money from one account to another'],\n", " 'sample_intents': ['transfer', 'transfer', 'transfer']}}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a subset of data only with the intents we are interested in\n", "intent_names = data['train'].features['intent'].names\n", "intent_ids = [intent_names.index(intent) for intent in intents]\n", "\n", "data_subset = data.filter(lambda example: example['intent'] in intent_ids)\n", "\n", "# View a sample of the data subset\n", "{\n", " split: {\n", " 'rows': len(data_subset[split]),\n", " 'sample_texts': data_subset[split]['text'][:3],\n", " 'sample_intents': [intent_names[i] for i in data_subset[split]['intent'][:3]],\n", " }\n", " for split in data_subset\n", "}" ] }, { "cell_type": "code", "execution_count": 9, "id": "d90e9674", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'balance': ['savings account balance at chase bank please',\n", " 'check chase bank for my checking balance',\n", " 'what is my checking account balance at chase',\n", " 'i want my checking balance at chase',\n", " \"what's my savings balance at chase\",\n", " 'do i have enough money in my chime bank account to take ashley '\n", " 'to the movies tuesday',\n", " 'check my visa account and see if i have enough money for dinner '\n", " 'tonight',\n", " 'is tehre enough in my bluebird account for groceries this week',\n", " 'do i have enough in my sears account to buy a new dishwasher',\n", " 'is there enough money in my discover account for a new pair of '\n", " 'jeans',\n", " 'what is the total of my bank accounts',\n", " 'what is in my bank accounts',\n", " 'what is the balance on my bank accounts',\n", " 'what is remaining in my bank acccounts',\n", " 'how much money is left in my bank accounts',\n", " 'what amount of money is in my bank accounts',\n", " 'what is the balance of my bank accounts',\n", " 'what what kind money is available in my bank accounts',\n", " 'what is the value of my bank accounts',\n", " 'how much funds do i have in my bank accounts',\n", " \"what's my checking account balance\",\n", " 'how much do i have in my checking',\n", " \"what's the balance of my savings\",\n", " 'how much is in savings',\n", " 'how much money is in my checking account',\n", " \"what's the amount of money accumulated in my bank accounts\",\n", " \"what's my current bank savings\",\n", " 'how much total cash do i have in the bank',\n", " \"what's my current checking balance\",\n", " 'can you tell me my current bank accounts balance',\n", " \"what's the balance of my bank accounts\",\n", " 'what is my current balance on my home equity line of credit',\n", " 'please find my balance on my chase mastercard',\n", " 'can you tell me my checking account balance',\n", " 'perform a search for my most recent balance on my amex account',\n", " 'how much is the current balance in my td bank savings account',\n", " 'what is my balance in checking account',\n", " 'how much is available in my savings account',\n", " 'tell me what i have in my money market account',\n", " 'how much money do i have in my account',\n", " 'how much is left of mastercard',\n", " 'what is the available balance in savings',\n", " 'what is the balance on my visa',\n", " 'do i have enough money in my chase account for a new '\n", " 'refrigerator',\n", " 'do i have enough money in my charles schwab account to get a new '\n", " 'baseball bat',\n", " \"i'd like to know my bank balance please\",\n", " \"what's the balance on my bank account\",\n", " 'will the money in my capital one account cover a new washing '\n", " 'machine',\n", " 'could you check my bank balance for me',\n", " 'will the amount in my chase bank account right now cover the '\n", " 'cost of a new dryer',\n", " 'i want to get a new shirt; will the money in my td ameritrade '\n", " 'account cover it',\n", " 'how much do i have in the bank',\n", " \"what's my bank balance\",\n", " 'what is my bank balance',\n", " 'is there any money left',\n", " 'do i have any cash left',\n", " 'can you tell me how much cash i have',\n", " 'do i have enough in my boa account for a new pair of skis',\n", " 'do i have enough in my chase account for new nikes',\n", " 'can the funds in my wells fargo account cover my lift tickets',\n", " 'do i have enough to cover new skis in my bank of america account',\n", " 'do i have enough in my chase account for a plane ticket',\n", " 'do you know how much i have in checking',\n", " \"what is my saving's account balance\",\n", " \"what's the balance in my checking\",\n", " 'how much do i have in savings',\n", " 'could you tell me what my checking account balance is',\n", " \"what's my total net worth in all of my bank accounts\",\n", " \"what's my account balance\",\n", " 'i want to view my balance',\n", " 'check my bank balance',\n", " 'how much money do i have in all of my accounts',\n", " 'how much money do i have in all of my accounts combined',\n", " \"what's my checking balance\",\n", " 'how much money do i have total',\n", " \"what's the total of my bank accounts\",\n", " 'how much money do i have in checking',\n", " 'i wish to know the balance of my bank of american account',\n", " 'what is the balance of my bank of american account',\n", " 'i need to know my bank balance',\n", " 'i wanna know the balance of my bank of american account',\n", " \"i'd like to know the balance of my bank of american account\",\n", " 'i wish to know the balance of my bank of american account now',\n", " 'please tell me my bank balance',\n", " 'i wanna know my bank balance',\n", " 'please let me know what my current bank balance is',\n", " 'what is status of my bank account',\n", " 'what is my balance',\n", " 'what is the balance of my bank account',\n", " 'what is is the details of my bank account',\n", " 'is there enough money in my discover account for a vacation',\n", " 'is there enough money in my account for expenses',\n", " 'do i have enough money in my first hawaiian bank account for a '\n", " 'vacation',\n", " 'is there enough money in my bank of hawaii for vacation',\n", " 'is there enough money in my discover account for airplane '\n", " 'tickets',\n", " 'what is my savings account balance',\n", " 'what is my savings balance',\n", " 'what is my money market account balance',\n", " 'what is my bank balance for all accounts',\n", " 'tell me my payroll account balance'],\n", " 'bill_due': ['when should i pay my bill by',\n", " 'when is my car insurance due',\n", " \"when's the gas bill due\",\n", " 'when do i pay the utilities',\n", " 'do i pay my rent this week',\n", " 'i need to know when i must pay my car bill',\n", " 'how much is my water bill',\n", " 'is my bill due this week',\n", " 'how much time left to pay my bill',\n", " 'what day is my car insurance due',\n", " \"when's the electric bill due\",\n", " \"when's the next phone bill\",\n", " 'when is my next car payment',\n", " 'when is too late to pay my cable bill',\n", " 'can you alert me when my phone bill is due',\n", " 'on what day do i pay my car payment',\n", " 'how can i know when my cable bill is due',\n", " 'what day do i have to pay for my capital one card',\n", " 'how long do i have left to pay for my chase credit card',\n", " 'what is the due date of my att bill',\n", " 'when is my xfinity bill due',\n", " 'how many more days before my verizon bill is due',\n", " 'i cant remember when my bill is due',\n", " 'what is the due date of my bill',\n", " 'what day is my bill due',\n", " 'when is my bill due',\n", " 'tell me when my bill will be due',\n", " 'how do i locate when my gas bill is do',\n", " 'i would like to know my electric bills date it needs to be '\n", " 'payed',\n", " 'i need to know where to locate when my gas bill is due',\n", " 'can you tell me when my electric bill is due',\n", " 'i need to know the due date for my credit card',\n", " 'where should i look for when my gas bill is due',\n", " 'can you tell me the date my credit card is due',\n", " \"i'd like to find the date that my gas bill is due\",\n", " 'how do i know when to pay my gas bill',\n", " 'when do i have to pay my water bill by',\n", " 'when do i have to pay my electric bill by',\n", " 'when do i have to pay my internet bill by',\n", " 'when do i have to pay my gas bill by',\n", " 'tell me when i have to pay my bill',\n", " 'what date do i have to pay my bill',\n", " 'give me the date my bill is due',\n", " 'on what day do i have to pay my nordstrom bill',\n", " 'when is my chase visa due',\n", " 'when do i need to pay my annual payment for auto insurance',\n", " 'what is the exact due date for my mortgage payment',\n", " 'what is the due date for my metronorth monthy pass',\n", " 'can you tell me when my peco energy bill is due this month',\n", " 'please find the exact payment due date for my sprint phone bill',\n", " 'what is the due date on my xfinity internet bill for february',\n", " 'what is the latest date that i can pay my direct tv bill this '\n", " 'month',\n", " 'i need the new due date for my pgw gas bill',\n", " 'what time do i have to pay z bill',\n", " 'whats the due date for z bill',\n", " 'how soon before my catering bill is due',\n", " \"how long before my lawn guy bill's due\",\n", " \"when's my heater bill due\",\n", " 'what date is my water bill due',\n", " 'whats the deadline for amex payment',\n", " 'when is my visa due',\n", " 'when is visa due',\n", " 'do you know when i need to pay my mastercard',\n", " \"what's the due date for my american express payment\",\n", " 'when is my mortgage payment due',\n", " \"what's the due date for the credit card\",\n", " \"when's the next time i have to pay the insurance\",\n", " 'how long do i have to pay the gas bill',\n", " \"when's the rent due\",\n", " 'when does the car payment come due',\n", " 'when does the electric bill up',\n", " 'how do i check when my mortgage is next up for payment',\n", " 'when do i owe the state payment for my car tags',\n", " 'when do i owe the rent',\n", " 'do i need to pay my credit card bill already',\n", " 'do you know when my next electric bill is due',\n", " 'where do i find information on when the water bill is due',\n", " 'how do i ascertain when my next insurance payment will be',\n", " 'what do i do to check when my next credit card payment is',\n", " 'when do i need to pay my at&t bill',\n", " 'what is the due date of my at&t bill',\n", " 'is my at&t bill do soon',\n", " 'when is my at&t bill due',\n", " 'what date is my at&t bill due',\n", " \"when's the ac bill due\",\n", " 'what day is the z bill due',\n", " \"what's the due date for the renting bill\",\n", " 'i want to know when a bill is due',\n", " 'what day is the bill due',\n", " 'i would like to know when the bill is due',\n", " 'what is the due date for a bill',\n", " 'when is the bill due',\n", " 'how do i find when my water bill needs to be paid',\n", " 'when is my water bill due',\n", " 'how do i find when my cable bill is due',\n", " 'when is my electric bill due',\n", " 'when do i need to pay the water bil',\n", " 'how do i find when my medical bill is due',\n", " 'when do i need to pay the cable bill',\n", " 'how do i find when my electric bill is due'],\n", " 'card_declined': ['why did my card not get accepted',\n", " 'why did my card not get accepted there',\n", " 'please tell me why did my card not get accepted',\n", " 'why did my card not get accepted please',\n", " 'why did my card not get accepted then',\n", " 'find out why my card was declined',\n", " 'please tell me why my card was declined',\n", " 'for what reason did my card get declined',\n", " 'can you tell me why my card was declined',\n", " 'i need to know why my card was declined',\n", " \"i need to know why my card was just declined at walgreen's\",\n", " 'where can i find out why my card was recently declined at '\n", " 'amazoncom',\n", " 'i tried using my card at chipotle yesterday and it was '\n", " 'declined; why',\n", " 'why was my card declined for my monthly netflix '\n", " 'subscription payment',\n", " 'can you tell me what caused my card to get declined at '\n", " 'starbucks this morning',\n", " 'i wonder why my card got declined yesterday',\n", " 'why was my card declined',\n", " 'my card was declined yesterday, why',\n", " 'is my card working properly',\n", " 'do i have enough funds in my card',\n", " 'how do i fix my card being declined',\n", " 'tell me why was my card declined at the zoo',\n", " 'explain why was my card declined at boston market',\n", " 'why did you decline my card at sfo',\n", " 'how come my card was declined at rosses',\n", " 'why did my card get declined at the dentist office',\n", " \"why did macy's decline my card\",\n", " \"bloomingdale's declined my card and i'd like to know why\",\n", " \"why wouldn't nordstrom accept my card\",\n", " \"please tell me why walgreen's wouldn't take my card\",\n", " 'can you tell me why walmart declined my card',\n", " \"i couldn't buy a mug from target because my card got \"\n", " 'declined',\n", " 'stopped by target to get a mug but my card declines',\n", " 'at target trying to buy a mug and my card was declined',\n", " 'i went to target to buy a mug but my card did not work',\n", " 'trying to buy a mug from target but my card declined',\n", " 'why was my card not accepted yesterday',\n", " \"tell me why my card didn't work yesterday\",\n", " 'what was the issue with my card yesterday',\n", " 'what was the problem with my card yesterday',\n", " 'what was wrong with my card yesterday',\n", " 'i was at home depot trying to buy plants and my card got '\n", " 'declined',\n", " 'i need to know why my card declined yesterday',\n", " 'i wish to know why my card was declined yesterday',\n", " 'i was at target trying to buy candles and my card got '\n", " 'declined',\n", " 'find out why my card declined yesterday',\n", " 'i was at sears trying to buy clothes and my card got '\n", " 'declined',\n", " 'why was my card declined yesterday',\n", " 'i was at zales trying to buy a ring and my card got '\n", " 'declined',\n", " 'i was at macys trying to buy shoes and my card got '\n", " 'declined',\n", " 'my card declined yesterday and i want to know why',\n", " 'i wish to know why my card was declined',\n", " 'tell me why my card was declined yesterday',\n", " \"so it turns out my card was declined at applebee's and i \"\n", " 'wanna know why',\n", " 'so why was my card declined yesterday',\n", " 'tell me why my card got declined',\n", " \"it turns out my card was declined at fry's and i would \"\n", " 'like to know why',\n", " 'let me know why my card got declined',\n", " 'let me know why my card was declined yesterday',\n", " 'i wanna know why my card was declined',\n", " 'i really need to know why my card was denied',\n", " \"so my card was declined at fry's and i wanna know why\",\n", " 'turns out my card was declined at wal mart and i wanna '\n", " 'know why',\n", " 'i think my card was declined at wal mart and i want to '\n", " 'know why',\n", " 'let me know why my card got declined the other day',\n", " 'please let me know why my card was declined yesterday',\n", " 'i was trying to buy qtips at walmart and my card got '\n", " 'declined',\n", " 'i was at walmart today and my card got declined when i '\n", " 'went to buy qtips',\n", " 'at walmart my card got declined when i was buyng qtips',\n", " 'buying qtips today, my card got declined at walmart',\n", " 'i was buying qtips today at walmart and my card got '\n", " 'declined',\n", " \"my card didn't go through when i was buying a case of \"\n", " 'water at walmart',\n", " 'i was at walmart when my card was declined i was only '\n", " 'trying to buy some candy',\n", " 'my card got declined at target while buying a tv',\n", " \"i was at albertsons trying to buy milk when my card wasn't \"\n", " 'accepted',\n", " 'my card was not accepted for buying a computer at best buy',\n", " 'please help me figure out why my card was declined '\n", " 'yesterday',\n", " 'why did my card get rejected yesterday',\n", " 'can you tell me why my card was declined yesterday',\n", " 'my card was declined at the store yesterday, what happened',\n", " 'i was at costco trying to buy groceries and my card got '\n", " 'declined',\n", " \"my card was declined at the casino and i'm wondering why\",\n", " 'i was at safeway trying to buy groceries and my card got '\n", " 'declined',\n", " 'i was at whole foods trying to buy groceries and my card '\n", " 'got declined',\n", " \"i was at trader joe's trying to buy groceries and my card \"\n", " 'got declined',\n", " 'i was at costco trying to buy sheets and my card got '\n", " 'declined',\n", " 'my card did not work yesterday',\n", " 'i tried to use my credit card yesterday, but it did not '\n", " 'work',\n", " 'my card was malfunctioning and was returned to me',\n", " 'for which reason was my card declined yesterday',\n", " 'yesterday, my card was declined',\n", " \"i don't understand why walgreens declined my card\",\n", " 'i tried to make a purchase yesterday but my card was '\n", " 'declined why',\n", " \"i don't understand why my card was declined yesterday\",\n", " 'what is the reason for whole foods declining my card',\n", " 'can you explain why my card was declined',\n", " 'how come starbucks declined my card when i tried to use it '\n", " 'to pay',\n", " 'how come my card was not accepted yesterday',\n", " 'find out what happened to make my card get declined '\n", " 'yesterday',\n", " 'why was my card declined at safeway'],\n", " 'credit_score': ['how do i look up my credit score',\n", " 'please look up my credit score',\n", " 'can you figure out how to find my credit score',\n", " 'are you able to lookup my credit rating',\n", " 'how can i find my credit rating',\n", " 'where can i check my credit rating',\n", " 'say my credit score',\n", " 'give me my credit score',\n", " 'i want my credit score',\n", " 'inform me of my credit score',\n", " 'clue me in on my credit score',\n", " 'provide me with my credit score',\n", " 'my credit score is',\n", " 'i need my credit score',\n", " 'find my credit score for me',\n", " 'what is my credit score',\n", " 'please get my credit score',\n", " 'can you find my credit score',\n", " \"i'd like to know what my credit rating is\",\n", " 'please tell me my credit rating',\n", " 'what in the world is my credit rating',\n", " 'my credit score is what',\n", " 'tell me my credit score please',\n", " 'can you tell me my credit score',\n", " 'find my credit score and tell it to me',\n", " 'get my credit score',\n", " 'what is my exact credit score',\n", " 'how good is my credit score',\n", " \"what's my credit score rating\",\n", " 'what kind of credit score do i have',\n", " 'if i want my credit score, how do i find it',\n", " 'how do i find information about my credit score',\n", " 'what is the process of finding my credit score',\n", " 'how do i get my credit score',\n", " 'how do i locate my current credit score',\n", " 'where is my credit score',\n", " 'how do i see my credit score',\n", " 'how exactly do i find my credit score',\n", " 'how do i locate my credit score',\n", " 'how do i find out what my credit score is',\n", " 'i really wanna know my credit score',\n", " 'i wish to know my credit rating',\n", " 'let me understand my credit rating',\n", " 'i wanna know my credit rating now',\n", " 'i gotta know my credit score',\n", " 'tell me my credit rating',\n", " 'i would love to know my credit score',\n", " 'i wish to know my credit score',\n", " 'i wanna know my credit score',\n", " 'tell me what my credit rating is',\n", " 'can you provide me my credit score',\n", " \"i'd like the number for my credit score\",\n", " 'tell me my credit score',\n", " 'verify with me my credit score',\n", " 'can you help me find my credit score',\n", " 'would you tell me my credit score',\n", " 'how do i find my credit score',\n", " 'will you tell me my credit score',\n", " 'do you know my credit score',\n", " 'how to see my credit score',\n", " 'can i see my credit score',\n", " 'hows my credit score',\n", " 'websites that share credit ratings',\n", " \"what's my credit rating\",\n", " 'how do i find out my credit rating',\n", " 'how to locate my credit score',\n", " 'where is my credit score located',\n", " 'where can i find my credit score',\n", " 'find my credit score',\n", " 'tell me the steps to getting my credit score',\n", " 'help me locate my credit score',\n", " 'lets look up my credit score',\n", " 'i would like to look up my credit score please',\n", " 'i want to find out what my credit score is',\n", " 'help me find my credit score',\n", " 'what is my fica score',\n", " 'can you reveal my credit score',\n", " 'how is my credit score rated',\n", " 'any idea what my credit score is',\n", " 'show me my credit score please',\n", " 'let me know what my credit score is',\n", " 'is my credit score high',\n", " 'i am trying to find my credit score',\n", " 'show me my credit score',\n", " 'how does my credit score look',\n", " 'is my credit report low',\n", " 'whats my credit rating',\n", " 'what is my current credit score',\n", " 'where can i check my credit score',\n", " 'where can i see my credit score',\n", " 'how can i find out my credit score',\n", " 'inform me of my current credit rating',\n", " 'please notify me of my credit rating',\n", " 'i would like to be told about my credit rating',\n", " 'can you tell me about my credit rating',\n", " 'tell me my current credit rating',\n", " 'what is the number of my credit score',\n", " \"what's my credit score\",\n", " 'let me know my credit score',\n", " 'how is my credit score'],\n", " 'direct_deposit': ['do a websearch for direct deposit set up',\n", " 'search google for how to set up direct deposit',\n", " 'info on setting up direct deposit',\n", " 'info on direct deposit set-up',\n", " 'tell me how to get my paycheck on direct deposit',\n", " 'what are the steps for setting up direct deposit for my '\n", " 'paycheck',\n", " 'give me instructions to set up direct deposit for my '\n", " 'paycheck',\n", " 'i want to set up direct deposit for my paycheck, what do '\n", " 'i need to do',\n", " 'how do i set up direct deposit for my paycheck',\n", " \"i'd really like to set up a direct deposit for my \"\n", " 'paycheck',\n", " 'i wanna set up a direct deposit for my paycheck',\n", " \"i'd like to set up a direct deposit for my paycheck\",\n", " 'i would like to set up direct deposit',\n", " 'set up direct deposit for me',\n", " 'i need to set up a direct deposit for my paycheck',\n", " 'set up a direct deposit',\n", " 'i need to set up direct deposit',\n", " 'i want to set direct deposit',\n", " 'help me get my pay check deposited directly to my home '\n", " 'checking account',\n", " 'set my paycheck up for direct deposit',\n", " 'i want my paycheck to go directly to my bank account',\n", " 'set up direct deposit to my money market account for my '\n", " 'pay check',\n", " 'how do i get direct deposit for my paycheck',\n", " \"what's the procedure to get direct deposit for my \"\n", " 'paycheck',\n", " 'set up payroll direct deposit to my checking account',\n", " 'make it so my paycheck goes directly into my savings '\n", " 'account',\n", " 'how can i have my paycheck directly deposited',\n", " 'i need to get direct deposit on my bofa account',\n", " 'what do i need to set up direct deposit',\n", " 'what is needed for setting up direct deposit',\n", " 'can you walk me through setting up direct deposit',\n", " 'what is needed to set up direct deposit',\n", " 'i need help to set up direct deposit',\n", " 'would you help me set up direct deposit',\n", " 'assist me to set up direct deposit',\n", " 'tell me how to set up direct deposit',\n", " 'tell me how to set up direct deposit for my paycheck',\n", " 'how do i direct deposit my paycheck',\n", " 'what do i do to have my paycheck deposited directly in my '\n", " 'account',\n", " 'how can i have my paycheck directly deposited in my '\n", " 'account',\n", " 'i would like to set up a direct deposit, please tell me '\n", " 'how',\n", " 'how can i set up a direct deposit with my checking '\n", " 'account',\n", " 'can you tell me how to set up a direct deposit',\n", " 'can you help me set up a direct depost',\n", " 'how can i turn on direct deposit',\n", " 'how can i set up direct deposits from my job to my bank',\n", " 'what steps do i need to do to set up direct deposit',\n", " 'i want to set up direct deposit what do i need to do',\n", " 'what do i need to do to start direct deposit',\n", " 'how do i go about setting up direct deposit',\n", " 'how do i get direct deposit set up',\n", " \"i'd like to know about setting up direct deposit\",\n", " 'can you tell me how to set up direct depost',\n", " 'how do i get my check directly deposited',\n", " 'direct deposit information',\n", " 'how do i set up instant paycheck',\n", " 'how do i direct deposit my check',\n", " 'help me set up direct deposit to my bank of hawaii '\n", " 'checking account',\n", " 'can you show me how to set up direct deposit for my '\n", " 'paycheck to my first hawaiian bank account',\n", " 'i want to set up direct deposit to my first hawaiian bank '\n", " 'account',\n", " 'can you show me how to set up my paycheck to be direct '\n", " 'deposit to my first hawaiian bank account',\n", " 'how do i set up direct deposit to my bank of hawaii '\n", " 'account',\n", " 'help me set up a direct deposit',\n", " 'how is a direct deposit set up',\n", " 'how would i go about setting up a direct deposit',\n", " 'i need to set up a direct deposit',\n", " 'tell me how to set up a direct deposit',\n", " 'onpay gives you two convenient ways to pay your employees',\n", " 'can you help me set up direct deposit',\n", " 'i need some guidance when it comes to direct deposit',\n", " 'i want to switch to direct deposit',\n", " 'how can i set up direct deposit',\n", " 'what are the steps to direct deposit my check',\n", " 'is there a specific way to set up direct deposit',\n", " 'how do i arrange a direct deposit into my savings account',\n", " 'how do you set up direct deposit',\n", " 'can you teach me how to set up direct deposit, or show me '\n", " 'who can',\n", " 'can you show me how to set up direct deposit',\n", " 'if i would like to set up direct deposit, how do i do it',\n", " 'direct deposit instructions',\n", " 'help setting up direct deposit',\n", " 'please help me set up direct deposit',\n", " 'what are the steps for setting up direct deposit',\n", " 'how does one go about setting up direct deposit',\n", " 'how to set up direct deposit for paychecks',\n", " \"what's needed to direct deposit my paycheck\",\n", " 'what is required to direct deposit my paycheck',\n", " 'how to direct deposit my paycheck',\n", " 'i need to get my paycheck direct deposited to my chase '\n", " 'account',\n", " \"i'd like to have my paycheck direct deposited to my chase \"\n", " 'account',\n", " 'what are the steps to set up direct deposit to my chase '\n", " 'account',\n", " 'how do i get my paycheck direct deposited to my chase '\n", " 'account',\n", " 'how do i set up direct deposit to my chase account',\n", " 'what do i do to enable direct deposit',\n", " 'can i get paychecks directly deposited to my bank of '\n", " 'america account',\n", " 'i have a great western bank account i want direct '\n", " 'deposits to go to',\n", " \"i'd like my paychecks direct deposited in my navyfed \"\n", " 'checking account',\n", " 'can you walk me through setting up direct deposits to my '\n", " 'bank of internet savings account',\n", " 'how do i get paychecks put directly in my sunflower '\n", " 'savings account',\n", " 'let me set up direct deposit for this'],\n", " 'freeze_account': ['can you block my chase account right away please',\n", " 'i want my chase account blocked immediately',\n", " 'i need you to block my chase account immediately',\n", " 'can you put a block on my chase account right away',\n", " 'please block my chase account right away',\n", " 'what do i do to freeze my account',\n", " 'place a hold on my bank account',\n", " 'how can i stop transactions on my account',\n", " 'freeze my account immediately',\n", " 'block my monkey market right now',\n", " 'put a stop on my deposit account',\n", " 'stop any future processing on my savings account',\n", " 'put a hault on my savings account',\n", " 'could you put a stop on my bank account, please',\n", " 'can you put a stop on my bank account now',\n", " 'can you put a stop on my bank account, please',\n", " 'could you freeze my account',\n", " 'could you freeze my account, please',\n", " 'can you put a stop on my bank account',\n", " 'can you freeze my account, please',\n", " 'could you freeze my account now',\n", " 'can you freeze my account',\n", " 'could you put a stop on my bank account',\n", " 'do you mind putting a stop on my bank account',\n", " 'i would like you to put a stop on my bank account',\n", " 'please turn my account to frozen',\n", " 'please make my account a frozen one',\n", " 'please make sure my account is frozen',\n", " 'is it too much trouble to put a stop on my bank account',\n", " 'i would love it if you could put a stop on my bank '\n", " 'account',\n", " 'i would appreciate it if you put a stop on my bank '\n", " 'account',\n", " 'please set my account as frozen',\n", " 'please freeze my account',\n", " 'i need my account frozen',\n", " \"i'd like my account frozen\",\n", " 'i would like my account frozen',\n", " 'i want my account frozen',\n", " 'i really want my account frozen',\n", " 'no payments on my bank account',\n", " 'turn off my bank account',\n", " 'stop all payments to my bank account',\n", " 'shut down my account',\n", " 'close out my account',\n", " 'turn off my account',\n", " 'i need to hold off on my account',\n", " 'plase hold my account for now',\n", " 'terminate the account',\n", " 'i am going to need a block put on my chase account right '\n", " 'away',\n", " 'would you please put a block on my chase account right '\n", " 'away',\n", " 'i would like a block put on my chase account asap',\n", " 'can you please put a block on my chase account quickly',\n", " 'i need a block put on my chase account right away',\n", " 'i need to put a freeze on my banking account',\n", " 'put a freeze on my bank account',\n", " 'freeze my bank account',\n", " 'can i freeze my bank account',\n", " 'i want you to immediatly block any further activity on my '\n", " 'bb&t bank account',\n", " 'block my citibank account right away',\n", " 'add a block to my capital one bank account so it cannot '\n", " 'be used any more',\n", " 'please immediatly block my presidential bank account '\n", " 'right now',\n", " 'put a block on my amalgamated bank account right now',\n", " 'please put a stop on my back account',\n", " 'please put a block on my td ameritrade account now',\n", " \"i'd like a stop placed on my bank account\",\n", " 'place a block on my capital one account right now',\n", " \"i'd like a block on my charles schwab account immediately\",\n", " 'i need you to place a stop on my bank account, thank you',\n", " 'place a stop on my bank account',\n", " 'i need you to freeze my account',\n", " 'i need you to block my mutualone account now',\n", " 'help me freeze my bank account, please',\n", " \"i'd like to put a freeze on my bank account\",\n", " 'block my chase account asap',\n", " 'can you please freeze my bank account',\n", " 'can you please put a stop on my bank account',\n", " 'pause my account',\n", " 'put a hold on my bank account',\n", " 'please do a stop on my bank account',\n", " 'stop payments from my bank',\n", " 'put a hold on my account',\n", " 'stop my account activity',\n", " 'please pause my banking actions',\n", " 'dont allow any action on my account',\n", " \"don't let payments go through using my bank account\",\n", " 'please freeze my bank account',\n", " 'i want my bank account frozen',\n", " 'my bank account must be frozen',\n", " 'please ask the bank to freeze my account',\n", " 'i need to freeze my bank account',\n", " 'put a block on my chase account right away',\n", " 'put a hold on my chase account right away please',\n", " 'freeze my account',\n", " 'place a stop on my main account for me please',\n", " 'please put a block on my wells fargo account',\n", " 'put a hold on my bank account please',\n", " 'put a hold on my bank of america account right away '\n", " 'please',\n", " 'can you notify the bank to put a stop on my account',\n", " 'can you put a stop on my account',\n", " 'i need a stop placed on my bank account please',\n", " 'can you put a block on my visa account right away'],\n", " 'pay_bill': ['i want to pay my bill, please',\n", " 'pay my water bill with my charles schwab account',\n", " \"i'd like to pay my bill\",\n", " 'pay my internet bill with my discover account',\n", " 'can you help me pay a bill',\n", " 'i need to pay my bill',\n", " 'can i pay a bill',\n", " 'use my capital one account to pay for my gas bill',\n", " 'pay my electric bill from my amex account',\n", " 'i want to pay my house bill',\n", " 'i want to pay my car bill',\n", " 'i want to pay my insurance bill',\n", " 'i want to pay my tax bill',\n", " 'i need help paying my auto insurance bill',\n", " 'i need help paying my hoa bill',\n", " 'i need help paying my tuition bill',\n", " 'i need help paying my phone bill',\n", " 'my water bill is due, pay it immediately',\n", " 'i need to pay my water bill',\n", " 'please go ahead and make my student loan payment',\n", " 'go ahead and pay my american express bill now',\n", " 'i need to pay my electric bill now',\n", " 'can you assist me in paying my electric bill',\n", " 'i want to do a payment on my water bill',\n", " 'pay my mortgage from my checkings accounts',\n", " 'do a car payment from my savings account',\n", " 'can you give me a hand paying my water bill',\n", " 'i want to pay off my student loan',\n", " 'i need a bit of hand holding getting my trash bill paid',\n", " 'i need to pay my mortgage',\n", " 'are you able to help me pay my mortgage',\n", " 'i want to pay my car payment',\n", " \"i need to pay this month's tv subscription fee\",\n", " 'will you aid me in paying my insurance premium',\n", " 'pay a bribe using my money market account',\n", " 'use my savings account to the pay the rent',\n", " 'use my checkings account to pay the electric bill',\n", " \"i'd like to pay my coned bill\",\n", " 'please help with paying my cell phone bill',\n", " \"i'd like to make a payment on my credit card bill\",\n", " 'i would like to pay my cell phone bill',\n", " 'please help me pay my cable bill',\n", " 'please tell me how to pay my gas bill',\n", " 'i need to get help paying my gas bill',\n", " 'i need help to pay my electric bill',\n", " 'can you help me pay my phone bill',\n", " 'will you take my bill payment',\n", " 'is it possible to pay my bill',\n", " 'how can i pay my bill',\n", " 'can i pay my bill',\n", " 'use my park bank account to pay my electric bill',\n", " 'pay my electric bill from my park bank account',\n", " 'pay my gas bill from my saving account',\n", " 'pay my cable bill from my facebook account',\n", " 'pay my cell bill from my deposit account',\n", " 'pay my electric bill from my paypal account',\n", " 'pay my water bill from my checking account',\n", " 'pay my phoe bill with my debit card',\n", " 'schedule a gas bill payment',\n", " 'pay the cable bill with my visa card',\n", " 'pay the red cross the monthly donation',\n", " 'i want to pay my internet bill',\n", " 'i want to pay my gas bill',\n", " 'i want to pay my electric bill',\n", " 'i want to pay my rent bill',\n", " 'i want to pay my water bill',\n", " 'make a payment on the electric bill',\n", " 'help me pay my electric bill',\n", " 'pay the electric bill',\n", " 'pay electric',\n", " 'please pay electric bill',\n", " 'i got to pay my cable bill',\n", " 'i need help paying my store bill',\n", " 'i wanna pay my cable bill from my checking account',\n", " 'i need help paying my cable bill',\n", " 'pay my cable bill from my checking account',\n", " 'i must pay my cable bill from my checking account',\n", " 'i gotta pay my cable bill',\n", " 'i must pay my cable bill',\n", " 'i wish to pay my cable bill',\n", " 'i need to pay my cable bill',\n", " 'i need help paying my rent bill',\n", " 'i need help paying my water bill',\n", " 'i really want to pay my cable bill from my checking account',\n", " 'i need help paying my electric bill',\n", " 'i need to pay my cable bill from my checking account',\n", " \"what's the best way to pay my bill\",\n", " \"i'd really like to pay this bill\",\n", " 'can you help me pay this bill',\n", " 'i am not sure how to pay my phone bill and need assistance',\n", " 'i need to make a bill payment',\n", " 'please pay my bill',\n", " 'can anyone help me pay my car bill',\n", " 'i want to pay that bill now',\n", " 'is there anyone available to help pay an internet bill',\n", " 'i need anyone who can help me pay my electric bill',\n", " 'help me pay my cable bill',\n", " 'can you pay the bill now',\n", " 'i need your help to pay my gas bill',\n", " 'pay my insurance bill'],\n", " 'report_fraud': [\"i'm afraid this charge on my account is fraud\",\n", " \"i think there's fraud on my account\",\n", " 'i think i have fraud on my account from walmart',\n", " \"there's been some fraudulent activity on my card\",\n", " \"i'm pretty sure this charge from sam's club is fraudulent\",\n", " \"i'm pretty sure this charge is fraudulent\",\n", " 'i need to report some fraudulent card activity',\n", " 'there are some questionable charges on my card',\n", " \"i think there's a fraudulent transaction on my account\",\n", " \"there's a shady charge from comcast on my account\",\n", " 'my account has a fraudulent transaction i think',\n", " 'can you help me with some fraudulent charges on my card',\n", " \"i think i've been the victim of fraud\",\n", " \"i'd like to report a fraudulent charge from people's \"\n", " 'natural gas',\n", " 'this costco charge looks fraudulent',\n", " \"there's fraudulent transaction going on\",\n", " 'i may have a fraudulent transaction',\n", " 'i suspect some suspicious activity',\n", " 'there seems to be fraudulent activity',\n", " 'i suspect fraudulent transaction',\n", " 'i need to report fraudulent activity on my card',\n", " \"i'm reporting fraudelent activity on my card\",\n", " 'there has been fraudulent activity on my card and i need to '\n", " 'report it',\n", " 'i need to make a report due to fraudulent activity on my '\n", " 'card',\n", " 'due to fraudulent activity on my card i need to make a '\n", " 'report',\n", " 'how do i report a fraudulent charge on my visa',\n", " 'how do i report fraud on my discover card',\n", " \"how do i let visa know about a charge i didn't make\",\n", " 'help me tell visa about fraud on my account',\n", " 'how do i tell mastercard about a fraudulent charge',\n", " \"i have activity on my bank of america card i don't \"\n", " 'recognize',\n", " 'i think a thief used my card',\n", " 'i have suspicious charges on my discovery card',\n", " \"i believe there's fraud on my card\",\n", " 'i need to report fraudulent activity on my mastercard',\n", " \"i think someone's using my visa card without my permission\",\n", " \"i have transactions on my card that aren't mine\",\n", " \"my card has purchases i don't recognize\",\n", " 'someone misused my card and put fraudulent transactions on '\n", " 'it',\n", " \"i have charges on my amex card i didn't make\",\n", " '\"disable my card account and contact company to report '\n", " 'fraudulent activty',\n", " '\"i need to report fraudulent activity to my card company',\n", " '\"please contact my credit card company to report fraudulent '\n", " 'activity on',\n", " 'it looks like someone made an unauthorized amazon purchase '\n", " 'on my account',\n", " 'i think my chase account has been compromised and fraud '\n", " 'committed',\n", " 'i may have had fraud committed on my account',\n", " \"i'm thinking someone may have used my card in a fraudulent \"\n", " 'way',\n", " 'there is a fraudulent charge for paypal on my bank account',\n", " 'i think the charge for uber on my account is fraudulent',\n", " 'looks like someone made an unauthorized charge to nike on '\n", " 'my account',\n", " 'i think someone stole my card and used it',\n", " 'i think someone made an illegal charge to my card',\n", " 'my account is showing a charge to venmo that i did not make',\n", " 'i gotta report fraudulent activity on my credit card',\n", " 'so i made a fraudulent transaction',\n", " 'i have a fraudulent transaction from wal mart on my account '\n", " 'right now',\n", " 'turns out i made a fraudulent transaction',\n", " 'report fraudulent activity on my debit card',\n", " 'i made a fraudulent transaction',\n", " 'i have a fraudulent transaction from wal mart showing on my '\n", " 'account',\n", " 'i think i made a fraudulent transaction',\n", " 'i have a fraudulent transaction from wal mart on my account',\n", " 'i must report fraudulent activity on my debit',\n", " 'i have a fraudulent transaction from red robin on my '\n", " 'account',\n", " 'report fraudulent activity on my credit card now',\n", " \"i have a fraudulent transaction from fry's on my account\",\n", " 'i got to report fraudulent activity on my credit card',\n", " \"i'm afraid there is a false transaction on my account\",\n", " 'i have detected fraudulent activity on my account',\n", " 'i see a fraudulent transaction from netflix on my account',\n", " 'i spotted a fraudulent transaction from microsoft on my '\n", " 'account',\n", " 'i see a suspicious transaction in my account history',\n", " 'it seems someone conducted a fraudulent transaction on my '\n", " 'account',\n", " 'i need help investigating a suspicious transaction',\n", " 'i need to report a fraudulent transaction from postmates on '\n", " 'my account',\n", " 'help me figure out where this fraudulent transaction from '\n", " 'google came from on my account',\n", " 'can you help me deal with this fraudulent transaction from '\n", " 'verizon on my account',\n", " 'i believe there are fraudulent charges on my card how can i '\n", " 'report them',\n", " \"i didn't make these purchases on my card; these are \"\n", " 'fraudulent charges i need to report them',\n", " 'how can i report fraudulent charges on my card',\n", " \"it seems that there's fraudulent activity on my card i'd \"\n", " 'like to file a report',\n", " 'i want to report fraudulent activity on my card',\n", " 'send information about suspicious credit card activity',\n", " 'please report information about activity on my credit card',\n", " 'send fraudulent activity information',\n", " 'please file a fraud report',\n", " 'report fraud on my card',\n", " 'i want to report fraudulent activity on my visa card, '\n", " 'please',\n", " 'i need to report fraudulent activity on my visa card, '\n", " 'please',\n", " 'i want to report fraudulent activity on my amex card',\n", " 'i want to report fraudulent activity on my visa card',\n", " 'i want to report fraudulent activity on my amex card, '\n", " 'please',\n", " 'i believe that there is some fraudulent activity on my '\n", " 'capital one account',\n", " 'there is an unauthorized transaction on my bank of america '\n", " 'checking',\n", " 'i see a purchase on my chase checking that i did not make',\n", " 'what steps do i take if there is a transaction that i do '\n", " 'not recognize on my navy federal credit union account',\n", " 'i need to talk to someone about a transaction that was not '\n", " 'made by me on my chase account',\n", " 'can you report credit card fraud for me',\n", " 'discover card reporting fraud',\n", " 'i need to know how to report fraud on my discover card'],\n", " 'transactions': ['before i make my mastercard payment can you tell me what '\n", " \"i've recently charged on it\",\n", " 'please tell me all of my recent transactions',\n", " 'i want to pay my amazon credit card but i need to know the '\n", " 'last few transactions',\n", " 'before i pay my walmart credit card did i make any '\n", " 'purchases using it recently',\n", " \"i need to know all the recent transactions i've made\",\n", " \"please tell me all the transactions i've made recently\",\n", " \"what are the last ten transactions i've made\",\n", " 'before i pay my capital one, what are the most recent '\n", " \"transactions i've made\",\n", " 'tell me the most recent charges on my chase credit card '\n", " 'before i pay the bill',\n", " \"read off to me the last five transactions i've made\",\n", " 'what transactions have i made on liquor in the past month',\n", " \"i'd like to see last week's atm transactions\",\n", " 'show me the transactions from costco yesterday',\n", " 'i need to know something about my latest transaction',\n", " 'can you check on a transaction for me',\n", " 'i need some information on a recent transaction',\n", " 'i need a recent transaction looked into',\n", " 'can you let me know my latest transactions',\n", " 'what was the last thing i purchased',\n", " 'what did i buy last',\n", " 'how much did my last purchase cost',\n", " 'how expensive was my most recent transaction',\n", " 'when was my most recent transaction',\n", " 'show me the transactions made on my business card for '\n", " 'supplies last quarter',\n", " 'may i get all of the food transactions that were made last '\n", " 'month',\n", " 'can you show me the transactions that were made last night '\n", " 'at the restaurant',\n", " 'what transactions did i make yesterday at the flea market',\n", " 'please show me what transactions i made on the first of '\n", " 'this month',\n", " 'please tell me my in-person transactions for the last three '\n", " 'days using my debit card',\n", " 'please give me my last ten debit card transactions in the '\n", " 'month of december',\n", " 'what were my last five transactions on my visa card',\n", " 'i would like to hear all transactions made on my amex for '\n", " 'the last ten days',\n", " 'can you list all of my online transactions for the month of '\n", " 'january',\n", " 'show me my transactions on groceries',\n", " 'show me my transactions on tacobell',\n", " 'show me my transactions on make up at sephora',\n", " 'show me my transactions on mcdonalds',\n", " 'show me my transactions on clothes at macys',\n", " 'let me see the list of tranaction on my discovery credit '\n", " 'card',\n", " 'let me know the list of transaction on my first hawaiian '\n", " 'bank',\n", " 'i want to see the list of transaction on my bank of hawaii',\n", " 'list me my recent transaction',\n", " 'what are my recent transaction',\n", " 'please show me my recent transaction',\n", " 'i want to see my recent transaction',\n", " 'let me check my transaction for first bank card',\n", " 'let me check my transaction on my citi card',\n", " 'show me my recent transactoin',\n", " 'on my card what have been my recent purchases',\n", " 'pull up my recent transactions on my mastercard',\n", " 'on the card can you give me the last transactions',\n", " 'whats my recent transactions on my card',\n", " 'on my card what all are my latest transactions',\n", " 'show my transaction statement',\n", " 'retrieve my recent transactions',\n", " 'view my transactions for the last week',\n", " 'my transaction history',\n", " 'may transactions for last 2 days',\n", " 'what was my last transaction',\n", " 'show me recent transactions',\n", " 'what transactions happened in the last week',\n", " 'show me the last month of transactions',\n", " 'show me the last five transactions',\n", " \"what's the last transaction i made yesterday\",\n", " \"show me yesterday's last transaction\",\n", " \"open yesterday's last payment\",\n", " 'show me last transaction',\n", " 'show me transactions related to travel',\n", " 'help me get access to my recent transaction history',\n", " 'i need to see my shopping transactions',\n", " 'get me access to a list of my recent transactions',\n", " 'i would like to take a look at my transaction history',\n", " 'can you pull up my most recent transactions',\n", " 'i want to see my entertainment transactions',\n", " 'where can i find my recent transaction history',\n", " 'can you show me transactions related to utilities',\n", " 'pull up my grocery transactions',\n", " 'what were my purchases on visa card last month',\n", " 'i spent what dollar amount last month on mastercard',\n", " 'what was the amount that i spent on fidelity visa last '\n", " 'month',\n", " 'what did i spend at target on my barclays card last month',\n", " 'what amount did i spend for food on chase visa on current '\n", " 'bill',\n", " 'i need to know last months transactions',\n", " 'i need to know the transactions i made on the 10th',\n", " 'i would like to see the transactions i made last week',\n", " 'can i see my transactions on the 7th',\n", " 'can i get my transactions for the date of the 23rd',\n", " 'show me the transaction on burger king',\n", " 'show me the transaction on macys',\n", " 'show me the transaction on mcdonald',\n", " 'show me the transaction on the food',\n", " 'show me the transaction on my car',\n", " 'bring up all purchases from target',\n", " 'show me my transactions within the past week',\n", " 'can i see my transactions from yesterday',\n", " 'show me all gas purchases within the last month',\n", " 'can you list all recent transactions',\n", " 'bring up my most recent purchases',\n", " 'i need to see all visa purchases for march',\n", " 'looking at january, show all wine purchases'],\n", " 'transfer': ['i need $20000 transferred from my savings to my checking',\n", " 'complete a transaction from savings to checking of $20000',\n", " 'transfer $20000 from my savings account to checking account',\n", " 'take $20000 from savings and put it in checking',\n", " 'put $20000 into my checking account from my savings account',\n", " 'send 100 dollars between bank of the west and bank of america '\n", " 'acccounts',\n", " 'send 50 dollars between bank of america and chase accounts',\n", " 'send 2000 dollars between chase and rabobank accounts',\n", " 'send 1200 dollars between usaa and navy federal accounts',\n", " 'send 400 dollars between city bank and usaa accounts',\n", " 'take $40 and transfer it to account a from b',\n", " 'transfer $40 from account a to b',\n", " 'move $40 from account b to account a',\n", " 'put $40 from account a to b',\n", " 'take $40 from account a and transfer it to account b',\n", " 'i need to transfer from this account to that one',\n", " 'need to transfer from one account to my other one',\n", " 'i would like to transfer from one account to my second one',\n", " 'i need to transfer from one account to my second one',\n", " 'i have to transfer from one account to my other one',\n", " 'send fifty dollars from me to carrie',\n", " 'transfer sixty dollars to dad from my biggest accnt',\n", " 'send over a hundred dollars from huntington into saving',\n", " 'move 57 dollars from saving into mom',\n", " 'transfer two hundred dollars between my portfolio and my money '\n", " 'market account',\n", " 'i need to transfer ten dollars from my bank of america account '\n", " 'to my capital one account',\n", " 'go ahead and send ten dollars from bank of america to capital '\n", " 'one',\n", " 'i must transfer ten dollars from my bank of america account to '\n", " 'my capital one account',\n", " 'i got to transfer ten dollars from my bank of america account '\n", " 'to my capital one account',\n", " 'i want you to send ten dollars from bank of america to capital '\n", " 'one',\n", " 'i need you to send ten dollars from bank of america to capital '\n", " 'one',\n", " 'i have to transfer ten dollars from my bank of america account '\n", " 'to my capital one account',\n", " 'send ten dollars from bank of america to capital one',\n", " 'please send ten dollars from bank of america to capital one',\n", " 'make a transfer of $200 from my savings account to my checking '\n", " 'account',\n", " 'please transfer $250 from checking to savings',\n", " 'transfer $500 from my checking to my savings',\n", " 'please transfer $100 from my checking to my savings account',\n", " 'transfer $500 from my money market savings account to my '\n", " 'checking account',\n", " 'send money from one account to another',\n", " 'transfer 200 dollars from paypal to savings',\n", " 'transfer $10 from checking to savings',\n", " 'send $100 from paypal to my bank',\n", " 'please transfer 100 dollars between my amazon payments and '\n", " 'savings accounts',\n", " 'i would like to make a transfer',\n", " 'i want to transfer funds between accounts',\n", " 'send over 50 dollars between my shared and not shared accounts',\n", " 'whats the quickest way to money from one account to another',\n", " 'what is the process to move money from one account to another',\n", " 'i need you to send 500 dollars from my high tier account to my '\n", " 'regular checking account',\n", " 'can you transfer money from a to b',\n", " 'send my money between accounts',\n", " 'how can i send money from one account to another',\n", " 'i need to move money from one account to another',\n", " \"send $20 from debit to steve's account\",\n", " 'send 20 dollars from savings to checking',\n", " 'please transfer my funds',\n", " 'please transfer $50 to my checking account from credit',\n", " 'go ahead and move $200 from amazon to my bank account',\n", " 'help me move my money',\n", " 'i need my money to be moved',\n", " 'help me move my money please',\n", " 'i need to move my money',\n", " 'move my money please',\n", " 'i would like to transfer $5 from savings to checking',\n", " 'transfer $5 from savings to checking',\n", " 'can you transfer $5 from savings to checking',\n", " 'send $5 from savings to checking',\n", " 'please transfer $5 from savings to checking',\n", " 'please switch $s checking to mortage',\n", " 'please transfer $s from saving to checking',\n", " 'could you transfer $x from saving to checking',\n", " 'help me transfer $x from credit to debit',\n", " 'please transfer $x from checking to saving',\n", " 'transfer money to another account',\n", " 'requesting money transfer between accounts',\n", " 'transfer money from this account to that account',\n", " 'can you transfer money from my account',\n", " 'money transfer request',\n", " 'send 100 dollars from checking to savings',\n", " 'transfer 100 dollars checking to savings',\n", " 'transfer 100 dollars between checking and savings',\n", " 'take 100 dollars from checking and send it to savings',\n", " 'move 100 dollars from checking to savings',\n", " 'can we wire some money over to the other account i have',\n", " 'time to move some cash from one account to another',\n", " 'i need to throw some money into that other account',\n", " \"let's send some money to another account\",\n", " 'i want to initiate a transfer from one account to the other',\n", " 'transfer money from one place to another',\n", " 'move money from one account to another',\n", " 'transfer between two accounts',\n", " 'make a transfer between accounts',\n", " 'send money to another account',\n", " 'i need ten dollars sent from b of a to washington mutual',\n", " 'transfer fifty dollars between my chase and bank of america '\n", " 'account',\n", " 'deposit ten bucks from my chase account to my ally account',\n", " 'put a hundred dollars in my ally account from bluebird',\n", " 'i want seventy bucks transferred from b of a to chase',\n", " 'can i initiate a one-time transfer from my savings account to '\n", " 'my money market account']}\n" ] } ], "source": [ "# Group the dataset by intent classes\n", "train_by_intent = defaultdict(list)\n", "\n", "for example in data_subset['train']:\n", " intent_name = intent_names[example['intent']]\n", " train_by_intent[intent_name].append(example['text'])\n", "\n", "train_by_intent = dict(train_by_intent)\n", "pprint(train_by_intent)" ] }, { "cell_type": "markdown", "id": "da14d6ab", "metadata": {}, "source": [ "By grouping the dataset by intent classes, it is easier to visualize what each route and its respective references might look like." ] }, { "cell_type": "markdown", "id": "b1307f13", "metadata": {}, "source": [ "## Helper Functions" ] }, { "cell_type": "markdown", "id": "a8a7807c", "metadata": {}, "source": [ "We define some reusable helper functions to measure the effectiveness of our strategies." ] }, { "cell_type": "code", "execution_count": null, "id": "856815a1", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from time import perf_counter\n", "\n", "### For profiling latency\n", "class profile_block:\n", " \"\"\"Context manager for profiling code block and measuring latency\"\"\"\n", " def __init__(self, label=\"Block\"):\n", " self.label = label\n", " self.elapsed = None\n", "\n", " def __enter__(self):\n", " self._start = perf_counter()\n", " return self\n", "\n", " def __exit__(self, exc_type, exc, tb):\n", " self.elapsed = perf_counter() - self._start\n", "\n", "def get_router_classification(router, text):\n", " # Normalize the router output into the same dictionary format used by\n", " # the rest of the evaluation helpers.\n", " pred = router(statement=text)\n", " return {\"name\": pred.name,\n", " \"distance\": pred.distance}\n", "\n", "def evaluate_classification(classifier, split=\"test\"):\n", " \"\"\"Evaluate a classifier on a CLINC split.\"\"\"\n", " eval_split = data_subset[split]\n", "\n", " num_correct = 0\n", " total_distance = 0.0\n", " num_distance_values = 0\n", " latencies_ms = []\n", " case_results = []\n", " num_examples = len(eval_split)\n", "\n", " for example in eval_split:\n", " text = example['text']\n", " true_intent = intent_names[example['intent']]\n", "\n", " # Profile and time the classification call\n", " with profile_block('classification_call') as timer:\n", " result = classifier(text)\n", " \n", " latencies_ms.append(timer.elapsed * 1000)\n", " \n", " predicted_intent = result['name']\n", " distance = result['distance']\n", "\n", " is_correct = predicted_intent == true_intent\n", "\n", " if is_correct:\n", " num_correct += 1\n", "\n", " if distance is not None:\n", " total_distance += distance\n", " num_distance_values += 1\n", "\n", " # Keep the raw per-example outputs so we can later inspect mistakes,\n", " # sort by router distance, or build additional analyses.\n", " case_results.append({\n", " 'text': text,\n", " 'true_intent': true_intent,\n", " 'predicted_intent': predicted_intent,\n", " 'distance': distance,\n", " 'is_correct': is_correct,\n", " })\n", "\n", " # Summarize the split using both accuracy and latency percentiles.\n", " aggregate_results = {\n", " 'num_examples': num_examples,\n", " 'accuracy': round(num_correct / num_examples, 4),\n", " \"avg_latency\": round(float(np.mean(latencies_ms)), 2),\n", " \"p95_latency\": round(float(np.percentile(latencies_ms, 95)), 2),\n", " \"p99_latency\": round(float(np.percentile(latencies_ms, 99)), 2),\n", " }\n", "\n", " if num_distance_values:\n", " aggregate_results['avg_distance'] = round(total_distance / num_distance_values, 4)\n", "\n", " per_test_case_results = {\n", " 'per_test_case': case_results,\n", " }\n", "\n", " return aggregate_results, per_test_case_results\n" ] }, { "cell_type": "markdown", "id": "b39aa612", "metadata": {}, "source": [ "## Part I: Choice of embedding model" ] }, { "cell_type": "markdown", "id": "1b110863", "metadata": {}, "source": [ "### Strategy 1: Pick an embedding model that better suits your use case" ] }, { "cell_type": "markdown", "id": "c6650c92", "metadata": {}, "source": [ "Sometimes, simply picking another embedding model (either an open source one or even better, a finetuned one) might pay good dividends in improving routing performance." ] }, { "cell_type": "markdown", "id": "2b18a0c0", "metadata": {}, "source": [ "#### Scenario setup" ] }, { "cell_type": "markdown", "id": "a80a842a", "metadata": {}, "source": [ "For this scenario, we'll be creating semantic routers using routes and references from the training dataset, and exploring how we can improve routing performance simply by picking another embedding model." ] }, { "cell_type": "code", "execution_count": 11, "id": "d456dd2c", "metadata": {}, "outputs": [], "source": [ "scenario_routes = [Route(name=intent, references=references) \n", " for intent, references in train_by_intent.items()\n", " ]" ] }, { "cell_type": "markdown", "id": "45d2cf01", "metadata": {}, "source": [ "#### Create a basic router" ] }, { "cell_type": "markdown", "id": "b0ccc377", "metadata": {}, "source": [ "We create a basic router as a baseline for this scenario using the `HFTextVectorizer` class from RedisVL, leveraging the pre-trained embedding models provided by the `sentence-transformers` library. (Refer to this [link](https://sbert.net/docs/sentence_transformer/pretrained_models.html) for the list of available embedding models provided by the library.)\n", "\n", "For the baseline, we use the `sentence-transformers/all-MiniLM-L6-v2` model." ] }, { "cell_type": "code", "execution_count": 12, "id": "ed56408e", "metadata": {}, "outputs": [], "source": [ "basic_router_minilm = SemanticRouter(\n", " name='basic-router-miniLM-L6',\n", " routes=scenario_routes,\n", " vectorizer=HFTextVectorizer(model=\"sentence-transformers/all-MiniLM-L6-v2\"),\n", " redis_client=client,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "id": "54c9bad5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy on MiniLM-L6 embedding model: 86.67%\n" ] } ], "source": [ "basic_router_minilm_metrics, basic_router_minilm_per_test_case = evaluate_classification(\n", " lambda text: get_router_classification(basic_router_minilm, text))\n", "\n", "print(f\"Accuracy on MiniLM-L6 embedding model: {100*basic_router_minilm_metrics['accuracy']:.2f}%\")" ] }, { "cell_type": "markdown", "id": "55d593f2", "metadata": {}, "source": [ "#### Picking a different embedding model" ] }, { "cell_type": "markdown", "id": "753fc7a5", "metadata": {}, "source": [ "Now, we pick a different model. `sentence-transformers` provides us with `sentence-transformers/all-mpnet-base-v2`, which scores the highest on their performance benchmark.\n", "\n", "This is also the embedding model used by `HFTextVectorizer` by default." ] }, { "cell_type": "code", "execution_count": 14, "id": "03ae6b9d", "metadata": {}, "outputs": [], "source": [ "basic_router_mpnet = SemanticRouter(\n", " name='basic-router-mpnet',\n", " routes=scenario_routes,\n", " vectorizer=HFTextVectorizer(model=\"sentence-transformers/all-mpnet-base-v2\"),\n", " redis_client=client,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 15, "id": "8d5a4a1f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy on mpnet embedding model: 92.67%\n" ] } ], "source": [ "basic_router_mpnet_metrics, basic_router_mpnet_per_test_case = evaluate_classification(\n", " lambda text: get_router_classification(basic_router_mpnet, text))\n", "\n", "print(f\"Accuracy on mpnet embedding model: {100*basic_router_mpnet_metrics['accuracy']:.2f}%\")" ] }, { "cell_type": "markdown", "id": "dff8aa01", "metadata": {}, "source": [ "By simply picking a \"better\" embedding model, we have managed to improve the routing accuracy. Imagine how much the performance would increase by if we use a finetuned embedding model!" ] }, { "cell_type": "markdown", "id": "8ff334fe", "metadata": {}, "source": [ "## Part II : Bootstrapping better Semantic Routes" ] }, { "cell_type": "markdown", "id": "a510bb4e", "metadata": {}, "source": [ "### Strategy 2: LLM-based data augmentation" ] }, { "cell_type": "markdown", "id": "20b93994", "metadata": {}, "source": [ "In practice, we might be unable to get our hands on references that represent the semantic space of each intent well. This can happen due to lack of access to (human) expert generation of references, especially if there are many intents to generate representative references for.\n", "\n", "A strategy we can employ here is to use an LLM to generate synthetic data to augment the current set of references." ] }, { "cell_type": "markdown", "id": "0dadf918", "metadata": {}, "source": [ "#### Scenario setup" ] }, { "cell_type": "markdown", "id": "f7319bee", "metadata": {}, "source": [ "To simulate this scenario, we downsample the CLINC dataset to obtain 3 labelled references per intent. These samples are high-quality, similar to what a human expert might curate for a semantic router." ] }, { "cell_type": "code", "execution_count": 16, "id": "df263aa7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'balance': ['what is my bank balance for all accounts',\n", " \"what's my bank balance\",\n", " 'perform a search for my most recent balance on my amex account'],\n", " 'bill_due': ['give me the date my bill is due',\n", " 'i would like to know when the bill is due',\n", " 'i need to know the due date for my credit card'],\n", " 'card_declined': ['i was at zales trying to buy a ring and my card got '\n", " 'declined',\n", " 'can you tell me why my card was declined',\n", " 'why did my card not get accepted'],\n", " 'credit_score': ['can you find my credit score',\n", " 'i wanna know my credit rating now',\n", " 'lets look up my credit score'],\n", " 'direct_deposit': ['let me set up direct deposit for this',\n", " \"i'd like to set up a direct deposit for my paycheck\",\n", " 'can you teach me how to set up direct deposit, or show me '\n", " 'who can'],\n", " 'freeze_account': ['could you put a stop on my bank account, please',\n", " 'please block my chase account right away',\n", " 'would you please put a block on my chase account right '\n", " 'away'],\n", " 'pay_bill': ['i need to get help paying my gas bill',\n", " 'can i pay a bill',\n", " 'go ahead and pay my american express bill now'],\n", " 'report_fraud': ['i need to report fraudulent activity on my card',\n", " \"i'm afraid this charge on my account is fraud\",\n", " 'i have a fraudulent transaction from wal mart on my account '\n", " 'right now'],\n", " 'transactions': ['help me get access to my recent transaction history',\n", " 'i would like to take a look at my transaction history',\n", " 'what are my recent transaction'],\n", " 'transfer': ['send 50 dollars between bank of america and chase accounts',\n", " 'make a transfer of $200 from my savings account to my checking '\n", " 'account',\n", " 'transfer $40 from account a to b']}\n" ] } ], "source": [ "num_samples = 3\n", "\n", "scenario_subset = {intent: random.sample(references, num_samples) \n", " for intent, references in train_by_intent.items()\n", " }\n", "pprint(scenario_subset)" ] }, { "cell_type": "code", "execution_count": 17, "id": "65f4e6a6", "metadata": {}, "outputs": [], "source": [ "scenario_routes = [Route(name=intent, references=references) \n", " for intent, references in scenario_subset.items()\n", " ]" ] }, { "cell_type": "markdown", "id": "8616cdb6", "metadata": {}, "source": [ "#### Basic semantic router behaviour on limited data" ] }, { "cell_type": "code", "execution_count": 18, "id": "b3f4c927", "metadata": {}, "outputs": [], "source": [ "# Set up a basic semantic router\n", "scenario_router_base = SemanticRouter(\n", " name='base-router-limited-data',\n", " routes=scenario_routes,\n", " vectorizer=HFTextVectorizer(),\n", " redis_client=client,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 19, "id": "6f061abb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy on basic router (with limited data): 83.00%\n" ] } ], "source": [ "scenario_router_metrics, scenario_router_per_test_case = evaluate_classification(\n", " lambda text: get_router_classification(scenario_router_base, text))\n", "\n", "print(f\"Accuracy on basic router (with limited data): {100*scenario_router_metrics['accuracy']:.2f}%\")" ] }, { "cell_type": "markdown", "id": "3172ef75", "metadata": {}, "source": [ "#### Augmenting route references \n", "using LLM-generated synthetic data " ] }, { "cell_type": "code", "execution_count": 20, "id": "c8dfe38b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'balance': [\"What's my checking balance?\",\n", " 'Show my account balance',\n", " 'How much money do I have in savings?',\n", " 'Check my available balance',\n", " \"What's left in my account?\",\n", " 'Tell me my current balance',\n", " 'View balance for account ending in 4432',\n", " 'How much is in my checking right now?',\n", " 'Get my savings balance',\n", " 'Show available funds',\n", " 'What is my ledger balance?',\n", " 'Check the balance on my joint account',\n", " 'See my account totals',\n", " 'Do I have enough money in checking?',\n", " 'Display my balances',\n", " 'How much cash do I have available?',\n", " 'I want to see my current account balance',\n", " 'Balance for my student account',\n", " 'Let me know my available amount',\n", " \"What's the balance on my main account?\"],\n", " 'bill_due': ['When is my credit card bill due?',\n", " \"What's my next bill due date?\",\n", " 'Check my upcoming bill due date',\n", " 'When do I need to pay my electric bill?',\n", " 'Show bills due this month',\n", " 'Is my mortgage payment due soon?',\n", " 'Tell me when my next payment is due',\n", " 'What bills are coming up?',\n", " 'When is my phone bill due?',\n", " 'See my due dates',\n", " 'Do I have any bills due today?',\n", " 'Find my next due payment',\n", " 'When is my loan payment due?',\n", " 'Show my upcoming bill deadlines',\n", " \"What's due this week?\",\n", " 'Check the due date for my utilities',\n", " 'Let me know my next bill due',\n", " 'When does my internet bill need to be paid?',\n", " 'Are any bills past due?',\n", " 'Display upcoming due dates'],\n", " 'card_declined': ['My card was declined',\n", " 'Why is my debit card being declined?',\n", " \"My transaction didn't go through\",\n", " 'Card declined at checkout',\n", " 'My card keeps getting rejected',\n", " 'Why was my card denied?',\n", " 'My purchase was declined',\n", " \"The card didn't work\",\n", " \"I can't use my card right now\",\n", " 'My debit card was refused',\n", " 'Card declined for an online payment',\n", " \"My card isn't working at the store\",\n", " \"Why won't my card go through?\",\n", " 'The payment was declined on my card',\n", " 'My card got declined at the gas station',\n", " 'I was told my card was invalid',\n", " 'My bank card is not being accepted',\n", " 'Declined transaction on my card',\n", " 'Can you tell me why my card failed?',\n", " 'My card was turned down'],\n", " 'credit_score': [\"What's my credit score?\",\n", " 'Show my current credit score',\n", " 'Check my credit score',\n", " 'Let me see my credit rating',\n", " 'Can you tell me my FICO score?',\n", " 'View my credit score details',\n", " 'I want to know my credit score',\n", " 'Display my latest credit score',\n", " 'Has my credit score changed?',\n", " 'Get my score update',\n", " 'Show me my credit health',\n", " 'What is my current score?',\n", " 'Pull my credit score',\n", " 'See my credit report score',\n", " 'Give me my credit score info',\n", " 'Check if my score went up',\n", " 'How good is my credit score right now?',\n", " 'Open my credit score dashboard',\n", " \"What's my latest reported score?\",\n", " 'See my score history'],\n", " 'direct_deposit': ['Set up direct deposit',\n", " 'How do I enable direct deposit?',\n", " 'I need my direct deposit information',\n", " 'Get my routing and account number for payroll',\n", " 'Help me start direct deposit',\n", " 'Where can I find direct deposit details?',\n", " 'Send me my direct deposit form',\n", " 'Set my paycheck up for direct deposit',\n", " 'I want to switch my pay to direct deposit',\n", " 'Show my direct deposit instructions',\n", " 'Can I use this account for payroll deposit?',\n", " 'Provide my employer deposit information',\n", " 'Direct deposit setup for my checking account',\n", " 'I need a voided check for direct deposit',\n", " 'Find my account details for direct deposit',\n", " 'Enable paycheck deposit to my account',\n", " 'Give me the form for employer direct deposit',\n", " 'How can I receive my salary by direct deposit?',\n", " 'Set up my work paycheck deposit',\n", " 'I want to add direct deposit to this account'],\n", " 'freeze_account': ['Freeze my account',\n", " 'Lock my debit card account right away',\n", " 'Temporarily freeze access to my account',\n", " 'Put a hold on my bank account',\n", " 'I need to lock my account',\n", " 'Suspend my account for now',\n", " 'Freeze transactions on my account',\n", " 'Block my account immediately',\n", " 'Restrict my account until I confirm activity',\n", " 'Lock down my checking account',\n", " 'Can you freeze my account today?',\n", " 'Pause my account access',\n", " 'Disable my account temporarily',\n", " 'Stop all activity on my account',\n", " 'Secure my account by freezing it',\n", " 'Freeze my savings account',\n", " 'I want to place a freeze on my account',\n", " 'Shut off access to my account for now',\n", " 'Prevent any new transactions on my account',\n", " 'Please lock my bank account'],\n", " 'pay_bill': ['Pay my electric bill',\n", " 'Make a bill payment',\n", " 'Pay my credit card bill',\n", " 'Send payment to Verizon',\n", " 'Pay my water bill today',\n", " 'Use my checking account to pay a bill',\n", " 'Submit a payment for my mortgage',\n", " 'Pay my internet bill',\n", " 'Make a utility payment',\n", " 'Pay the amount due on my card',\n", " 'Send $120 to my phone provider',\n", " 'Process my bill payment now',\n", " 'I need to pay my gas bill',\n", " 'Pay my rent bill',\n", " 'Make a payment to Chase',\n", " 'Pay my insurance bill',\n", " 'Take care of my bill',\n", " 'Pay Comcast from my checking account',\n", " 'Schedule a bill payment',\n", " 'Pay my monthly loan bill'],\n", " 'report_fraud': ['I need to report fraud',\n", " 'Report a fraudulent charge',\n", " \"There is a transaction I don't recognize\",\n", " 'My account has suspicious activity',\n", " 'I want to report unauthorized charges',\n", " 'Someone used my card without permission',\n", " 'Flag this as fraud',\n", " 'I think my account was compromised',\n", " 'Report a scam transaction',\n", " \"There's fraud on my debit card\",\n", " 'Open a fraud claim',\n", " \"I didn't make this purchase\",\n", " 'Report suspicious account activity',\n", " 'My card was used fraudulently',\n", " 'Help me report identity theft on my account',\n", " 'This charge is unauthorized',\n", " 'I need to dispute a fraudulent payment',\n", " 'There are unknown transactions on my account',\n", " 'Submit a fraud report',\n", " 'I believe this transfer was fraudulent'],\n", " 'transactions': ['Show my recent transactions',\n", " 'Let me see my transaction history',\n", " 'What were my last 10 purchases?',\n", " 'Display recent account activity',\n", " 'List my latest transactions',\n", " 'Show transactions for this week',\n", " 'I want to review my account activity',\n", " 'Pull up my recent debit card charges',\n", " 'What transactions posted today?',\n", " 'See all transactions from last month',\n", " 'Give me my payment history',\n", " 'Show withdrawals and deposits',\n", " 'View spending activity on my checking account',\n", " 'What did I buy yesterday?',\n", " 'Open my transaction list',\n", " 'Show posted transactions only',\n", " 'Can I see recent transfers and payments?',\n", " 'Display activity for account ending in 7744',\n", " 'Bring up my account statement activity',\n", " 'I need to check my recent charges'],\n", " 'transfer': ['Move $200 from checking to savings',\n", " 'Transfer money to my savings account',\n", " 'Send $75 from my checking to my credit card',\n", " 'Shift funds between my accounts',\n", " 'Move $1,000 into checking',\n", " 'Transfer $50 to my joint account',\n", " 'Can I move money from savings to checking?',\n", " 'Make an internal transfer today',\n", " 'Send $300 from account ending in 1123 to 8891',\n", " 'Transfer funds now',\n", " 'Move some money to cover my balance',\n", " 'Schedule a transfer from savings',\n", " 'I need to transfer cash between accounts',\n", " 'Put $25 into my vacation savings',\n", " 'Move money out of my checking account',\n", " 'Send $600 to my brokerage account',\n", " 'Transfer between my Bank accounts',\n", " 'Shift $90 from one account to another',\n", " 'Move funds to my student checking',\n", " 'Complete a transfer for me']}\n" ] } ], "source": [ "# NBVAL_SKIP\n", "synthetic_reference_prompt = f\"\"\"\n", "You are helping generate synthetic route references for a semantic router.\n", "\n", "Generate valid JSON with this schema:\n", "{{\n", " \\\"transfer\\\": [\\\"...\\\"],\n", " \\\"balance\\\": [\\\"...\\\"],\n", " \\\"freeze_account\\\": [\\\"...\\\"],\n", " \\\"transactions\\\": [\\\"...\\\"],\n", " \\\"pay_bill\\\": [\\\"...\\\"],\n", " \\\"credit_score\\\": [\\\"...\\\"],\n", " \\\"bill_due\\\": [\\\"...\\\"],\n", " \\\"report_fraud\\\": [\\\"...\\\"],\n", " \\\"direct_deposit\\\": [\\\"...\\\"],\n", " \\\"card_declined\\\": [\\\"...\\\"]\n", "}}\n", "\n", "Requirements:\n", "- Generate only route references, not evaluation pairs.\n", "- Keep examples stylistically similar to the existing references.\n", "- Make them diverse and realistic.\n", "- Return only valid JSON.\n", "\"\"\"\n", "\n", "synthetic_reference_user_input = \"Generate 20 new references per intent. Return only valid JSON.\"\n", "\n", "response = await openai_client.responses.create(\n", " model=\"gpt-5.4\",\n", " instructions=synthetic_reference_prompt,\n", " input=synthetic_reference_user_input,\n", " reasoning={\"effort\": \"low\"},\n", " text={\"verbosity\": \"low\"},\n", " )\n", "\n", "synthetic_generated_references = json.loads(response.output_text)\n", "pprint(synthetic_generated_references)\n" ] }, { "cell_type": "markdown", "id": "70f7c595", "metadata": {}, "source": [ "The LLM is able to generate route references that are quite similar to the original ones from the dataset." ] }, { "cell_type": "code", "execution_count": null, "id": "0a3f3d3d", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "augmented_route_references = {intent: references + synthetic_generated_references[intent] \n", " for intent, references in scenario_subset.items()\n", " }\n", "\n", "scenario_routes_augmented = [Route(name=intent, references=references) \n", " for intent, references in augmented_route_references.items()\n", " ]" ] }, { "cell_type": "markdown", "id": "1bf50fbf", "metadata": {}, "source": [ "#### Semantic router performance with augmented route references" ] }, { "cell_type": "code", "execution_count": null, "id": "6ef6ed56", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "# Set up a basic semantic router\n", "scenario_router_augmented = SemanticRouter(\n", " name='llm-augmented-banking-router',\n", " routes=scenario_routes_augmented,\n", " vectorizer=HFTextVectorizer(),\n", " redis_client=client,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "24358b52", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy on basic router (with LLM-augmented data): 89.33%\n" ] } ], "source": [ "# NBVAL_SKIP\n", "scenario_router_augmented_metrics, scenario_router_augmented_per_test_case = evaluate_classification(\n", " lambda text: get_router_classification(scenario_router_augmented, text)\n", ")\n", "\n", "print(f\"Accuracy on basic router (with LLM-augmented data): {100*scenario_router_augmented_metrics['accuracy']:.2f}%\")" ] }, { "cell_type": "markdown", "id": "914571db", "metadata": {}, "source": [ "By generating synthetic references for each route using an LLM, we have managed to push the accuracy up." ] }, { "cell_type": "markdown", "id": "e21c7723", "metadata": {}, "source": [ "Note: for this strategy, few-shot prompting could be used to guide the LLM's generation. However, it might result in less effective data augmentation if the LLM-generated references are very similar to the original references. The idea is to get the LLM to generate synthetic references that better define the semantic space of each intent, rather than to generate similar references to those we already have." ] }, { "cell_type": "markdown", "id": "f2de4135", "metadata": {}, "source": [ "### Strategy 3: Bootstrapping intent routes using LLM" ] }, { "cell_type": "markdown", "id": "9cd95d00", "metadata": {}, "source": [ "In practice, even if we have abundant references, we might not have access to human experts to classify them into their respective routes. When this happens, a strategy we can adopt is simply to use the LLM as a temporary classifier. \n", "\n", "In this scenario, we have identified route definitions and a bunch of unclassified references from our production environment. We use an LLM to generate labels for these unlabelled references, and then use this pseudo-labelled dataset to build our semantic router. This effectively bootstraps a labelled dataset using an LLM, since we generate training data from unlabelled inputs.\n", "\n", "This method is a practical alternative when manually labelling route references is expensive, because the LLM acts like a strong zero-shot classifier to generate higher quality reference labels for the router." ] }, { "cell_type": "markdown", "id": "cb2b4f32", "metadata": {}, "source": [ "#### Scenario setup" ] }, { "cell_type": "markdown", "id": "27b94a14", "metadata": {}, "source": [ "For this scenario, we sample the CLINC dataset to obtain a few references, which represent unclassified samples from production." ] }, { "cell_type": "code", "execution_count": 24, "id": "019f6fea", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['transfer 200 dollars from paypal to savings',\n", " 'send money to another account',\n", " 'please transfer $50 to my checking account from credit',\n", " 'put $40 from account a to b',\n", " 'send 100 dollars between bank of the west and bank of america acccounts',\n", " 'transfer $40 from account a to b',\n", " 'time to move some cash from one account to another',\n", " 'i need to transfer from one account to my second one',\n", " 'need to transfer from one account to my other one',\n", " 'transfer $20000 from my savings account to checking account',\n", " 'please transfer $100 from my checking to my savings account',\n", " 'send 20 dollars from savings to checking',\n", " 'please transfer $x from checking to saving',\n", " 'help me move my money please',\n", " 'please send ten dollars from bank of america to capital one',\n", " 'i need my money to be moved',\n", " 'put $20000 into my checking account from my savings account',\n", " 'i want seventy bucks transferred from b of a to chase',\n", " 'send money from one account to another',\n", " 'please transfer 100 dollars between my amazon payments and savings accounts',\n", " 'could you tell me what my checking account balance is',\n", " 'do i have enough in my chase account for a plane ticket',\n", " \"what's my current bank savings\",\n", " 'i wish to know the balance of my bank of american account',\n", " \"i'd like to know the balance of my bank of american account\",\n", " \"what's my total net worth in all of my bank accounts\",\n", " 'how much money do i have in all of my accounts combined',\n", " 'how much is left of mastercard',\n", " 'check chase bank for my checking balance',\n", " 'i want to get a new shirt; will the money in my td ameritrade account cover '\n", " 'it',\n", " 'what is my bank balance for all accounts',\n", " 'i wish to know the balance of my bank of american account now',\n", " 'how much do i have in savings',\n", " 'do i have any cash left',\n", " 'what is my balance',\n", " \"what's my account balance\",\n", " 'please let me know what my current bank balance is',\n", " 'how much money do i have in checking',\n", " 'do you know how much i have in checking',\n", " 'what is my bank balance',\n", " 'i am going to need a block put on my chase account right away',\n", " \"i'd like a block on my charles schwab account immediately\",\n", " 'please block my chase account right away',\n", " 'can you notify the bank to put a stop on my account',\n", " 'dont allow any action on my account',\n", " 'do you mind putting a stop on my bank account',\n", " 'place a stop on my main account for me please',\n", " 'put a stop on my deposit account',\n", " 'please put a block on my td ameritrade account now',\n", " 'please freeze my bank account',\n", " 'i need my account frozen',\n", " 'can you freeze my account',\n", " 'close out my account',\n", " 'turn off my account',\n", " 'help me freeze my bank account, please',\n", " 'can you please put a block on my chase account quickly',\n", " 'block my monkey market right now',\n", " 'add a block to my capital one bank account so it cannot be used any more',\n", " 'i would like a block put on my chase account asap',\n", " 'can you please freeze my bank account',\n", " 'looking at january, show all wine purchases',\n", " 'please show me my recent transaction',\n", " 'i want to pay my amazon credit card but i need to know the last few '\n", " 'transactions',\n", " 'may i get all of the food transactions that were made last month',\n", " 'can you show me transactions related to utilities',\n", " \"i'd like to see last week's atm transactions\",\n", " 'what did i spend at target on my barclays card last month',\n", " 'let me check my transaction on my citi card',\n", " 'please tell me all of my recent transactions',\n", " 'let me check my transaction for first bank card',\n", " 'please give me my last ten debit card transactions in the month of december',\n", " 'whats my recent transactions on my card',\n", " 'help me get access to my recent transaction history',\n", " \"show me yesterday's last transaction\",\n", " 'bring up all purchases from target',\n", " \"before i pay my capital one, what are the most recent transactions i've made\",\n", " 'what amount did i spend for food on chase visa on current bill',\n", " 'show me my transactions on mcdonalds',\n", " \"what's the last transaction i made yesterday\",\n", " 'bring up my most recent purchases',\n", " 'i need to pay my cable bill',\n", " 'can anyone help me pay my car bill',\n", " 'i need to pay my mortgage',\n", " 'pay electric',\n", " 'i want to pay my internet bill',\n", " 'can you assist me in paying my electric bill',\n", " 'please go ahead and make my student loan payment',\n", " 'pay the electric bill',\n", " 'pay my water bill from my checking account',\n", " 'how can i pay my bill',\n", " 'help me pay my cable bill',\n", " 'pay the cable bill with my visa card',\n", " 'i need to pay my bill',\n", " 'pay my water bill with my charles schwab account',\n", " 'my water bill is due, pay it immediately',\n", " 'schedule a gas bill payment',\n", " 'pay my mortgage from my checkings accounts',\n", " \"i'd like to pay my bill\",\n", " \"i need to pay this month's tv subscription fee\",\n", " 'can you help me pay my phone bill',\n", " 'what is my current credit score',\n", " 'how do i locate my current credit score',\n", " 'would you tell me my credit score',\n", " 'is my credit score high',\n", " 'verify with me my credit score',\n", " 'how do i look up my credit score',\n", " 'i need my credit score',\n", " 'show me my credit score please',\n", " 'i want my credit score',\n", " 'how to locate my credit score',\n", " 'tell me my credit rating',\n", " 'how can i find out my credit score',\n", " 'provide me with my credit score',\n", " 'clue me in on my credit score',\n", " 'please look up my credit score',\n", " 'find my credit score',\n", " 'i wanna know my credit score',\n", " 'inform me of my credit score',\n", " 'tell me the steps to getting my credit score',\n", " 'where can i check my credit score',\n", " 'is my at&t bill do soon',\n", " 'what is the latest date that i can pay my direct tv bill this month',\n", " 'when do i pay the utilities',\n", " 'what date do i have to pay my bill',\n", " 'can you tell me the date my credit card is due',\n", " 'when is my visa due',\n", " 'do i pay my rent this week',\n", " 'when do i need to pay the water bil',\n", " 'how do i check when my mortgage is next up for payment',\n", " 'give me the date my bill is due',\n", " 'what time do i have to pay z bill',\n", " 'when is my chase visa due',\n", " 'when is my xfinity bill due',\n", " 'how much time left to pay my bill',\n", " 'when is the bill due',\n", " \"what's the due date for the renting bill\",\n", " 'how long do i have left to pay for my chase credit card',\n", " 'what is the due date for my metronorth monthy pass',\n", " 'what day is the z bill due',\n", " 'can you tell me when my electric bill is due',\n", " 'what steps do i take if there is a transaction that i do not recognize on my '\n", " 'navy federal credit union account',\n", " 'my account has a fraudulent transaction i think',\n", " \"it seems that there's fraudulent activity on my card i'd like to file a \"\n", " 'report',\n", " 'there are some questionable charges on my card',\n", " 'can you report credit card fraud for me',\n", " 'i got to report fraudulent activity on my credit card',\n", " 'looks like someone made an unauthorized charge to nike on my account',\n", " 'i have detected fraudulent activity on my account',\n", " \"i have charges on my amex card i didn't make\",\n", " 'there seems to be fraudulent activity',\n", " 'please report information about activity on my credit card',\n", " 'discover card reporting fraud',\n", " \"my card has purchases i don't recognize\",\n", " 'i need to know how to report fraud on my discover card',\n", " 'there is a fraudulent charge for paypal on my bank account',\n", " 'i may have a fraudulent transaction',\n", " 'it looks like someone made an unauthorized amazon purchase on my account',\n", " \"there's fraudulent transaction going on\",\n", " \"i'm reporting fraudelent activity on my card\",\n", " 'help me figure out where this fraudulent transaction from google came from '\n", " 'on my account',\n", " 'how do i go about setting up direct deposit',\n", " 'how do i set up instant paycheck',\n", " 'i want to set up direct deposit for my paycheck, what do i need to do',\n", " 'can i get paychecks directly deposited to my bank of america account',\n", " 'i want to set direct deposit',\n", " 'can you show me how to set up direct deposit for my paycheck to my first '\n", " 'hawaiian bank account',\n", " \"what's needed to direct deposit my paycheck\",\n", " 'info on setting up direct deposit',\n", " 'how can i turn on direct deposit',\n", " 'help me set up direct deposit to my bank of hawaii checking account',\n", " 'how do i set up direct deposit to my chase account',\n", " 'how do i get my paycheck direct deposited to my chase account',\n", " 'what do i do to have my paycheck deposited directly in my account',\n", " 'direct deposit information',\n", " 'let me set up direct deposit for this',\n", " 'info on direct deposit set-up',\n", " 'i want my paycheck to go directly to my bank account',\n", " 'tell me how to set up a direct deposit',\n", " 'how can i set up a direct deposit with my checking account',\n", " 'i need to set up a direct deposit',\n", " \"my card didn't go through when i was buying a case of water at walmart\",\n", " 'my card was declined yesterday, why',\n", " 'i wish to know why my card was declined yesterday',\n", " 'let me know why my card got declined the other day',\n", " 'why was my card declined for my monthly netflix subscription payment',\n", " 'buying qtips today, my card got declined at walmart',\n", " 'i tried to make a purchase yesterday but my card was declined why',\n", " 'why was my card declined yesterday',\n", " \"why wouldn't nordstrom accept my card\",\n", " 'i was at walmart when my card was declined i was only trying to buy some '\n", " 'candy',\n", " 'where can i find out why my card was recently declined at amazoncom',\n", " 'i went to target to buy a mug but my card did not work',\n", " \"i couldn't buy a mug from target because my card got declined\",\n", " \"why did macy's decline my card\",\n", " 'i was at home depot trying to buy plants and my card got declined',\n", " 'at target trying to buy a mug and my card was declined',\n", " 'my card declined yesterday and i want to know why',\n", " 'find out why my card was declined',\n", " \"i need to know why my card was just declined at walgreen's\",\n", " 'why did my card get declined at the dentist office']\n" ] } ], "source": [ "from itertools import chain\n", "num_samples = 20\n", "\n", "scenario_references_unclassified = [random.sample(references, num_samples) for references in train_by_intent.values()]\n", "scenario_references_unclassified = list(chain.from_iterable(scenario_references_unclassified))\n", "pprint(scenario_references_unclassified)" ] }, { "cell_type": "markdown", "id": "1b5331f5", "metadata": {}, "source": [ "#### Classifying these references using LLM" ] }, { "cell_type": "code", "execution_count": 25, "id": "5d17273d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'balance': ['could you tell me what my checking account balance is',\n", " 'do i have enough in my chase account for a plane ticket',\n", " \"what's my current bank savings\",\n", " 'i wish to know the balance of my bank of american account',\n", " \"i'd like to know the balance of my bank of american account\",\n", " \"what's my total net worth in all of my bank accounts\",\n", " 'how much money do i have in all of my accounts combined',\n", " 'how much is left of mastercard',\n", " 'check chase bank for my checking balance',\n", " 'i want to get a new shirt; will the money in my td ameritrade '\n", " 'account cover it',\n", " 'what is my bank balance for all accounts',\n", " 'i wish to know the balance of my bank of american account now',\n", " 'how much do i have in savings',\n", " 'do i have any cash left',\n", " 'what is my balance',\n", " \"what's my account balance\",\n", " 'please let me know what my current bank balance is',\n", " 'how much money do i have in checking',\n", " 'do you know how much i have in checking',\n", " 'what is my bank balance'],\n", " 'bill_due': ['is my at&t bill do soon',\n", " 'what is the latest date that i can pay my direct tv bill this '\n", " 'month',\n", " 'when do i pay the utilities',\n", " 'what date do i have to pay my bill',\n", " 'can you tell me the date my credit card is due',\n", " 'when is my visa due',\n", " 'do i pay my rent this week',\n", " 'when do i need to pay the water bil',\n", " 'how do i check when my mortgage is next up for payment',\n", " 'give me the date my bill is due',\n", " 'what time do i have to pay z bill',\n", " 'when is my chase visa due',\n", " 'when is my xfinity bill due',\n", " 'how much time left to pay my bill',\n", " 'when is the bill due',\n", " \"what's the due date for the renting bill\",\n", " 'how long do i have left to pay for my chase credit card',\n", " 'what is the due date for my metronorth monthy pass',\n", " 'what day is the z bill due',\n", " 'can you tell me when my electric bill is due'],\n", " 'card_declined': [\"my card didn't go through when i was buying a case of \"\n", " 'water at walmart',\n", " 'my card was declined yesterday, why',\n", " 'i wish to know why my card was declined yesterday',\n", " 'let me know why my card got declined the other day',\n", " 'why was my card declined for my monthly netflix '\n", " 'subscription payment',\n", " 'buying qtips today, my card got declined at walmart',\n", " 'i tried to make a purchase yesterday but my card was '\n", " 'declined why',\n", " 'why was my card declined yesterday',\n", " \"why wouldn't nordstrom accept my card\",\n", " 'i was at walmart when my card was declined i was only '\n", " 'trying to buy some candy',\n", " 'where can i find out why my card was recently declined at '\n", " 'amazoncom',\n", " 'i went to target to buy a mug but my card did not work',\n", " \"i couldn't buy a mug from target because my card got \"\n", " 'declined',\n", " \"why did macy's decline my card\",\n", " 'i was at home depot trying to buy plants and my card got '\n", " 'declined',\n", " 'at target trying to buy a mug and my card was declined',\n", " 'my card declined yesterday and i want to know why',\n", " 'find out why my card was declined',\n", " \"i need to know why my card was just declined at walgreen's\",\n", " 'why did my card get declined at the dentist office'],\n", " 'credit_score': ['what is my current credit score',\n", " 'how do i locate my current credit score',\n", " 'would you tell me my credit score',\n", " 'is my credit score high',\n", " 'verify with me my credit score',\n", " 'how do i look up my credit score',\n", " 'i need my credit score',\n", " 'show me my credit score please',\n", " 'i want my credit score',\n", " 'how to locate my credit score',\n", " 'tell me my credit rating',\n", " 'how can i find out my credit score',\n", " 'provide me with my credit score',\n", " 'clue me in on my credit score',\n", " 'please look up my credit score',\n", " 'find my credit score',\n", " 'i wanna know my credit score',\n", " 'inform me of my credit score',\n", " 'tell me the steps to getting my credit score',\n", " 'where can i check my credit score'],\n", " 'direct_deposit': ['how do i go about setting up direct deposit',\n", " 'how do i set up instant paycheck',\n", " 'i want to set up direct deposit for my paycheck, what do '\n", " 'i need to do',\n", " 'can i get paychecks directly deposited to my bank of '\n", " 'america account',\n", " 'i want to set direct deposit',\n", " 'can you show me how to set up direct deposit for my '\n", " 'paycheck to my first hawaiian bank account',\n", " \"what's needed to direct deposit my paycheck\",\n", " 'info on setting up direct deposit',\n", " 'how can i turn on direct deposit',\n", " 'help me set up direct deposit to my bank of hawaii '\n", " 'checking account',\n", " 'how do i set up direct deposit to my chase account',\n", " 'how do i get my paycheck direct deposited to my chase '\n", " 'account',\n", " 'what do i do to have my paycheck deposited directly in my '\n", " 'account',\n", " 'direct deposit information',\n", " 'let me set up direct deposit for this',\n", " 'info on direct deposit set-up',\n", " 'i want my paycheck to go directly to my bank account',\n", " 'tell me how to set up a direct deposit',\n", " 'how can i set up a direct deposit with my checking '\n", " 'account',\n", " 'i need to set up a direct deposit'],\n", " 'freeze_account': ['i am going to need a block put on my chase account right '\n", " 'away',\n", " \"i'd like a block on my charles schwab account immediately\",\n", " 'please block my chase account right away',\n", " 'can you notify the bank to put a stop on my account',\n", " 'dont allow any action on my account',\n", " 'do you mind putting a stop on my bank account',\n", " 'place a stop on my main account for me please',\n", " 'put a stop on my deposit account',\n", " 'please put a block on my td ameritrade account now',\n", " 'please freeze my bank account',\n", " 'i need my account frozen',\n", " 'can you freeze my account',\n", " 'close out my account',\n", " 'turn off my account',\n", " 'help me freeze my bank account, please',\n", " 'can you please put a block on my chase account quickly',\n", " 'block my monkey market right now',\n", " 'add a block to my capital one bank account so it cannot '\n", " 'be used any more',\n", " 'i would like a block put on my chase account asap',\n", " 'can you please freeze my bank account'],\n", " 'pay_bill': ['i need to pay my cable bill',\n", " 'can anyone help me pay my car bill',\n", " 'i need to pay my mortgage',\n", " 'pay electric',\n", " 'i want to pay my internet bill',\n", " 'can you assist me in paying my electric bill',\n", " 'please go ahead and make my student loan payment',\n", " 'pay the electric bill',\n", " 'pay my water bill from my checking account',\n", " 'how can i pay my bill',\n", " 'help me pay my cable bill',\n", " 'pay the cable bill with my visa card',\n", " 'i need to pay my bill',\n", " 'pay my water bill with my charles schwab account',\n", " 'my water bill is due, pay it immediately',\n", " 'schedule a gas bill payment',\n", " 'pay my mortgage from my checkings accounts',\n", " \"i'd like to pay my bill\",\n", " \"i need to pay this month's tv subscription fee\",\n", " 'can you help me pay my phone bill'],\n", " 'report_fraud': ['what steps do i take if there is a transaction that i do '\n", " 'not recognize on my navy federal credit union account',\n", " 'my account has a fraudulent transaction i think',\n", " \"it seems that there's fraudulent activity on my card i'd \"\n", " 'like to file a report',\n", " 'there are some questionable charges on my card',\n", " 'can you report credit card fraud for me',\n", " 'i got to report fraudulent activity on my credit card',\n", " 'looks like someone made an unauthorized charge to nike on '\n", " 'my account',\n", " 'i have detected fraudulent activity on my account',\n", " \"i have charges on my amex card i didn't make\",\n", " 'there seems to be fraudulent activity',\n", " 'discover card reporting fraud',\n", " \"my card has purchases i don't recognize\",\n", " 'i need to know how to report fraud on my discover card',\n", " 'there is a fraudulent charge for paypal on my bank account',\n", " 'i may have a fraudulent transaction',\n", " 'it looks like someone made an unauthorized amazon purchase '\n", " 'on my account',\n", " \"there's fraudulent transaction going on\",\n", " \"i'm reporting fraudelent activity on my card\",\n", " 'help me figure out where this fraudulent transaction from '\n", " 'google came from on my account'],\n", " 'transactions': ['looking at january, show all wine purchases',\n", " 'please show me my recent transaction',\n", " 'i want to pay my amazon credit card but i need to know the '\n", " 'last few transactions',\n", " 'may i get all of the food transactions that were made last '\n", " 'month',\n", " 'can you show me transactions related to utilities',\n", " \"i'd like to see last week's atm transactions\",\n", " 'what did i spend at target on my barclays card last month',\n", " 'let me check my transaction on my citi card',\n", " 'please tell me all of my recent transactions',\n", " 'let me check my transaction for first bank card',\n", " 'please give me my last ten debit card transactions in the '\n", " 'month of december',\n", " 'whats my recent transactions on my card',\n", " 'help me get access to my recent transaction history',\n", " \"show me yesterday's last transaction\",\n", " 'bring up all purchases from target',\n", " 'before i pay my capital one, what are the most recent '\n", " \"transactions i've made\",\n", " 'what amount did i spend for food on chase visa on current '\n", " 'bill',\n", " 'show me my transactions on mcdonalds',\n", " \"what's the last transaction i made yesterday\",\n", " 'bring up my most recent purchases',\n", " 'please report information about activity on my credit card'],\n", " 'transfer': ['transfer 200 dollars from paypal to savings',\n", " 'send money to another account',\n", " 'please transfer $50 to my checking account from credit',\n", " 'put $40 from account a to b',\n", " 'send 100 dollars between bank of the west and bank of america '\n", " 'acccounts',\n", " 'transfer $40 from account a to b',\n", " 'time to move some cash from one account to another',\n", " 'i need to transfer from one account to my second one',\n", " 'need to transfer from one account to my other one',\n", " 'transfer $20000 from my savings account to checking account',\n", " 'please transfer $100 from my checking to my savings account',\n", " 'send 20 dollars from savings to checking',\n", " 'please transfer $x from checking to saving',\n", " 'help me move my money please',\n", " 'please send ten dollars from bank of america to capital one',\n", " 'i need my money to be moved',\n", " 'put $20000 into my checking account from my savings account',\n", " 'i want seventy bucks transferred from b of a to chase',\n", " 'send money from one account to another',\n", " 'please transfer 100 dollars between my amazon payments and '\n", " 'savings accounts']}\n" ] } ], "source": [ "# NBVAL_SKIP\n", "# Use the LLM to generate route classifications for unclassified samples (from production)\n", "classifier_prompt = f\"\"\"\n", "You are an intent classifier for a banking assistant.\n", "\n", "Choose exactly one intent label from this list:\n", "{', '.join(intents)}\n", "\n", "Return only the intent label and nothing else.\n", "\"\"\".strip()\n", "\n", "LLM_MODEL = 'gpt-5.4-nano'\n", "MAX_CONCURRENCY = 20\n", "semaphore = asyncio.Semaphore(MAX_CONCURRENCY)\n", "\n", "async def classify_bootstrap_reference(reference, semaphore):\n", " async with semaphore:\n", " response = await openai_client.responses.create(\n", " model=LLM_MODEL,\n", " instructions=classifier_prompt,\n", " input=reference,\n", " reasoning={\"effort\": \"low\"},\n", " text={\"verbosity\": \"low\"},\n", " )\n", " return {\n", " \"text\": reference,\n", " \"predicted_intent\": response.output_text.strip(),\n", " }\n", "\n", "results = await asyncio.gather(*[\n", " classify_bootstrap_reference(reference, semaphore)\n", " for reference in scenario_references_unclassified\n", "])\n", "\n", "scenario_predictions = defaultdict(list)\n", "for item in results:\n", " scenario_predictions[item[\"predicted_intent\"]].append(item[\"text\"])\n", "\n", "pprint(dict(scenario_predictions))" ] }, { "cell_type": "code", "execution_count": null, "id": "e40127cd", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "scenario_routes_bootstrapped = [Route(name=intent, references=references) \n", " for intent, references in scenario_predictions.items()\n", " ]" ] }, { "cell_type": "markdown", "id": "6bdd8565", "metadata": {}, "source": [ "These predictions can be used to create routes for the semantic router." ] }, { "cell_type": "code", "execution_count": null, "id": "c39145e7", "metadata": {}, "outputs": [], "source": [ "# NBVAL_SKIP\n", "scenario_router_bootstrapped = SemanticRouter(\n", " name='bootstrapped-banking-router',\n", " routes=scenario_routes_bootstrapped,\n", " vectorizer=HFTextVectorizer(),\n", " redis_client=client,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "7631aae9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy on basic router (with LLM-bootstrapped routes): 90.33%\n" ] } ], "source": [ "# NBVAL_SKIP\n", "scenario_router_bootstrapped_metrics, scenario_router_bootstrapped_per_test_case = evaluate_classification(\n", " lambda text: get_router_classification(scenario_router_bootstrapped, text)\n", ")\n", "\n", "print(f\"Accuracy on basic router (with LLM-bootstrapped routes): {100*scenario_router_bootstrapped_metrics['accuracy']:.2f}%\")" ] }, { "cell_type": "markdown", "id": "09c65d9a", "metadata": {}, "source": [ "Using an LLM to create a bootstrapped dataset for unclassified route references, we managed to achieve pretty decent routing accuracy." ] }, { "cell_type": "markdown", "id": "ab9c6367", "metadata": {}, "source": [ "The strategies we explored in this section improve the route set itself; we now turn to improving production accuracy at inference time." ] }, { "cell_type": "markdown", "id": "c3fd13c3", "metadata": {}, "source": [ "## Part III: Improving Production Accuracy with Fallback" ] }, { "cell_type": "markdown", "id": "c0975458", "metadata": {}, "source": [ "### Strategy 4: Using LLM as a fallback classifier" ] }, { "cell_type": "markdown", "id": "0842c1e2", "metadata": {}, "source": [ "While the semantic router is fast, it starts to degrade in accuracy when the distance between the input query and the closest route gets bigger. This can happen if there are overlaps in the semantic neighbourhood of each route, or if the semantic space of each route is ill-defined.\n", "\n", "On the other hand, an LLM is a better classifier (in terms of accuracy) but suffers from higher latency, which is problematic for each conversation turn in the chatbot.\n", "\n", "To get the best of both worlds, a hybrid model can be built where a LLM classifier is used as a fallback classifier for cases when the routing performance of the semantic router starts to degrade." ] }, { "cell_type": "markdown", "id": "95a5d670", "metadata": {}, "source": [ "#### Pure-LLM classifier" ] }, { "cell_type": "code", "execution_count": 29, "id": "2d5420ce", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'num_examples': 200,\n", " 'accuracy': 0.99,\n", " 'avg_latency': 958.43,\n", " 'p95_latency': 1334.37,\n", " 'p99_latency': 2882.83}" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# NBVAL_SKIP\n", "MAX_CONCURRENCY = 20\n", "\n", "semaphore = asyncio.Semaphore(MAX_CONCURRENCY)\n", "\n", "async def classify_example(example):\n", " async with semaphore:\n", " text = example['text']\n", " with profile_block('llm_call') as timer:\n", " response = await openai_client.responses.create(\n", " model=LLM_MODEL,\n", " input=[\n", " {'role': 'system', 'content': classifier_prompt},\n", " {'role': 'user', 'content': text},\n", " ],\n", " )\n", " predicted_intent = response.output_text.strip()\n", " true_intent = intent_names[example['intent']]\n", " return {\n", " 'is_correct': predicted_intent == true_intent,\n", " 'latency_ms': timer.elapsed * 1000,\n", " }\n", "\n", "async def evaluate_llm(split):\n", " results = await asyncio.gather(*[\n", " classify_example(example)\n", " for example in data_subset[split]\n", " ])\n", "\n", " num_correct = sum(result['is_correct'] for result in results)\n", " latencies_ms = [result['latency_ms'] for result in results]\n", " num_test_examples = len(data_subset[split])\n", "\n", " return {\n", " 'num_examples': num_test_examples,\n", " 'accuracy': round(num_correct / num_test_examples, 4),\n", " 'avg_latency': round(float(np.mean(latencies_ms)), 2),\n", " 'p95_latency': round(float(np.percentile(latencies_ms, 95)), 2),\n", " 'p99_latency': round(float(np.percentile(latencies_ms, 99)), 2),\n", " }\n", "\n", "await evaluate_llm(split=\"validation\")" ] }, { "cell_type": "markdown", "id": "ed6a7700", "metadata": {}, "source": [ "The LLM is a strong classifier, albeit having high latency per classification. " ] }, { "cell_type": "markdown", "id": "211a2de3", "metadata": {}, "source": [ "#### Semantic Router-only classification" ] }, { "cell_type": "markdown", "id": "4b5ec393", "metadata": {}, "source": [ "We create a semantic router with all examples from the training set." ] }, { "cell_type": "code", "execution_count": 30, "id": "b9ace6c2", "metadata": {}, "outputs": [], "source": [ "baseline_routes = [\n", " Route(\n", " name=intent,\n", " references=texts,\n", " )\n", " for intent, texts in sorted(train_by_intent.items())\n", "]\n", "\n", "baseline_router_full_train_set = SemanticRouter(\n", " name='banking-router-full-train-set',\n", " routes=baseline_routes,\n", " vectorizer=HFTextVectorizer(),\n", " redis_client=client,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 31, "id": "f6547bbc", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'num_examples': 200,\n", " 'accuracy': 0.905,\n", " 'avg_latency': 14.15,\n", " 'p95_latency': 24.53,\n", " 'p99_latency': 36.37,\n", " 'avg_distance': 0.3709}" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "baseline_router_full_train_set_metrics, baseline_router_full_train_set_per_test_case = evaluate_classification(\n", " lambda text: get_router_classification(baseline_router_full_train_set, text), split=\"validation\"\n", ")\n", "baseline_router_full_train_set_metrics" ] }, { "cell_type": "markdown", "id": "44d6b1fe", "metadata": {}, "source": [ "The semantic router-only classifier has a lower accuracy on the validation set, but a much better latency per classification." ] }, { "cell_type": "markdown", "id": "7d7282dd", "metadata": {}, "source": [ "#### Hybrid approach" ] }, { "cell_type": "markdown", "id": "b5022206", "metadata": {}, "source": [ "A good middle ground between this accuracy-latency tradeoff can be to use a hybrid approach. \n", "\n", "Each intent classification task in the semantic router has a routing distance associated with it, which represents the distance between the input query and the route it is classified as. Generally, the semantic router tends to be more reliable at the lower routing distances (the idea is that queries with shorter routing distance tend to be more similar in semantic space to the representative references of the route, so the semantic router is able to classify the query with a higher level of confidence). At larger distances, the semantic router might be more prone to routing mistakes (perhaps due to overlap of semantic space of the defined routes). \n", "\n", "For this hybrid approach to work, we must pinpoint a routing distance beyond which the semantic router starts to degrade in performance and becomes less reliable. Using this cutoff, we can then form a decision tree where:\n", "- Queries with routing distance below cutoff: Use the semantic router for classification, since it is accurate and reliable.\n", "- Queries with routing distance above cutoff: Fall back to the LLM for classification, since the semantic router is less reliable. This happens regardless of the classification from the semantic router.\n", "\n", "This hybrid approach allows us to reduce the cost and latency of LLM inference, since a subset of queries are classified using the semantic router, while at the same time having a high level of classification accuracy, since the other subset is classified using the powerful LLM." ] }, { "cell_type": "markdown", "id": "3f77da05", "metadata": {}, "source": [ "##### Finding a cutoff distance" ] }, { "cell_type": "markdown", "id": "d3f9f96c", "metadata": {}, "source": [ "To find a cutoff distance for our hybrid solution, we use a hold-out validation set (to prevent data leakage when measuring performance):\n", "1. Measure the semantic distance between the input query and its matched route for every example in the validation set\n", "2. Sort the results by distance to pinpoint where routing errors start to emerge\n", "3. Choose a cutoff based on the acceptable accuracy" ] }, { "cell_type": "code", "execution_count": 32, "id": "3e254ddd", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
texttrue_intentpredicted_intentdistanceis_correctcumulative_accuracy
0what's my current credit scorecredit_scorecredit_score0.215868True1.000000
1relate to me what my credit score iscredit_scorecredit_score0.220622True1.000000
2can you tell me my credit ratingcredit_scorecredit_score0.221561True1.000000
3how do i go about setting up paycheck direct d...direct_depositdirect_deposit0.228010True1.000000
4where's my credit scorecredit_scorecredit_score0.228468True1.000000
.....................
195can i afford a new tv from my savings accountbalancebalance0.461696True0.908163
196show all purchases of video gamestransactionstransactions0.463341True0.908629
197list all purchases of video gamestransactionstransactions0.466707True0.909091
198show me all video games purchasedtransactionstransactions0.473267True0.909548
199can i get beer within my deposit accountbalanceNaNNaNFalse0.905000
\n", "

200 rows × 6 columns

\n", "
" ], "text/plain": [ " text true_intent \\\n", "0 what's my current credit score credit_score \n", "1 relate to me what my credit score is credit_score \n", "2 can you tell me my credit rating credit_score \n", "3 how do i go about setting up paycheck direct d... direct_deposit \n", "4 where's my credit score credit_score \n", ".. ... ... \n", "195 can i afford a new tv from my savings account balance \n", "196 show all purchases of video games transactions \n", "197 list all purchases of video games transactions \n", "198 show me all video games purchased transactions \n", "199 can i get beer within my deposit account balance \n", "\n", " predicted_intent distance is_correct cumulative_accuracy \n", "0 credit_score 0.215868 True 1.000000 \n", "1 credit_score 0.220622 True 1.000000 \n", "2 credit_score 0.221561 True 1.000000 \n", "3 direct_deposit 0.228010 True 1.000000 \n", "4 credit_score 0.228468 True 1.000000 \n", ".. ... ... ... ... \n", "195 balance 0.461696 True 0.908163 \n", "196 transactions 0.463341 True 0.908629 \n", "197 transactions 0.466707 True 0.909091 \n", "198 transactions 0.473267 True 0.909548 \n", "199 NaN NaN False 0.905000 \n", "\n", "[200 rows x 6 columns]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Sort the route matches by distance\n", "import pandas as pd\n", "distance_df = pd.DataFrame(baseline_router_full_train_set_per_test_case['per_test_case'])\\\n", " .sort_values(by='distance').reset_index(drop=True)\n", "distance_df['cumulative_accuracy'] = distance_df['is_correct'].cumsum() / (distance_df.index + 1)\n", "distance_df" ] }, { "cell_type": "code", "execution_count": 33, "id": "59e1dff4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
texttrue_intentpredicted_intentdistanceis_correctcumulative_accuracy
79tell me how i know when to pay my chase billbill_duetransactions0.366768False0.987500
91can you assist me on my fradulent activity on ...report_fraudfreeze_account0.376051False0.978261
93transfer 50 dollars from my checking account t...transferdirect_deposit0.377061False0.968085
99pay my monthy mortgage paymentpay_billbill_due0.380842False0.960000
118how do i find out when my visa bill is duebill_duebalance0.399812False0.957983
119i need 200 dollars transferred from my long is...transferbalance0.400601False0.950000
122submit full payment to chase for my visa billpay_billbill_due0.402301False0.943089
135make a quarterly payment on my life insurance ...pay_billbill_due0.408546False0.941176
144make an eft to my savings from my checking acc...transferfreeze_account0.413562False0.937931
145pay my water bill from my chase account pleasepay_billtransactions0.413982False0.931507
146use my savings account to pay xfinitypay_billbill_due0.414301False0.925170
156please immediately block my navy federal credi...freeze_accountdirect_deposit0.420984False0.923567
159transfer ten dollars from my wells fargo accou...transferdirect_deposit0.423809False0.918750
163put one hundred and seventy five bucks in my w...transferfreeze_account0.426543False0.914634
167please let me review all items on my mastercar...transactionsbill_due0.431640False0.910714
187send from my bbc money market to my t rowe pri...transferpay_bill0.450126False0.914894
188do i have more than $100 in my pnc accountbalancecard_declined0.452765False0.910053
193is my credit good enough to get a new cardcredit_scorecard_declined0.459514False0.907216
199can i get beer within my deposit accountbalanceNaNNaNFalse0.905000
\n", "
" ], "text/plain": [ " text true_intent \\\n", "79 tell me how i know when to pay my chase bill bill_due \n", "91 can you assist me on my fradulent activity on ... report_fraud \n", "93 transfer 50 dollars from my checking account t... transfer \n", "99 pay my monthy mortgage payment pay_bill \n", "118 how do i find out when my visa bill is due bill_due \n", "119 i need 200 dollars transferred from my long is... transfer \n", "122 submit full payment to chase for my visa bill pay_bill \n", "135 make a quarterly payment on my life insurance ... pay_bill \n", "144 make an eft to my savings from my checking acc... transfer \n", "145 pay my water bill from my chase account please pay_bill \n", "146 use my savings account to pay xfinity pay_bill \n", "156 please immediately block my navy federal credi... freeze_account \n", "159 transfer ten dollars from my wells fargo accou... transfer \n", "163 put one hundred and seventy five bucks in my w... transfer \n", "167 please let me review all items on my mastercar... transactions \n", "187 send from my bbc money market to my t rowe pri... transfer \n", "188 do i have more than $100 in my pnc account balance \n", "193 is my credit good enough to get a new card credit_score \n", "199 can i get beer within my deposit account balance \n", "\n", " predicted_intent distance is_correct cumulative_accuracy \n", "79 transactions 0.366768 False 0.987500 \n", "91 freeze_account 0.376051 False 0.978261 \n", "93 direct_deposit 0.377061 False 0.968085 \n", "99 bill_due 0.380842 False 0.960000 \n", "118 balance 0.399812 False 0.957983 \n", "119 balance 0.400601 False 0.950000 \n", "122 bill_due 0.402301 False 0.943089 \n", "135 bill_due 0.408546 False 0.941176 \n", "144 freeze_account 0.413562 False 0.937931 \n", "145 transactions 0.413982 False 0.931507 \n", "146 bill_due 0.414301 False 0.925170 \n", "156 direct_deposit 0.420984 False 0.923567 \n", "159 direct_deposit 0.423809 False 0.918750 \n", "163 freeze_account 0.426543 False 0.914634 \n", "167 bill_due 0.431640 False 0.910714 \n", "187 pay_bill 0.450126 False 0.914894 \n", "188 card_declined 0.452765 False 0.910053 \n", "193 card_declined 0.459514 False 0.907216 \n", "199 NaN NaN False 0.905000 " ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Find the candidate distances for the cutoff threshold. These candidates are picked based on the cumulative accuracy.\n", "# Cumulative accuracy measures the empirical accuracy of the router on examples with distance <= the corresponding cutoff.\n", "distance_df.loc[~distance_df['is_correct']].drop_duplicates(subset='cumulative_accuracy', keep='first')" ] }, { "cell_type": "markdown", "id": "9484e19d", "metadata": {}, "source": [ "We choose a cutoff of 0.40, which gives roughly 95% router accuracy on lower-distance examples. Based on the validation set here, around 60% of queries (120 out of 200) rely on the semantic router for classification. The remaining 40% fall back to the LLM for classification.\n", "\n", "Remember there is an accuracy-latency tradeoff here, so the cutoff distance can be adjusted based on the latency and accuracy accepted by our chatbot." ] }, { "cell_type": "markdown", "id": "d6d81141", "metadata": {}, "source": [ "##### Creating a hybrid classifier" ] }, { "cell_type": "code", "execution_count": 34, "id": "a9c5af4e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'num_examples': 200,\n", " 'accuracy': 0.96,\n", " 'avg_latency': 433.28,\n", " 'p95_latency': 1224.48,\n", " 'p99_latency': 1693.6,\n", " 'avg_distance': 0.3709}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# NBVAL_SKIP\n", "DISTANCE_CUTOFF = 0.40\n", "\n", "sync_openai_client = OpenAI(api_key=OPENAI_API_KEY)\n", "\n", "def hybrid_classifier(text):\n", " router_result = get_router_classification(baseline_router_full_train_set, text)\n", " router_distance = router_result['distance']\n", "\n", " # Use the router result if the router_distance <= cutoff. otherwise, fall back onto LLM\n", " if router_distance is not None and router_distance <= DISTANCE_CUTOFF:\n", " return router_result\n", "\n", " response = sync_openai_client.responses.create(\n", " model=LLM_MODEL,\n", " instructions=classifier_prompt,\n", " input=text,\n", " reasoning={\"effort\": \"low\"},\n", " text={\"verbosity\": \"low\"},\n", " )\n", "\n", " return {\n", " 'name': response.output_text.strip(),\n", " 'distance': router_distance,\n", " }\n", "\n", "hybrid_router_metrics, hybrid_router_per_test_case = evaluate_classification(\n", " hybrid_classifier,\n", " split='validation', \n", ")\n", "hybrid_router_metrics\n", "\n" ] }, { "cell_type": "markdown", "id": "f7f13d42", "metadata": {}, "source": [ "##### Performance comparison on test set" ] }, { "cell_type": "code", "execution_count": 35, "id": "59942d9d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Semantic router only classifier on test set: \n", " {'accuracy': 0.9267,\n", " 'avg_distance': 0.3596,\n", " 'avg_latency': 14.1,\n", " 'num_examples': 300,\n", " 'p95_latency': 25.23,\n", " 'p99_latency': 32.54}\n" ] } ], "source": [ "baseline_router_test, _ = evaluate_classification(\n", " lambda text: get_router_classification(baseline_router_full_train_set, text), split=\"test\"\n", ")\n", "print(f\"Semantic router only classifier on test set: \\n {pformat(baseline_router_test)}\")" ] }, { "cell_type": "code", "execution_count": 36, "id": "200381c3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "LLM classifier on test set: \n", " {'accuracy': 0.98,\n", " 'avg_latency': 1058.38,\n", " 'num_examples': 300,\n", " 'p95_latency': 1842.76,\n", " 'p99_latency': 3150.38}\n" ] } ], "source": [ "# NBVAL_SKIP\n", "llm_classifier_test = await evaluate_llm(split=\"test\")\n", "print(f\"LLM classifier on test set: \\n {pformat(llm_classifier_test)}\")" ] }, { "cell_type": "code", "execution_count": 37, "id": "3431d47c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hybrid classifier on test set: \n", " {'accuracy': 0.97,\n", " 'avg_distance': 0.3596,\n", " 'avg_latency': 404.65,\n", " 'num_examples': 300,\n", " 'p95_latency': 1276.82,\n", " 'p99_latency': 2547.15}\n" ] } ], "source": [ "# NBVAL_SKIP\n", "hybrid_classifier_test, _ = evaluate_classification(\n", " hybrid_classifier,\n", " split='test', \n", ")\n", "\n", "print(f\"Hybrid classifier on test set: \\n {pformat(hybrid_classifier_test)}\")" ] }, { "cell_type": "markdown", "id": "94baeba6", "metadata": {}, "source": [ "The hybrid approach finds a sweet spot between the semantic router-only approach and the LLM approach, in terms of accuracy and latency. While the p95 and p99 latency of the hybrid classifier is still rather high, the average latency has been reduced significantly." ] }, { "cell_type": "markdown", "id": "f8a1da62", "metadata": {}, "source": [ "## Conclusion" ] }, { "cell_type": "markdown", "id": "464b36c1", "metadata": {}, "source": [ "In this notebook, we explored the different methods to improve the accuracy of the semantic router. \n", "\n", "In Part I, we have seen how the routing accuracy varies with embedding model, so it is important to choose a good embedding model for our use case." ] }, { "cell_type": "markdown", "id": "767c1c9d", "metadata": {}, "source": [ "In Part II, we navigate practical cases where it is difficult to obtain high-quality routes and references. An LLM can be used as a useful upstream assistant to refine route definitions and identify representative references, helping generate a clean and well-curated set of routes that is more semantically separable. We explore two strategies that leverage the LLM for doing so:\n", "1. Using the LLM to generate synthetic data to augment the list of route references for each route\n", "2. Using the LLM as a temporary classifier to identify routes for unclassified references (obtained from production)\n", "\n", "These methods are useful in effectively applying the semantic router in production. Furthermore, this is a one-time upstream effort that invokes the LLM outside of production, so the cost of the LLM is only incurred once.\n", "\n", "However, it is not the case that routing performance definitely improves with more references. If a LLM is unavailable, more (user-generated) examples can actually hurt routing performance if these examples are repetitive, noisy, inconsistent, or wrongly labelled. When this happens, careful selection of representative references can help restore route quality.\n", "\n", "When it comes to the semantic router, quality matters more than quantity, so a small set of high-quality examples that define the semantic neighbourhood well usually outperforms a larger but messier set that introduces overlaps in semantic space." ] }, { "cell_type": "markdown", "id": "3e232176", "metadata": {}, "source": [ "In Part III, we also introduced a hybrid method that navigates the accuracy-latency tradeoff of an LLM-only and a semantic router-only approach, finding a sweet spot where the overall accuracy is good and latencies are reasonable. In this approach, we find that the routing performance of a semantic router degrades only at higher distances, allowing us to find a cutoff point where the semantic router becomes noticeably less reliable. Beyond this cutoff point, we fall back to the LLM for its superior but slower classification ability." ] }, { "cell_type": "markdown", "id": "6c459292", "metadata": {}, "source": [ "### Next Steps" ] }, { "cell_type": "markdown", "id": "2b13894d", "metadata": {}, "source": [ "Check out [this notebook](03_router_visualization.ipynb) where we discuss methods to visualize your semantic router and determine if your route references are of good quality." ] }, { "cell_type": "markdown", "id": "578b1e4d", "metadata": {}, "source": [ "## Cleanup" ] }, { "cell_type": "code", "execution_count": 44, "id": "82e50cba", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "client.flushall()" ] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.12.5)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.5" } }, "nbformat": 4, "nbformat_minor": 5 }