{ "cells": [ { "metadata": {}, "cell_type": "markdown", "source": [ "# Semantic Caching with LangCache\n", "\n", "LangCache is Redis' new managed service for semantic caching. Semantic caching is an intelligent caching strategy that stores and retrieves responses based on the meaning of queries rather than exact text matches. Unlike traditional caching that requires identical strings, semantic caching can return cached responses for questions that are semantically similar, even when phrased differently.\n", "\n", "## Semantic Caching vs. Traditional Caching vs. LLM Re-generation\n", "\n", "**Traditional caching** stores responses using exact query strings as keys:\n", "- **Fast retrieval** for identical queries\n", "- **Cache misses** for any variation in phrasing, even minor differences\n", "- **Low cache hit rates** in conversational applications where users rarely phrase questions identically\n", "\n", "**LLM re-generation** involves calling the language model for every query:\n", "- **Flexible** handling of any question variation\n", "- **High API costs** and latency for repeated similar questions\n", "\n", "**Semantic caching** uses vector similarity to match queries with cached responses:\n", "- **High cache hit rates** by matching semantically similar questions\n", "- **Cost reduction** by avoiding redundant LLM calls for similar queries\n", "- **Fast retrieval** through vector similarity search\n", "\n", "In this notebook, we'll implement semantic caching using LangCache to demonstrate how semantic similarity can improve cache hit rates compared to exact string matching." ] }, { "metadata": {}, "cell_type": "markdown", "source": "## Getting Started with LangCache" }, { "metadata": {}, "cell_type": "markdown", "source": [ "### Creating a Redis Database on Redis Cloud\n", "\n", "- Go to https//cloud.redis.io and create a new account if you don't have one yet.\n", "- Once logged in, click on the plus sign next to \"Databases\" on the sidebar:\n", "\n", "\"\"\n", "\n", "- Select the free option:\n", "\n", "\"\"\n", "\n", "- Select `AWS` as the vendor and `us-east-1` as the region:\n", "\n", "\"\"\n", "\n", "- Create the database and wait for it to be available. It should take less than a minute:\n", "\n", "\"\"\n", "\n", "- Once the database is available. Click on LangCache on the side bar:\n", "\n", "\"\"\n", "\n", "- Then, click on Quick Create to easily get started:\n", "\n", "\"\"\n", "\n", "- Finally, copy the credential variable to use them in the next steps of this recipe:\n", "\n", "\"\"" ] }, { "metadata": {}, "cell_type": "markdown", "source": "## Installing Dependencies" }, { "metadata": { "ExecuteTime": { "end_time": "2025-11-24T10:58:14.966956Z", "start_time": "2025-11-24T10:58:14.709686Z" } }, "cell_type": "code", "source": [ "%use ktor-client\n", "%use serialization\n", "%use coroutines\n", "\n", "import io.ktor.client.HttpClient\n", "import io.ktor.client.engine.cio.CIO\n", "import io.ktor.client.plugins.contentnegotiation.ContentNegotiation\n", "import io.ktor.serialization.kotlinx.json.json\n", "\n", "val client = HttpClient(CIO) {\n", " install(ContentNegotiation) {\n", " json(Json { ignoreUnknownKeys = true })\n", " }\n", "}" ], "outputs": [], "execution_count": 6 }, { "metadata": {}, "cell_type": "markdown", "source": [ "## Configuring LangCache\n", "\n", "### Access Configuration" ] }, { "metadata": { "ExecuteTime": { "end_time": "2025-11-24T10:58:17.013283Z", "start_time": "2025-11-24T10:58:16.981396Z" } }, "cell_type": "code", "source": [ "val apiKey = System.getenv(\"LANG_CACHE_API_KEY\")\n", "val cacheId = \"28e9625f77be4186b295ef6d3577c6d0\"\n", "val baseUrl = \"https://aws-us-east-1.langcache.redis.io/v1/caches/$cacheId\"" ], "outputs": [], "execution_count": 7 }, { "metadata": {}, "cell_type": "markdown", "source": "### Modeling the requests and responses from the API" }, { "metadata": { "ExecuteTime": { "end_time": "2025-11-24T10:58:19.573691Z", "start_time": "2025-11-24T10:58:19.398696Z" } }, "cell_type": "code", "source": [ "@Serializable\n", "data class CacheEntryRequest(\n", " val prompt: String,\n", " val response: String? = null\n", ")\n", "\n", "@Serializable\n", "data class CacheEntryResponse(\n", " val entryId: String\n", ")\n", "\n", "@Serializable\n", "data class CacheEntry(\n", " val id: String,\n", " val prompt: String,\n", " val response: String,\n", " val attributes: Map = emptyMap(),\n", " val similarity: Double? = null,\n", " @SerialName(\"search_strategy\")\n", " val searchStrategy: String? = null\n", ")\n", "\n", "@Serializable\n", "data class SearchResponse(\n", " val data: List\n", ")" ], "outputs": [], "execution_count": 8 }, { "metadata": {}, "cell_type": "markdown", "source": "### Storing in LangCache" }, { "metadata": { "ExecuteTime": { "end_time": "2025-11-24T10:26:16.657268Z", "start_time": "2025-11-24T10:26:15.476865Z" } }, "cell_type": "code", "source": [ "import io.ktor.client.call.*\n", "import io.ktor.client.request.*\n", "import io.ktor.http.*\n", "\n", "runBlocking {\n", " val saveResponse: CacheEntryResponse = client.post(\"$baseUrl/entries\") {\n", " header(\"Authorization\", \"Bearer $apiKey\")\n", " contentType(ContentType.Application.Json)\n", " setBody(CacheEntryRequest(\n", " prompt = \"How does semantic caching work?\",\n", " response = \"Semantic caching stores and retrieves data based on meaning, not exact matches.\"\n", " ))\n", " }.body()\n", "\n", " println(\"Save response: $saveResponse\")\n", "}" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Save response: CacheEntryResponse(entryId=fda1b671e21b06a0a957c04b1692ab90)\n" ] } ], "execution_count": 4 }, { "metadata": {}, "cell_type": "markdown", "source": "### Retrieving from LangCache" }, { "metadata": { "ExecuteTime": { "end_time": "2025-11-24T10:26:24.215885Z", "start_time": "2025-11-24T10:26:23.627109Z" } }, "cell_type": "code", "source": [ "import io.ktor.client.call.*\n", "import io.ktor.client.request.*\n", "import io.ktor.http.*\n", "\n", "runBlocking {\n", " val searchResponse: SearchResponse = client.post(\"$baseUrl/entries/search\") {\n", " header(\"Authorization\", \"Bearer $apiKey\")\n", " contentType(ContentType.Application.Json)\n", " setBody(CacheEntryRequest(prompt = \"What is semantic caching?\"))\n", " }.body()\n", "\n", " println(\"Search response: $searchResponse\")\n", "}" ], "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Search response: SearchResponse(data=[CacheEntry(id=fda1b671e21b06a0a957c04b1692ab90, prompt=How does semantic caching work?, response=Semantic caching stores and retrieves data based on meaning, not exact matches., attributes={}, similarity=0.9292393, searchStrategy=null)])\n" ] } ], "execution_count": 5 } ], "metadata": { "kernelspec": { "display_name": "Kotlin", "language": "kotlin", "name": "kotlin" }, "language_info": { "name": "kotlin", "version": "2.2.20-dev-4982", "mimetype": "text/x-kotlin", "file_extension": ".kt", "pygments_lexer": "kotlin", "codemirror_mode": "text/x-kotlin", "nbconvert_exporter": "" } }, "nbformat": 4, "nbformat_minor": 0 }