{ "cells": [ { "cell_type": "markdown", "id": "ff1d14f8", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:38:54.316710+00:00", "exception": false, "start_time": "2026-05-28T02:38:54.316710+00:00", "status": "completed" }, "tags": [] }, "source": [ "# 24 · Corrective RAG (CRAG) — grade docs, fall back to web\n", "\n", "> **TL;DR.** Retrieve from corpus → **grade each doc** as relevant/ambiguous/irrelevant via a categorical LLM judgement → Python composes a routing decision (use-retrieved / web-fallback / mixed) → answer.\n", ">\n", "> **Reach for it when** corpus coverage is incomplete and falling back to the web for out-of-corpus questions is acceptable.\n", "> **Avoid when** retrieved docs are always relevant (CRAG adds a grade-call per doc for no benefit) or when web fallback is disallowed (security/compliance).\n", "\n", "| Property | Value |\n", "|---|---|\n", "| Origin | Yan et al., *Corrective RAG* (2024). [arXiv:2401.15884](https://arxiv.org/abs/2401.15884) |\n", "| Grader | Categorical per-doc label (`relevant` / `ambiguous` / `irrelevant`) — deterministic-picker |\n", "| Picker | Python composes route from grade counts |\n", "| Web tool | Tavily (`agentic_architectures.tools.web_search_tool`) — gracefully degrades if no `TAVILY_API_KEY` |\n", "| Default LLM | Llama-3.3-70B |\n", "| Cost | 1 retrieve + `top_k` grade-calls + (optional) 1 web-search + 1 answer = `top_k + 2` to `top_k + 3` calls |\n", "\n", "**Why deterministic-picker matters here.** If the grader emitted a numeric \"relevance score 0-1\", we'd be back to the flat-scoring pathology (Mental Loop nb 10 §11). Three categorical labels give the LLM something concrete to commit to per-doc; Python counts the labels to route — the deciding signal `(n_relevant, n_irrelevant)` is fully Python-computed." ] }, { "cell_type": "markdown", "id": "31767b56", "metadata": { "papermill": { "duration": 0.015888, "end_time": "2026-05-28T02:38:54.332598+00:00", "exception": false, "start_time": "2026-05-28T02:38:54.316710+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 2 · Architecture at a glance\n", "\n", "```mermaid\n", "flowchart LR\n", " A([task]) --> R[RETRIEVE
vector search top-k]\n", " R --> G[GRADE
per-doc categorical relevance]\n", " G --> RT[ROUTE
Python composes from counts]\n", " RT -->|use_retrieved| ANS[ANSWER]\n", " RT -->|use_web| W[WEB SEARCH] --> ANS\n", " RT -->|use_mixed| W\n", " ANS --> Z([final])\n", "\n", " style G fill:#fff3e0,stroke:#f57c00\n", " style RT fill:#fce4ec,stroke:#c2185b\n", " style W fill:#e3f2fd,stroke:#1976d2\n", " style ANS fill:#e8f5e9,stroke:#388e3c\n", "```" ] }, { "cell_type": "markdown", "id": "06d4b227", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:38:54.337413+00:00", "exception": false, "start_time": "2026-05-28T02:38:54.337413+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 3 · Theory\n", "\n", "### 3.0 · Why categorical relevance, not a score\n", "\n", "`_DocGrade.relevance: Literal['relevant', 'ambiguous', 'irrelevant']` — three discrete labels. The grader can't slide-into-a-flat-band because there is no number to slide. Python tallies the labels and the route is deterministic:\n", "\n", "```python\n", "rel_frac = n_relevant / n\n", "if rel_frac >= threshold: → use_retrieved\n", "elif n_irrelevant == n: → use_web\n", "else: → use_mixed\n", "```\n", "\n", "### 3.1 · When CRAG beats plain RAG\n", "\n", "- **In-corpus query** with high-relevance retrievals → CRAG behaves identically to plain RAG. The grade-call cost is the only overhead.\n", "- **Out-of-corpus query** → plain RAG hallucinates from irrelevant docs; CRAG detects the irrelevance and pivots to web.\n", "- **Partially-in-corpus** (multi-hop where only some facts are in corpus) → CRAG mixes retrieved + web; plain RAG misses the gap.\n", "\n", "### 3.2 · Where this sits\n", "\n", "| Pattern | Strategy when retrieval is poor |\n", "|---|---|\n", "| Plain RAG | Use whatever was retrieved (will hallucinate from junk) |\n", "| [Agentic RAG (nb 23)](./23_agentic_rag.ipynb) | Agent can choose not to retrieve, but no fallback |\n", "| **CRAG (this nb)** | **Grade docs; fall back to web search if irrelevant** |\n", "| [Self-RAG (nb 25)](./25_self_rag.ipynb) | Self-emitted reflection tokens decide retrieve/answer per claim |\n", "| [Adaptive RAG (nb 26)](./26_adaptive_rag.ipynb) | Pre-route at task-level (no-RAG / single / multi) |" ] }, { "cell_type": "markdown", "id": "2e3449a6", "metadata": { "papermill": { "duration": 0.01095, "end_time": "2026-05-28T02:38:54.348363+00:00", "exception": false, "start_time": "2026-05-28T02:38:54.337413+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 4 · Setup" ] }, { "cell_type": "code", "execution_count": 1, "id": "cbbde1c5", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:38:54.350653Z", "iopub.status.busy": "2026-05-28T02:38:54.350653Z", "iopub.status.idle": "2026-05-28T02:38:57.047241Z", "shell.execute_reply": "2026-05-28T02:38:57.047241Z" }, "papermill": { "duration": 2.696588, "end_time": "2026-05-28T02:38:57.047241+00:00", "exception": false, "start_time": "2026-05-28T02:38:54.350653+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
LLM: meta-llama/Llama-3.3-70B-Instruct  ·  Corpus: 12 docs  ·  Web fallback: Tavily ───────────────────────────────\n",
       "
\n" ], "text/plain": [ "\u001b[1;36mLLM: meta-llama/Llama-\u001b[0m\u001b[1;36m3.3\u001b[0m\u001b[1;36m-70B-Instruct · Corpus: \u001b[0m\u001b[1;36m12\u001b[0m\u001b[1;36m docs · Web fallback: Tavily\u001b[0m \u001b[92m───────────────────────────────\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from agentic_architectures import get_llm, enable_langsmith, settings\n", "from agentic_architectures.architectures import CorrectiveRAG\n", "from agentic_architectures.data import STARDUST_CORPUS\n", "from agentic_architectures.tools import web_search_tool\n", "from agentic_architectures.ui import print_md, print_header, print_step\n", "\n", "enable_langsmith()\n", "llm = get_llm(provider=\"nebius\", model=\"meta-llama/Llama-3.3-70B-Instruct\", temperature=0.2)\n", "\n", "# Wrap Tavily as a simple callable returning list[str] of snippets.\n", "_tavily = web_search_tool(max_results=3)\n", "def web_search_fn(query: str) -> list[str]:\n", " try:\n", " result = _tavily.invoke(query)\n", " if isinstance(result, list):\n", " return [str(r.get('content', r))[:400] for r in result]\n", " return [str(result)[:1000]]\n", " except Exception as e:\n", " return [f\"(web search unavailable: {e})\"]\n", "\n", "print_header(f\"LLM: {llm.model} · Corpus: {len(STARDUST_CORPUS)} docs · Web fallback: Tavily\")" ] }, { "cell_type": "markdown", "id": "4d112544", "metadata": { "papermill": { "duration": 0.004022, "end_time": "2026-05-28T02:38:57.060547+00:00", "exception": false, "start_time": "2026-05-28T02:38:57.056525+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 5 · Library walkthrough\n", "\n", "Source: [`src/agentic_architectures/architectures/corrective_rag.py`](../src/agentic_architectures/architectures/corrective_rag.py).\n", "\n", "The `_DocGrade` schema is the deciding commitment per doc; `_route` is pure Python composing the route from label counts." ] }, { "cell_type": "code", "execution_count": 2, "id": "b2b90696", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:38:57.070361Z", "iopub.status.busy": "2026-05-28T02:38:57.070361Z", "iopub.status.idle": "2026-05-28T02:38:57.095421Z", "shell.execute_reply": "2026-05-28T02:38:57.095421Z" }, "papermill": { "duration": 0.030103, "end_time": "2026-05-28T02:38:57.095421+00:00", "exception": false, "start_time": "2026-05-28T02:38:57.065318+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--- _DocGrade schema ---\n", "{\n", " \"description\": \"Per-document relevance grade \\u2014 categorical (deterministic-picker).\",\n", " \"properties\": {\n", " \"relevance\": {\n", " \"description\": \"Categorical assessment of this doc's relevance to the question. 'relevant' = directly answers part of the question; 'ambiguous' = related but doesn't directly answer; 'irrelevant' = off-topic or wrong entity.\",\n", " \"enum\": [\n", " \"relevant\",\n", "...\n", "\n", "--- _route source ---\n", " def _route(self, state: CorrectiveRAGState) -> dict[str, Any]:\n", " \"\"\"Python-composed route — deterministic-picker.\"\"\"\n", " grades = state[\"doc_grades\"]\n", " n = len(grades)\n", " if n == 0:\n", " route = \"use_web\"\n", " else:\n", " relevant = sum(1 for g in grades if g[\"relevance\"] == \"relevant\")\n", " irrelevant = sum(1 for g in grades if g[\"relevance\"] == \"irrelevant\")\n", " rel_frac = relevant / n\n", " if rel_frac >= self.relevance_threshold:\n", " route = \"use_retrieved\"\n", " elif irrelevant == n:\n", " route = \"use_web\"\n", " else:\n", " route = \"use_mixed\"\n", " return {\n", " \"route\": route,\n", " \"history\": [{\"stage\": \"route\", \"route\": route}],\n", " }\n", "\n" ] } ], "source": [ "from agentic_architectures.architectures.corrective_rag import _DocGrade, CorrectiveRAG\n", "import json, inspect\n", "print('--- _DocGrade schema ---')\n", "print(json.dumps(_DocGrade.model_json_schema(), indent=2)[:400] + '...')\n", "print()\n", "print('--- _route source ---')\n", "print(inspect.getsource(CorrectiveRAG._route))" ] }, { "cell_type": "markdown", "id": "9a2ddb9d", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:38:57.095421+00:00", "exception": false, "start_time": "2026-05-28T02:38:57.095421+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 6 · Build the graph" ] }, { "cell_type": "code", "execution_count": 3, "id": "74358e14", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:38:57.111329Z", "iopub.status.busy": "2026-05-28T02:38:57.111329Z", "iopub.status.idle": "2026-05-28T02:39:04.189300Z", "shell.execute_reply": "2026-05-28T02:39:04.188684Z" }, "papermill": { "duration": 7.077971, "end_time": "2026-05-28T02:39:04.189300+00:00", "exception": false, "start_time": "2026-05-28T02:38:57.111329+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAKcAAAJ2CAIAAAC1iQUYAAAQAElEQVR4nOydCUAU1R/H38ye3CAKigiIiKKigGhqZhre933nkeZZaWpW/zxSM80rKzM1S/PKTFPRvDVvzRMBrzwQlUOR+1r2mPn/dgeWBRbYhV3Znfc+0Tr75s2x8533fr/35s37CVmWRQTMECICfhDVcYSojiNEdRwhquMIUR1HTKC6LAfdPJ3y6nmeLEulUDAKGUtRiGsPUgLEqtSf8J1lqPyvtGYtq1nLaBZozQLSWYDzYoonIhpRqGC5YOeIYhEckFan6KxiWRVVeIqQB74xhSlCKSUQUGIbQQ1PafCbTvbVBQgnqMq01/f8EJf0XKZUsCIxLbEViMU0XG6FjEE0hRj1bmkhzSgZWkAxcBQGxKBYFUvRmoOyqFg60hGYFkEKLGsSddYiKn8ZaW8pzSawT4Ty82uOixil7q9ENE0xqsJfCnqrlGyejJVnK5UqlqaRUw1xn4m17Z2xkL+Cqm/7+mnaK7mdo7B+sEPbPq7Iyvn3SGr0xbScTKWdg/C9hXUR3zFa9fP7X906m+ZUXTxythfiXcHYueLZq/g830D77mNrIv5inOo7lz/LSFH0m+xZw0uM+MuGL2KkNvSoOd6Ipxih+rGtL1/E5r7L32uhy58r41lKNXhGHcRHDFV9+9KnSiUaPccLYcOuVXFZaYr3Fvog3kEbkmnf2nilnMVKcmDwjNp2zsKdy54h3lG+6o+jcuJjckfPw6JiL8aQGZ7pqYpLfycjflG+6se3Jwa97YJwpdPQWjf/SUP8ohzV/9mVBJ0kbXpWQ7ji28xGaifY830c4hHlqP7fjcyAUCeEN2/1dXvxVIZ4RFmqx9zOVSqYt/q91oK+a9eu+fPnI+P57LPP9u/fj8xA/WBbWkhdCOePdS9L9WvHk+2dX3dvzJ07d1CFqPCGhlC9pvhhZBbiC2W11zfOifEJsO84ogYyA0+ePFm3bt3169fhBJo2bTpq1KigoKAJEybcuHGDy7Bt27aGDRv+8ccf586di46OlkgkISEhU6dO9fT0hLWzZ88WCAS1atXasmXLsmXL4Cu3lb29/enTp5Gpifgn/d+jyROX+iJeUFZZl8sYv2B7ZAbkcjkIDLL98MMPP/30k1Ao/Pjjj2Uy2YYNG5o0adKjR49r166B5BEREcuXL2/WrNmKFSsWLFiQkpIyZ84cbg8ikeihhlWrVgUHB1+4cAES586daw7JgaB2TmDsEF8o6/k61AI+DWyQGYiNjQUJhw0bBtLC16VLl0IRVyqVxbIFBgaCmffy8oLbAr4qFAq4OdLT052cnCiKio+P37p1q1QqhVV5eXnIrAjUz4Wf3s3zCpAg66dU1VNfKCgKmempGgjp4uLy5Zdfdu/evXnz5lCaQ0NDS2aDyuD58+crV66EGj47O5tLhNsFVIeFunXrcpK/NtJT4d7ig+ql1vDq0ScUMhNgpH/++ee2bdvu2LFj3Lhxffv2PXToUMlsZ86cmTFjRqNGjSDz1atX16xZU2wn6DUCtQtiePLGSKmqV3MX6g4+MTk+Pj7Tp08/ePAgGGY/P7958+bdu3evWJ69e/eCiwcenL+/P1z0zMxMVHUwDGvnKEK8oMxeGgrFPTCLvQQHPjw8HBagim7Xrt0333wDlvvu3bvFsoEJd3Nz0349deoUqjqgDPgG2CJeUJbqIhH14GYGMgMg58KFC1evXv3s2TPw7DZt2gSuHFh3WFWnTh2w4lCfg/2GIn758mXw52Ht9u3buW0TEhJK7hBqe7g/tJmRqYm6mKn2cnhS1MtU3dFV/PxRLjIDIPD//ve/w4cP9+vXb8CAATdv3oS2u6+vujXcv39/qMyhVn/w4MGUKVPatGkDpr1169aJiYnQeAMb/9FHHx05cqTkPt977z24V2bOnJmba/pz/u9qhsTGoKfSVkFZvTSRZzPO7Xs5dZUfwp6fPnlUr6l953fdES8o6/5t2s4R7ogbJ/n2nNFYkuIUSiXDG8lRuW9B1G3sEHE2LSTMubQMkyZNKul7AyqVCmoRrnelJPv27XN2LnWflQG686BpoHcVnBJN0+oGmD5OnDhR2tke3hRfzZ0PzXQt5Y+bWzPjYYdB7o1bO+hd++rVK+he1bsK+stKa1J7eHggswF9dsh4SjulrDTVpgWPP/y2PuIR5b/xFBpW7fz+pNJUr169OrIwTHtLbV8W69vEEfGL8v3SVj2qObiIdq16jvDj4IYEoYDqMY4/Fp3DoNbI8E/rpKcowtcnIJy4eCAVGq7jFvHwBSgj3oLY+nWsrYNwwIe1EQac2PEy5nb2+4v5+c6bcW88/Tr/CS1AY+b5IF6jfrErVTHha56MoSiJ0W837v4+7mWszLeZY9dRZhljU7Wc+etV9IU0F3fJ8Nn8fNeJoyJvMsc/URz6JU6eq6ruKenQv2YNL6uf8CI9WXV8e+KL2FxaQL3dz71Ra7OMILIcKj5rwb0rWRf+fiXPViGasrGjbZ1Edg4CoYCVK4rskBZSjLIwRT17AMOq5yvQPMblvuafCk2xOg+wC6Yj4OYxUE9Soc2vngsDcQlcVvUyrZkCg2UL98ltKxAilc7jGMjGMEgkhnVUVoYyO12ZnaFQKVmxVBDSwblFZyxG/ldqrgqOyDOZj29nZqYqFQoGtFTkFVVdR1f18TSTl1Dqw1Lar9pzQazutCIaLSlWpZ7FhKVoWic/i0oM+VDPZMFyOy/YJ1Vk+hPdExBKkFBAwx3p4CzyrG/bsotZOgotFhOobm62bt0KT12nTZuGCCbCCkwyPC8vrYecUDGI6jhiBSMFiOomxzpUF4n4MnbJMiA1PI4Q1XGEqI4jVnA1FQoFUd20kLKOI0R1HCGq4whRHUesw5sjvTSmhZR1HCGq4whRHUeIXccRUtZxhKiOI0R1HCGq44h1qE68OdNCyjqOWMHVdHd3p2n+zP9kCViB6klJSdBkRwTTYQWqQ/VujokDcYaojiNEdRwhquMIUR1HiOo4QlTHEaI6jhDVcYSojiNEdRwhquMIUR1HiOo4QlTHEaI6jlju3JJdu3Z9+fKlbgrDMH5+frt370aEymG5I5PCwsKQehraQmxtbYcMGYIIlcZyVR8xYkTt2kXiTtSpU6dv376IUGksV3UPDw+uuHMIBIIePXqQIdImwaLHno4ZM8bT05NbhoX+/fsjgimwaNWdnZ2hfCNNxPuOHTva2/M8RMNrw2gf/tGt3JjorNyc/KHKlIBiVWx+iIaCYA66E/FziQUhIBCjUieq4zYgncgPAoRURU+rYGZ/hlVdvXoVFoKDQsRiMaWJJ6CNLQCfRaIOUOp4nNxu4RDwRaXKX8WFgNBGjWC40+MiC7Bchvz9SG2EtfzsmrSyQ/zFCNVVKrTpyydKuUokFshlWlXVARzyo3UUSKUb4SE/7EN+WAaWZTQhINRVTP6y5mvhsnoFpdmA1e6fC/1AaVapEwrigbAanXWiSahPhco/BzgEBfdMQWUG/xaoXrA5t/+CZS4DQhIbWi5nBQLUZ1JttzpixEcMVV0lRxvmPg5o7tK8iwvCgNuXMm+eShrwoScvhTdU9fWfxbTq5u4bZIuwQZ6D/vj28ZRlPIzqZpA3d2zbS5GExkpyQGyLHF3Ef37HwzilBqn+8qnMoRqODWW3OtKMZBniHQapnpfDUAIc3yplaUaexyDeYdAzN2j/MHg+9WJYRmXpkc8qAJkNoCzUri4PRSeql4m6k4CiEO8wTHWahT44hCEU4mVhN0x1Rt3tivBD3a1IanjcgIcIiI91HFG9LNQPcrAt6yyy+MDN5kH9bAfbsg5PIfGd+YuPN7xhZZ3hZT1nEGrTzjsMUp0SsDSeLTdE8dK2GVbWVRQvOybLheVpLWfp9rpPv7AtWzeiqoJSD+vhHwapTkHfnNluj737di35Zn5pa4cMfrdpYDCqKnDupWGhb85szxvv379Txtrhw8YggqkxVxGGmnnPnt+nffx+h7DQjMwMSDly9MCUD8Z069EWPnfv2cGN3Jo+Y8LRYwePHfsbsv334N6ev3YOGNTl/IXTYZ1a/vDjClS0hr99O3L2px/07tPh3dH91/70bXZ2NiRu/OXHHr3a6U4vvPOPLZ26tMrJySntoIajHnPLRx/eMNXVNbxxP14kEh08tNfPr8HyZT/a2tieOHnkm2UL/Os33LEtfPy4qSDAmrUrIdvqVRsCApp07tzjn5PXYK1YLM7JyQ4P3/35Zwv79Rmsu8Pncc9mzZ4iy5Ot+WHTogUrHj9+8PGMCUqlskP7ziDwlSsXtTnPnf+ndau3bG1LPajhwF1SOHybRximOkMZ++OhkDg6On04dVZo8zeEQuGhQ/uaNg2ePu0zF5dqIcEtxo6etG/frtTUlJJbyWSyoUNHdwzr6unppbvqxInDIqEI9Pby8vHx8Z01c+6Dh/ehVqhXr76HhycozWVLTn51507UO+90gWW9B01PT0MGo+6b42P/lKG/qQJOTQP/RtwCwzDRt2+1CG2tXRUc3AISI6Nu6t2wYYPGJRNv377VsGFjJydn7mvNmrVAbG4PnTp2O3f+lEqlfpHi7LlTNjY2bd9sX9pB7967jYwBX2+Ohl4aodHmDaprbkEul4Pd/eXXtfCnm6FkWS+2oS5ZWZn37t8B819kDynJ8NkxrNtvW36+cfNqi9BW58//89Zb70DtAnWG3oOmp6Uiw6EQvv3wjKrw1aEKIJVKwcp27tSjXbsw3XSPWp6G76Saa/XAwKCxYybpJjo5qos+2AKo5y9cOO3vHxBx6/rSJd+XcdA6nt7IYFh+dsMb+KQVqrnK3fL16vlnZmUGB+WXVCiFCQlxbm7uRuzBt/6x4383axqijQHz5Mljre0Hn+7gwb+8vX3BmQATXsZBXV2rI8Ph6ZNWw+w6S1Xyx78/7gMoi4cO7wfLGhUVsXDR5zNmTYKaH1bVrl3n7t1oqJ9Lq/A5Bg4cAduCEw5V97Nnses3fP/e+CGPYx5ya9u375T4IuHIkfAOHToLBIIyDkpCyKDX1iMLlfOGddsjI2/2G9AJGmDZ2VlfLVolkUhgVa8e/cFV/mT21EePH5SxB0cHx182/mEjtZk4eeSoMQOgJv9k1lxolXFra3t4NvAPgBZ/WIcuZR9Ur9NQGnwdS2PQe24bvnjk4ibtOqY2wowL4YmPIrKmrvRD/MLA0ZK0xU5VRagABo6lYTEdTAM1PLajKhhzPn2xaMCH52OPrGFjaYQsCZnJJwx70qqkGCzLOqXpiecfBr7xBNYNz3FziJddsgaOqmBVWPrwrOZZK+Idhtl1Xr7YiTEGP3MTIAJvMPSZG6NCGEJRWHtzmokE8YPlp1k3sIYvMhsjweoxsG8OkX54PmGQ6mIbSiTBsYYXCcViKQ/9WINUt7UX5mbgWNYzU+USGx7e7gb9pOAOrpkpPJxisVyS4nLrNXVAvMMg1esH2zjVkO5e8RThxN41z6Cgv9m7GuIdRswP/8+uV4+isz18bGrXd2AYMXNIKQAAEABJREFUfaPPCuZqL+LxF53AXc8M7aWlFywXhADQOdVim+scQjtBfJH9aSaxL7LPggAGhbsqWBAiYdyTnLhHWa41xX2neCA+YlwsiPN7U/67laGQMQq9s6sa0r7T5NGIQJW9bWGeEjcEW+yRiL5VunkKl0vEANCmaA8nEFESqcCroX3H4caMprUqLDeKn5Zt27a9evVq+vTpiGAirGDmMaVSKRSSGdJMCVEdR6ygMapQKEjwPtNiBaqTsm5ySA2PI0R1HCGq44h1qE68OdNCyjqOENVxhKiOI1ZwNaGXhqhuWkhZxxGiOo4Q1XGEqI4j1uHNkV4a00LKOo4Q1XGEqI4jxK7jCCnrOEJUxxEruJr+/v5EddNiBVfzwYMHZHpv02IFqkNBh0oeEUwHUR1HiOo4QlTHEaI6jhDVcYSojiNEdRwhquOIFagOD9xI35xpIWUdR4jqOEJUxxGiOo4Q1XGEqI4jlju3ZFhYWGpqKkKFM8jCZ506dcLDwxGhcljuzGOtW7dG6sDANKhOaxCLxX369EGESmO5qo8cOdLb21s3BQp6v379EKHSWK7qDRs2DA0N1X6FEt++fftq1Xg4W/vrx6LnlhwxYoSvry+3DOW+f//+iGAKLFp1Hx+fNm3aIE1Bb9u2rYcHP+fof/2YpuUWeyc3O0OhDq5QfMJ+RLOI4VIKAjIUBmYoGiWigCJz+Ic2GPCwsYJhVEH1+kRfykDaFaye/CVWlRU/gMPOVly3mRRhRmVbbvt+TEx8mgNXUylnWN0gD1qNNeE7NHmLx2koqnpBojpGA10QlEO7beGWmg3hrClUdKeo6Kr8ABI6OZG++0wopiGpWk3pkBm1ETZUSvUDGxJfxee91aeWu68YWS1pL1RndifQFDv88zoIDyqu+s4VcXl5TP8PeHKljm5KyMlUjJrrhTCggt5cVhJKfSnjjeRAl7G1cnOUty9kIQyooOqXjiWLbfj2xqG9g/je9QyEARVULjNVxsO45AJVbg6DMKCCqoPHDn+IXyjywJ0nqmOGuuFJ8bAKKwlRvRBK3dVg6TENTUIFVaeFiBbw7QJR+b07/KeCqjNKxKj4doFYrmMQAyrYcqMFiOZfNHqLD1lrKiooHaNCDO+KhabHntj10hGIaJp3jqBAQOHRcKuo6ip4+Mm7YasqFUO8OeygKVxMe0VVp3nYn8GSlls5MHwsFSyFSVmvoA9PURZRKP45fbxDWGhaWioyCSyDSdOtgqprBkvxDxqPhluFa3iK5V/Tljx9KQ+152PcBUpNTVmydN7tO5FedXz69Bn0/PnTc+f/+W3TbljVp1/YqJHjz54/FRl5c/++UzRF/7l725Wrl548eeRarXqbNm+/N3ayVJo/knXd+u+OHf/b1sY2LKyrp2eRl2OOHD0QfmBPTMzDunX93unQeUD/YRQeKhpLhcu60aVi2YqFT589Wb5srbtbzTU/rgDV6YJOXZFIdPDQ3pCQlu+OHA9y7vh9M/x98b+vnJycs7Iyf1izXCAQTJzwEeTcH757f/ifn326IDi4xcWLZ7Zs/Vm7/xMnj3yzbEGf3gMXL1oV8+TRsuULEhLjP5w6y+ATREIhhckzt4p2phvp7aZnpF++fH7woHcbBTRxda0+c8acxMT4wp1RlKOjEygU2vwNoVA4eNDIjRt+b/92x+Cg0LfadujQvvOVqxe5nH/t3fl2u45vtwtzdHDs2qVXSHAL7U4OHdrXtGnw9GmfubhUg/Sxoyft27cLKhhkMEolq1BiUTdUVHUjey6fxsbAZ5Mmzbiv9vb2ULJ1MzTwb6RdhqJ/9dqlyVNGderSClz0XX9u48RjWTYu7pmPj682p79/QP7pMEz07VstQltrV0FlAImRUTeRwZDn6+UBF4g24gJlZ6vHntrZ2WtToHDrZhCLC0fUb/j5Byi4EydOAxXd3Wtu/OXHQ4f3a3aSrVKpbGxstTmlUhtuQS6XKxSKX35dC3+6uzWqrGtqL+LNlQH4u4wRF0ii8cUUcrk2JTVNvx5QoA8c3DNwwPCePfJfWgbTzi3Y2dmBgc/Lk2kz5+bmcAvg69na2nbu1KNduzDdvXnU8kSEErymfvhaNdXvE4GTxdXPWVlZN25ccXevVTInFNnc3Nzq1d24r1CIL146yy2D+YdNbt+ORIPyM1/+97x2w3r1/DOzMsEV0O4nISHOzc0dGYxYQlvszB2mpaJ9c+DuGjOCqmbNWt7edX/bsiEu/jlIvvq7JbVq6X+vDKp6Ly+fw0fCIWd6ehp4/oFNgjIzM6B6h7Ud2nc6e+4UdMnB8u87f7tzJ0q74fvjPrhw4TTYAjDnUVERCxd9PmPWJLlO7VIu8jxGYUR2K6aifXNKljVyBNXsWfOgqfbuqH4fz5gAXliTxs1EQv2x+eZ+8bVUIh0zduDIUX2bh7QcP/4D+NpvQEdoiY0cMa5H977QlgMv79Llc1Mmz0AaowCfgYFBG9ZthxZ/vwGdZs2eAp7EV4tWSSQSRChBBd9z+/PbZ2lJyqGf1jV8Eyi4MpkMvDPu6+dfTBcKhIsWrkAWw57vY8GHHz3XB/GdirbcjH/SumDhZ1DKoT8O5N+67Zfr1//t3XsgsiSgADAs8eFLB7rhGSMrifnzv1m+YuHPG9ckJb3w9qo7f+7SFqGtkCVBC3Dpv62g6lAkaCMvkZOj01cLVyLLBhMfvqKqs+pxFYhfMCqW9NKUCfTIMphUhzykou11gXE9staBeu5ShAMVba+rjOuRtQ6KzFzEZyrqw9PccBpeQfHvHa5SqLg3xz/HB5MXX1DF7Tofx8Nr2qKk5VYGfBxDrOl3Ii230gG3B5/6kH9U1JtDmJQKflJB1YVSWiThm+wSiZCMqigLRxcRo0I8Q6lU2ToIEAZUUPWwwW55OXyTPTdD1byDK8KAinZMCFAdf7udy54gvrDv++cOrmKvRliMvanUTOH//JEUcyfHv4VLs7cckdVy90p69Lm0GnUkvd6vifCgslEB/tn56kFkllLBqKf3KH1P6h5u3W4dnYZxYSCBoqEaSov3UHxX+YkFvUZGNrkFFAWeqWc92+7jjBhNa+2YKIqfCuVmFTXzumEXtDEhCtfmR2rI17BglVr/gnSt7H/t3ZucnPz++PG6WxXbiU5iwVdU8LJ1wX5Y6E8s0cdgYyNAVhzRoIKYaDy8ANk4mcv7VaAsWqIw3/4xxApmI1IqlUIhmTXJlFjBw0WFQiESiRDBdFiB6qSsmxzrqOFJWTctVqA61PCkrJsW4s3hCFEdR4jqOEJUxxGiOo5Yhw9PWm6mhZR1HCGq4whRHUeI6jhCVMcRojqOENVxhKiOI0R1HCGq4whRHUes4GoGBAQQ1U2LFVzNO3fuqFS8e4G2SrEC1aGgQyWPCKaDqI4jRHUcIarjCFEdR4jqOGIFqotEIoVCgQimg5R1HCGq4whRHUeI6jhCVMcRojqOENVxhKiOI0R1HKEsdkb0kJAQzayTOtOKsmyNGjWOHTuGCJXDcmcea9myJchM6wA3QVhYGCJUGstVffz48VCydVM8PT2HDh2KCJXGclUPDQ0NCAjQTYHS7+3tjQiVxqLnlhw3blzNmvlztru7uw8aNAgRTIFFqx4YGNisWTNuuWnTpg0aNEAEU2Dp88iOGjXKzc0NDPywYcMQwUSYpuWW8FB2bMeL3CyVUsWyhoV11BvS4bUBDQKaRhJbYWhH16Zv2SPMMEEvTWayat+G+Nr17NoPcrF3FGveWNDEX9CGgNCNf8pFb9D5kh/2oehCfoAIShvRQzeyhE4sCFQ0LgTSCRCBUJHoEzRVJMqoQB284s6l9It/JzlWE/g0tkE4Udmyfudyztn9L0Z8VhdZLTuXPmkQat9uQHWEDZW16xcPvmgcWg1ZM+36e9y9koFwolKqpySwijw2qKMTsmY8/MVwGW6dwUj4Stn1uMdZNC+CtcKvSHuRh7ChUqqzDKNQ8iEgt1zOwh/CBvJeOI4Q1XGEqK5GKEK0iNTwmKFUIEbBC7/UMCrnzWFUPHhFpVSHXlOMCgiPqGQNT/GjtMOTGEpA7DpmMAxiVcSuE3gNUR1HKqs6xYuOeLVbagXRqU1GJVVnWT50w2tkRxhRWR++cJCMNaO+d/lx+xqGtdZr/QZ0ik+IQ4QKYZXeXGJiQlpaKiJUlEqVdZY22pub/+XshYs+X7/h+w5hoWfPnYKUnJycr76eM3Bw1y7d2kycNHLf/j+5nDv/2NKtR1vthi9eJMImFy6cuRlxbdiIXpAyYmSfOfNmIs0E8rDDseMG9+jV7tPPP7p8+TwyEvgRWHlzlfqtFHRuMMbZdZFI9DjmIfwtXrSqaWAwpHz2v4/i458vWrhy185D7dqFfff9N3fv3S5jD8FBoUsWr4aF7dv2f7VwJSx8/8Oy3Xt29Os7ZMf2A2+3C5u/YPaZsyeRMcCP4IlbahiVu8Mpo305iqISE+MXzF/Wpk07Z2eXy/9eiIqK+GTm3ICGjZ2cnEcMHxsYGPTblg2G7zAvL+/osYPDh43p3WuAk6NT9259wt7pumXrz4hQOpVTnRvAbiTeXnWlUim3HBPzEJbr1q2nXetfP+D+/TuG7+2//+7K5fIWoa21KUHNmj9+/JBMR1kGVeDNiSUS7XJy8iuptMgbCLa2trm5OchgsrIy4fPDaeOKpUMdYET8bsyeGVdOdQpV8qUlOzs7mSxXNyU7J7u6a42SOVWM/iAgrtXVmWfO+KJ27Tq66drqxBAoAUX65gyGrWwhaeDfSCaTPXh4v75f/vuqd+9G+2gqfJFIDOVVG+DpaWyM3j141vaSaCoP8PK4lNTUFJZljQoQxDLEmzOGSqresmUbDw/PVasW37t/JyUl+Zdf14LqQwa9C6saNQoE8Y4cPYA0zbYdOzdrt6rj5QOfp08fv3M3GizCmNETwX0DrxAMPHjvs2ZPWf3dUkQonUo/falcDQ8lElpf69avnjJ1tFgs9vWtv2jhCnDjYRV49ZMnTd+w4fuVqxbDHTBh/IfTZ0zg3sqr7eHZtUuvTZvXNWnc7NtV64cOGVWvnj/cFjduXLGzs2/cqOnMmXMQoXQq9XZj5Ln0s3uTRs/3Q1bOlkWP/IMcOo10Q3hQudGSiCfQAgoJED5UbrRkpWt4C4GBzjmcvLnK2nWeNHRZHlVcBkBGUOEIses4Ulm7ToS3RirdXkd8QCiiaJxsHbHrapQKlsFpLvLKjpHlSWHHjEp6c0UncSNYCZX15njSTYMZxK7jSGXfglD3YFs/QhEtFJN3Wg3DqbqU5kUNT9PIzgmjaq9Soyq8A8Qg+qPIbGTNyLORQs627OKCsKGyY2l8Ax2uH3+FrJkDG5/W9DZikB0PMMH88FeOpUWcTmvT0827sS2yKpKfy0/uTPBqYNtpZA2EE6aJCnB0a9KT25nc4EllaXOMUoVzvOcvFvTja2aQL34a2oneuQnmi00DX8qOC/KrJ8xhi+Yp8ktpAS0QqFNq+rZE2tkAABAASURBVNj0nVwLYYYpo/g9vCZLTctlVYU7PHnyZGhoCycnx+JHLSoeW3p/vq725aLdLUsVnycpTyE/duRYr14983NSyMFZ1LClA8ISM8ZuXLduXe/evT08PJBlkJyc/Ouvv37yyScIeyw3YifBfJjljY/p06ffunULWSSRkZFz5uA+btr0Zf3QoUPNmjWrXbs2slT++++/O3fu9O3bF+GKiVXPzs6WSCRGvW1EeP2YsoafOXPmtWvXrEXyP/74Y+3atQhLTFbWb968WatWLW1YVavg6tWrcI8GBwcjzDCN6s+ePXN0dHRysr5gTwqFAjpwcDNJJqjhFyxYEBERYY2SI808OYsWLQIPFOFEZct6XFycjY1NtWrWHcjv3Llzfn5+YKEQHlRK9du3b9vb23t7eyPrB3ruHBwcxGIxwoCK1/CrVq2Crhh+SA64urqOGjXq0aNHCAMqWNZlMhk4QRKdeYX4wZkzZ1q1asW/31WMiqgOjXKapkNCQhAfuX//fv369eEHIv5i9G/bsmVLdHQ0XyUH/P3933jjDcRryDM3PahUqqioqKCgIMRTjCjrFy9ePHr0KMIAgUDQsGFD6G1EPMVQ1Y8dOxYbG9ulSxeEB1KpFPohRo4cifgIqeHLIjc3NzU11XKGA5mK8sv62bNnN23ahLAEijs489BmQfyiHNXBtkGn1dixYxGuwFPE+Ph46KtHPILU8AYBvVLg2NvZ2SFeUGpZP3/+PDxMQwQN4Nw9fPgQmnOIF+gv6+CuQxdV586dEUGHzZs3Q2ftsGHDkJVDangjAJc+Ly/P2dkZWTn6VYfqHXoqWrdujQh8RL9dh552eHaOCEW5dOnSkiVLkPWjf7xY27Zt+f3QqWIoFIpXr6z7tW0OYteNAIw6NOGsdISgLsSu4wix60YAl+XTTz9F1g+x60YA3XPErmOHUqnMzMx0cbH6eYuIXccRYteN4OnTp5MnT0bWD7HrRgD14suXL5H1Q+y6EYA3l5aW5urqiqwcYtdxhNh1I0hPTx8xYgSyfohdN47ExERk/RC7bgRwraCXpkYNq59+VH9ZJ3Zdl0mTJsXExAiFQm5aUviEihCevx0+fBhZJ8Sul8/QoUPlcvmLFy+geuc+4+Pj4fkbslr0qw52vU2bNoigoX379g0bNmSYwlmRYblBgwbIatFfwzdp0gQRdBg7duyjR49SUlK4r/b29lbtzOsv62DXL126hAgFtGzZMiAggPN84dPPzw+qQ2S1ELtuKOPHj+emK7KzswNLj6wZ0l43lMDAwKZNm4If5+PjY+1vClhKe/1hRO7lQ0k5WSq5TKVNLBnkgYOmKUYnZmSxmAEF4SDUUQJ040jozwwJNGLKi0BJqVFn07TckG4Ug6InSWksgJ4DlchfLDqFnr3p23850RHEElosFfg1s2/bt6yHBRbRD38+PDX6Ymp1DxsPXxuVSjeASCmxvg0IAc5FkNCXrUiqOmoEMmRvemKUlHNipZ2kJr3U0yuRqKt0yRgXxRCIqOTnyhfPsqW2gpGf1yktm/4aHuw6dEq8HtUvHky9fSltxOe+iGAyauz9/um+NfF9P9D/5n1Vt9dVKOJ0yvDP6iKCSen3kVdSQt6FAyl61+pXHdrrjRo1Qubn8G8vbezJZPJmwa2O3aOILL2rqri9np4slxLVzYOTqyg3R6l3VRW31/NyVHnZCkQwAyzLKOX6Y+uR9jqOkH54HKliu05RXM8G4bVSxXadZREZzPP6qWq7TrGkqJsLAUsL9F/cKrbr6s5oRDAPKopR6b+6lvB8nej+uqlqu86o/wivGdJex5Gqbq/TLCLenJkQoNI85apuryPizpkNplSPyRLGzZHCbh7YUh3lKn6+rvHmXl9h3/PXzrBOLZE1sPjrOR9OG4fMA+mHx5Gqbq/TLEWTGt48lO4xVfG4ufJG/xWn/8DOfXoPGj3qfaR+mzytb/+O7d/uOH/eUm7twMFdB/QfNmzo6Nu3I3/bsuHevdtOzi6tW701etQE7YT+0BsYnxD3669r/71yoXp1t2FDRnfu3KPsg2ZmZW7avO7fy+dT01Ia+Dfq2LFbj+59uVVHjh4IP7AnJuZh3bp+73ToDEfnHiZlZWX9uXvblauXnjx55Fqteps2b783drJUKoVVffqFjRo5/uz5U5GRN/fvO+Xo4Hjp0rnvfvgmKemlXz3/vn0Hd+vam9u5SCiKiLi+eMmctLRUWPXhh7MbBRhVB1PIKB/eYu16aGirO3fzp+a/cfOqu3vNqOgI7mtc/PPk5FeQ4Xncs1mzp8jyZGt+2LRowYrHjx98PGOCUlk4qmTJ0nmdOvVYuGBFk8bNlnwz/9mz2LIPumzZgju3I6dP/3zzr7sDApp8u3oJ3FWQfuLkkW+WLfCv33DHtvDx46bu3rNjzdqV3CZ/7d254/fNQwa/+/Xi1RMnTjt95jjchdwqkUh08NBeP78Gy5f9aGtjC5LPnT9r3HtTly75vm3bDsuWL4TdcjlfvEwMP7D7f58vglVyhXz5ioXGPakq3ZurYrturCMXEtzihzXLudeJb9263v7tTvv27wK9a3t4RkXddHZ2qe/XYPNvG6CUgN5OTuqJ3GfNnDtsRK/zF05DrYA0c8v07zf0jZbqexouPRTWk6eOjhk9oYyD3oq8MXTIqBahrWB5wvsfvv12RydH9Z4PHdrXtGnw9GmfwbKLS7WxoyctW7Fw5PD3YHnwoJFvtwvz9s4fBRodfevK1YsTJ3yENJWNo6PTh1NncaugFmn31judOnaDZThEdnZWTk42tyop6cW6n7Y62DvAMpzzipVfZWSkcz+qklR5e904moe8kZOTExOjjpwMpTywSVDDho2jo9TFPSoqonmI2j+/ffsWJGqvTs2atTw8PCOjCkPyvdHyTW4BLmhdn3oJiXFlHzQwMGjXn9t+Wrf64sWzCoWigX8A7JNhmOjbt1qEFhrB4OAWkMgdCAr01WuXJk8Z1alLqw5hobB5amrhcFUwE9wC5H/0+AGcrXbVpInTevcawC3Xq+fPSQ5w95lMJkOmoKrtupHdvjVquNWp4w2X29W1OmgPF/ruvWiQv0uXnnC5oUQitU3NvHf/Dlxr3Q1TU5K1y7a2ttplqY0NFKCyD/rp7C/Dw3ef+ucoiGdvZ9+v35BR774PJgPugF9+XQt/RQ6kUXfDzz9ATQB1O9wWYIY2/vLjocP7tXm0Ud5BRRBeIpHqPS5IoF2uyNgTCpXWOVfF/fCaURVGbQHFvSWYdqjMfX39QL/AwOCf1n0Lnt3z50/BcYMM1VyrQ+kcO2aS7lZcWeGAa805VgBUp7Vq1S77iOBwjRzx3ojhY6GiPnf+n63bfrG3d4A6HI7euVOPdu3CdDN71PIEA3Tg4J6BA4b37NGPS4QbUe+eJRIJXGeo1ZEZYLUfJaj69rqxd3FISMuffvrW3s6hWbPm8BUq+adPn5w4cdjLy6daNfXLXfV86x87/nezpiHaG/fJk8eenl7aPTx4cA9uC6SWPCc2NqbdW2FlHC49I/3kySPdu/WBGwW2gr+HD+//9+Ae0tTA4N4HB+VXKlD0ExLi3NzcYSE3NxcaCFy6XC6/eOms3p0LBIIGDRppHVLg541rIP/UKTNQpaGM7Zt7fe1140dQBQe1SHyRcOnSWfDAkaa6Bg8OfObmzfPDZw8cOAKqTXCnoUyDf75+w/fvjR/yOOYhtxaqTXCg4EaBKvqXTWvhE1pcZRxOKBCC+/3lwk+hoKekJB879veDh/fgVoNV74/74MKF01B1w+HAq1i46PMZsyaBZlCBwy14+Eg4uJlQCYGLB/kzMzOys7NL7r9Pr4FXr176Y9fWmxHX9ofv/n3nb3Xr1kNmpurfczMWe3t7KB/QFgd/nktp3Ljp3n27tF+hQv5l4x87d/42cfJIUBd8pU9mzYX2FVI78EpbWzuonKfPmAAGGGzEnC8W61YDJYGG/sIvl//w43KufxQkmTRxOtekhnK/Yd327Ts2wY0lk+U2btT0q0WroNKGVXO/+PrHtSvHjB0INcSUyTOCgkKvXLnYb0DH3zbvKbZ/8EgyMtPhxoJ7ApwVaCNAvYLMjP53WkF1qB5fw0tPvy2IZWl2wEc+iGBqrh5PvnspdepKv5KrLKAfnoylMQ9qu16Ky1TF883RQrDsVT9op1fv9qWt+vTTL9u+2R5ZI8b2zb02u84o4cSqvrBv3rS7tFUODo6Id1T5eHhkCaMqwI1COFHVdp1FZGS0uaBRaSW3qvvhaUTefTEfrFG9NK91PDwp6mai9Gtb9XadIqMlXztV3l5nyeuNr58qf8+NIs6cuaCMfAvi9Y2HZxEx7OaCtVi7TqgKqtquC1iaeHPmgRaqQ5XoX6U39bXZdRs7kUAoQAQzoMhDEhtjVH9tdt2ngX12phIRzMDL5zmOTiK9q6p4PHzL7k7gxEedT0cEU5OelNd/ov4hgRYxP/y6Tx83DHZu3q0aIpiC2Lt55/+K6/RuzXqBtnozWEqc1o1znqiUjFgqlOcZXOFzXqDe+ePp/IlPuABsxRKL5qS0L99oZ2LXTdRzuIIp3PXusHCC99KzafevXqV5Cl7yiMVSNEPIy8kDiKS0SsaqWLbryFo+TWxQKVjKuLnxX/ncu5IdE5WZlWVwi1HTl1tmvIViESQQo0f1Qkl0VNd3f6j7k9iE+BdubjUEtKC0bLqnV5rq2jOhKfVE/7C25LkVS4GvLCp59xRPkUrpmnXtWnR2QmVSxePmrI5u3bpt2bLF2oN2kvfXjUOhUIhEImTlkHhuxqFUKnVfRLJSSDw34+CH6qQf3jj4rDqx66WhUql4W8MTu64XKOjQjYGsH2LXjYAf1Tsidt0oeK46set64Y3qxK4bAT+6aBCx60ZB7DqOELuOI8Su4wjPVSd2XS+88eaIXTcCYtdxhNh1HCF2HUdAdWLXsYPYdRwhdh1H+G/Xnzx5gghFAcm1U4xbNaW+8XTz5k2o5/nhvJiKjh077tmzx8nJCVk5pbpswcHBIDwiFDB06ND169fzQHKEypzDtXHjxu3bt0cEhKZOnfrxxx/Xq2f2mdtfD+W80wr+y9OnT319fRHGfPnll6GhoT179kR8oZxGOfgvbm5u586dQ7iyZs0aHx8fPkmOylUdaWIvuLi4jBkzBuHHzp07ZTIZ/367obMWcBFsHB15OG12aZzQsHTpUsQ7DO12hSZcamoqPl03ERERUNB5KTkyXHXA29s7Pj5+yZIliO/ExcWBB7dx40bEU4yel4bVwONnM3l5eWFhYdAnjfiL0eJRFHXlypVr164hntKlS5cjR44gXlORItuqVSsw8OHh4Yh3DBw4cPPmzdBsQbzGImYesxAmT5783nvvtWjRAvGdSpln8HIfPnyIeMHcuXN79+6Ng+SokqrDA4nffvvt3r17yMr57rvv/P39u3XrhvCA1PBo+/btL1++hIcrCBtM0wBbuHBhWloaskKOHTvF05axAAAQAElEQVR2584drCRHplJ93rx50K0BXbbalD59zB5YuGJ0795du3z9+vU9e/YsXrwYYYbJOltWr15tY5M/cSk8l4Tu25MnTyILAxriUCeFhITAcmxs7Ndff71+/XqEH6Yc+5eTk/PJJ59AUx567rKysuD5LHRyIUvi4sWL0PUGpwfCwyeP+5rKxpQdqxMnTuQkV++Xpi3wPYpbt25RmrhH3EkOGDAAYYnJVB82bNjdu3d1++ehuFtUYQLJlcoi05BDJW+x/odZMZnqmZmZ8MnozG4NzaGrV68iiyEyMjIpKUn7FU4Vel75Mb7dWEym+sGDB2fMmNGoUSNnZ2fuuRwk/vvvv8hiAD8Dyjp3YjVr1nznnXfmz58PPjzCD6N7aZJiVOfCX2SkK+S56mJN0SzDUhTLhTVgEUsxrEohl6tUDPwnoAUSG6lQAJk0kRQ00QTVEXnVlpXlEllKMzk+V0downxpZ7pnKZaGfXNRHQQsq6IKIy3Q6uiz3LT42kOjgony4Qg0KsisyQlnJcuVMSwcl4LyLRIJEBKUDCkmsaVs7ETN36lWL8gG8RfjVL9wMCXqbLq9k9DBVZwnUyB9URc0114TeINLoTT/FuRRx1NQL8JaSntkzXJBGI4iMT0Ks4H8jCaYh24+kJBhNInawAvcQeF/TewGrfTF9quJncGgEqHkxBJBTqYqI1lex9+uxzh3xFOMUP341hdP7uUMnV0XYcCulbGu7uK+U2shPmKoXX98S/4oOhsTyYHBM72T4vMu/Z2K+Iihql8+kujiLkE44VHX5v7VDMRHDFU9N4txqcGH1zkNp5affa6Mn3ElDW2t5uWqGIRXaE2GUSnz+PkYGsc+CkOhWb6GiyaqlwrFUIinI04MVl3TwYIVDOLtTzZYdRbhNtRKIznedp3S9JvihfpG5+dvNlR1KOgMX61caah7dbEv6xRfPdrSoEhZx9CuI95CWm6lQnx49aNJ3FpuNPfElo8YXMMz2L0kw43SQHzEmBqepw5taZD2OgdeVbzagSVlHbfmuqYXHvOyjl1rnSvo/PzRxoyMNvIKXLp0bvHXc4YM69GtR9sZMyfdjMh/IyIm5lGHsNC7927PnTcLFgYP7f7TutUqlQpppjravWfH+xOGd+3+5sRJI3/euAbSww/s6dKtjfYFhlXffg1bwU64r7AW9s+tPXL0wJQPxsBX+IT9aP3PPv3C9uz5fdrH78OG2dnZBp6/ZrAlP8u6oaoXjHA3FJlMtnjJnLy8vM8+XfD14tVeXj5fzPk4JSUZaaaug8+Vq74KC+t67MilLz7/atef2/45fRwS//pr57btvw4cMHznjoO9eg34+9C+nX9sad78Dblc/uBB/twIUdER7u41b9+J5L5G374V2ryVUCg8cfLIN8sW+NdvuGNb+PhxU0H1NWtXcnngiAcP7fXza7B82Y9GTPCeP/aWhxiqurEVPFzcjRt2zpzxRXBQKPxNmjg9NzcXBNNmeLtdx/ZvdwQ9mjUL8ahV+7//7kLircgbDRo06tKlp7OzS88e/X5cs/mNlm/W9vDUypyamhIbG9O5U4/IqPxZzKOjIkJCWsLCoUP7mjYNnj7tMxeXaiHBLcaOnrRv3y7IjzQjrx0dnT6cOiu0+RsCgcDAn0Dxt5fG4Bre+OfrOTnZP6xZPnBwV6hXodaFlLS0wiGn/v4B2mV7e4esLPULU02aNLt+/d9lyxdCXZ2ekQ56+/n5Q3rzkDeio2/BAohd369BcHCLO7fVN0FS0suExHjQkmEYKPQtQltr9wl5IFF7czTwb4SMhcK+l8ZYx+bFi8RpH48PCW4594uvGzUKhNLWqUsr3Qx65ymEut3W1u7CxTNQV0Ol3b59p4nvf1S9eg2QEG4gpH5D8XpgYHCjgMDEFwkgecSt625u7nXqeINBUSgUv/y6Fv50d8iVdUAsFiMjIS03o5++nD5zHIwxGHVuKgPdUl4GcCtAxQ5/T548vnHjyuYtG7Kzs77+6tsWLVpnZKRDsYayO+rd9yUSCRgCsBfR0RFwYyGNQbG1tYWav127Im/Me9TyRJUA93541sgxZCCSg4OjdvaKM2cNmrfi6NGDUPPXrVvPx8cX/jKzMv8+tBfSnRyd/Or5X7xw5tGjB82aqmeaCGwSFBV18/qNK2PHTOK2rVfPH/KDD8F9haKfkBAHNQGqMCziay+0wd6ckT/f17d+cvIraFZBm+rfKxeh4Do5Ob98mVj2VidPHZn35ScXL54Fo3758vlz5081adyMWwWV/F97d8KtAPuBr5D+778X4uKegVHnMrw/7oMLF04fOrwfzHlUVMTCRZ/PmDUJ6htUYUg/vLFOfNg7XWJjH2/Z+vO3q5e0CG316ewvoQ224/fNmZkZgweNLG2rmTPmrPlxxRdzZ8BytWquUNUPGpifGdzyP3dv790rf3aJwMAgqPDBs+NuAi5lw7rt23dsWr/he5kst3Gjpl8tWgW2AFUY/nZMGfp240+fPKrXzL51L96+5lmS+9czLh14+eG3foh3GGzXseuG5zOGj4ymWAZhBW974Y0ZLclShnZq8QQW8Xb4kDHtdczKumYAEeIlBtfwNH/7LPDDiHFz2D1f569hN9yuU9iNm+Nvu8WYtxuxs+u8xai3G/Gq48m7L+prgFtRZ8kzN3X1jl+sEL7+YmPeaSUNN75A3mnFEUNVF0sFuL0AS1ECsYSf4WgNFdLeSZD2UoZwIv5BttSWn88eDL2X2/V3T3uZh3AiMSanyZsuiI8YqrpHPXFAqNPvS2MQHvyxPLa2v03zMEfER4ybH/7asbQrJ5Lt7MU2DnRenqrIjjRzvNNCxBTMO0pDE187LTz8MWyxgcaaPl42f2Z/dUJhhvz2QkG0AIrWzDGvM8k84rbKn4KeeyyKdCeczz+gzmkgTdABVh2NAHFHU88Sz6p7n7TjIqVSQW4Wk50hr9/MseOIGoinGB0LQpWFDm1LyEhV5OYUVV0jg0BIqZTaa58fn4HWzFnG6E79wEmkO7M/RTGas9Gu5x5uwyqVSsUyjFgi4mLKUFzYAQpptuJ2Q2mGrrPaF5QoTRXGqUvTSBuMhtJEiGAYtshZUJqja/JIbWh7Z9GbPd1rePF5NIEVxGndunVrSkrKtGnTEMFEWEFjTKFQcC9EEkyFFaiuVCrxjL9lPqygFwLsOlHdtFiB6qSsmxzrsOtEddNC7DqOENVxhKiOI0R1HCG9NDhCyjqOENVxhKiOI8Su4wgp6zhCVMcRojqOENVxxDpUJ96caSFlHUeI6jhCVMcRMpYGR6zgatra2uoNIUCoMFagulwDIpgOK1AdqndtWC+CSSCq4whRHUeI6jhCVMcRojqOENVxhKiOI9ahOhenm2AqSFnHEaI6jhDVcYSojiNEdRwhquOIFYxW0FV9yJAhiFBpLHduyb59+z59+hSh/KlFWQ1NmjTZunUrIlQOyy3rnTp1glJO0zSlARYcHByGDRuGCJXGclUfPny4l5eX9ivDMJ6ent27d0eESmO5qru4uPTs2VMikXBfpVJpv379EMEUWLQ3B/U5lG9uGco9Ud1UWLTqUNAHDRokFovBqPfq1YuMijcVpvThH0XlPL2bk54iV+UxMnmRUI9CAaVUFTkQOGmMzqEppJmany1xchQVGxvLsoyXlzdNa8bFs0U21OyKCxRRJNCEJhgAWzLuplRMC8QCB2dBbT/bhi3sEZaYQPXz+5IfRmbmZsI1ZgUCWhNbAbFFNdYNz1HamXARHXRhNWFFuJX5q6hS4iSXPEApOWmBOjKFSsWwjPrmkdoKvBvadhzuhnCiUqof2/byUWQm7MTW2baGj6OtiwRZFblZqlePkrNSZYxS5dXArteEWggPKqh6Thbauvgx1L/ufq4ute2QlZP1Ki/+bpJKqeo6ulbdxjaI71RE9StH0q6eeOXi4ewRwKtwZ8lPMl88SvZtZt/1XXfEa4xW/cUT+V8/Pg94xxvxlPunn7bpVT2wrQPiL8apfmZPyp3LaTyWnOPe6acevtLeE3lr5o1or8fczr2NgeRAw/ZecY9zL4QnI55ihOqHN8V7NsalhRPQ3jvidBriKYaqvuObZxJbkaM7//1bLXauNr/Of4L4iEGqZ6UwqS/y6rWujXDCJ8RdlqO6ezkb8Q6DVA/f8FxiJ0b4Ab1PFw8lId5hkOqprxQ1Ayw3QPGeA8uW/2CW0RY+IW6ybCXi3Ys35at+9VgadIfbu+BY1gFaSB/b/gLxi/KfXT6OyhJK8H3EKbWXJD7NQ/yifDmz0pRSOzM+Vrl64+Clq3sTXjys5e4XFNjxrdZDueGR85d06RI2ITsn7dipjRKxTYP6rfp0m+HoWB1W5eXlbN897+Hja7BJ6xb9kTmxc5amxqUjflF+Da9QMFJHc6l+49bRP/Yu8vRo8L8Ze7t1mnz24s79h77lVgkEotPnt1EUvfDzY7M/2hUTe+voPz9zq3btW/wq+dnEMWtGD/sm8eXje/9dQGbDzsVGpbL0EPXGUr7qLMOKpOaq4a9c3+/rHdy/12wH+2r1fUOhcF/498/MrBRubfVqnh3fHmtj4wBFvIFfq+dx9yAxPSPpVvSJDm3f9a7TxNHBtWeXD0RCKTIbIgcJo2IQvyhfdfWAFNYsP5thmJinkf7139CmgPAsy8Q8ieC+etYO0K6ysXGU5WXBQkpqHHy6u9XVrqqjk83kCATIKt4VMYryC7GAphgFMgdKpVylUhw5sQ7+dNMzs1MKFqmSW2XnqK2sRGyrTRGLzdhjqMhUUBTiGeWrTtEoJzPPFZn+yaNYLAXxmgd1b9r4Hd1012pldQLa2TrBp1wh06bI8szYfZaVKqN4N4lt+apLbOjcDBkyDx61/HNlmX6+zbmvSqUiOTXO2amsQQ0uzh7w+eRpJFexwyYPHl2xszPX+I7s9FyprQDxi/Jv4zr+tso8c71S2r3T5Oi7Z/69Hq628bER23Z9sX7TVKj5y9jE2cnNx6vZ0VMbXibFKhR52/+ci8xZBeemy5xr8C3+SPmqhw11UynN5cTW9Q76ePIWcN++/Kbr+s0f5sqyxo5YLhKV01AcNmC+l2fj1T+N+uKrDrY2ji1DeiPWXI0r8D1adHZF/MKgsTQ/fxEjspH4NOf5aLKSJNxNSYnPmLqiHuIXBjkqoZ2qZaflIvxITcjwD+bhADqDul+C2ztdOZr8LDKpTlP9T94uXtlz6PhavavA9JZWYw/tP69JwNvIRIBb8Mu2mXpXgaMAPX2UPvM/pN/cwEbt9W718nE61IOdRvBw+JChoyVTE5Tblz9p0qmu3rUKpVyp0P+IAppYYpH+vjNoZwsEpuz1y83N1Jtexp0ngn5HoX5nLfpEzFt9ajRr54R4hxFjZA+sT3z+ODegvRfCgP8uxtnaUSM/q4P4iBEdEL0m1rRzoB9eikN8J+baC4pl+Co5qsBbEPvXJybGyhq8xdsr8uhKoljMvPs5b38gqsBzChFBnQAAAU5JREFUhT4Ta9rY0ffPPkN85MHF50il4LfkqMJvNx7f8vK/W5n21e28gyx3PJ1RPI9+lf4iy8PHpt8HHojvVPxNZnkW2rYiNjdLaeMgqRXgZuNglZ3VSjnzPCopJz1XJKF7jKnp4YfFgP/KzlrwOCLnbHhSVrqSpimhiBbbSoRSWiCgGaTTicvSiMr/yqo7zRl9j1Bp9VP8YsmsZjoCbvYBVrMpq/lKaddqPgqnpKC4ienym+asZiYEVLBa8y/NqidCUOSqZLlyRqFSqVgbO0HLTq6Bb/H5dcZimGyGkmvH02LvZWelqRR5DPTbKxU6s4/QusMyWG7ewPw5KLQL6qlHSvSmUxrddGar0J2SoiBFo3uRveXLnP9VPQVG4d0kECG4KcUSga0j7eln27pnNYQflju3JMF8kFmdcISojiNEdRwhquMIUR1HiOo48n8AAAD//+Q1qggAAAAGSURBVAMARZWkVTAaZewAAAAASUVORK5CYII=", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import Image, display\n", "arch = CorrectiveRAG(\n", " llm=llm,\n", " documents=STARDUST_CORPUS,\n", " web_search_fn=web_search_fn,\n", " top_k=3,\n", " relevance_threshold=0.5,\n", ")\n", "graph = arch.build()\n", "try:\n", " display(Image(graph.get_graph().draw_mermaid_png()))\n", "except Exception as e:\n", " print(f\"(mermaid PNG render unavailable: {e}; see § 2)\")\n", " print(graph.get_graph().draw_mermaid())" ] }, { "cell_type": "markdown", "id": "91abee4e", "metadata": { "papermill": { "duration": 0.008188, "end_time": "2026-05-28T02:39:04.197488+00:00", "exception": false, "start_time": "2026-05-28T02:39:04.189300+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 8 · Live run — 3 task types with varying corpus coverage\n", "\n", "1. **In-corpus** — answer is in the Stardust corpus.\n", "2. **Out-of-corpus** — answer requires web search.\n", "3. **Mixed** — both corpus and web add value." ] }, { "cell_type": "code", "execution_count": 4, "id": "eae1540b", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:39:04.206884Z", "iopub.status.busy": "2026-05-28T02:39:04.206884Z", "iopub.status.idle": "2026-05-28T02:39:54.245341Z", "shell.execute_reply": "2026-05-28T02:39:54.245341Z" }, "papermill": { "duration": 50.045025, "end_time": "2026-05-28T02:39:54.245341+00:00", "exception": false, "start_time": "2026-05-28T02:39:04.200316+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TASK_TAG: in_corpus\n", " TASK: What propellant does the Phoenix-2 engine use?\n", " N_RETRIEVED: 3\n", " N_RELEVANT: 1\n", " N_AMBIGUOUS: 0\n", " N_IRRELEVANT: 2\n", " RELEVANCE_FRACTION: 0.33\n", " ROUTE: use_mixed\n", " N_WEB: 1\n", " FINAL_ANSWER: The Phoenix-2 engine uses methalox propellant, which is a combination of liquid methane and liquid oxygen.\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "TASK_TAG: out_of_corpus\n", " TASK: What is the current population of Iceland (2024)?\n", " N_RETRIEVED: 3\n", " N_RELEVANT: 0\n", " N_AMBIGUOUS: 0\n", " N_IRRELEVANT: 3\n", " RELEVANCE_FRACTION: 0.00\n", " ROUTE: use_web\n", " N_WEB: 1\n", " FINAL_ANSWER: I don't have enough information.\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "TASK_TAG: mixed\n", " TASK: Compare the Stardust 9 rocket's payload to SpaceX Falcon 9's payload to LEO.\n", " N_RETRIEVED: 3\n", " N_RELEVANT: 2\n", " N_AMBIGUOUS: 0\n", " N_IRRELEVANT: 1\n", " RELEVANCE_FRACTION: 0.67\n", " ROUTE: use_retrieved\n", " N_WEB: 0\n", " FINAL_ANSWER: I don't have enough information.\n", "\n" ] } ], "source": [ "TASKS = [\n", " (\"in_corpus\", \"What propellant does the Phoenix-2 engine use?\"),\n", " (\"out_of_corpus\", \"What is the current population of Iceland (2024)?\"),\n", " (\"mixed\", \"Compare the Stardust 9 rocket's payload to SpaceX Falcon 9's payload to LEO.\"),\n", "]\n", "\n", "for tag, q in TASKS:\n", " r = arch.run(q)\n", " print(f\"TASK_TAG: {tag}\")\n", " print(f\" TASK: {q[:80]}\")\n", " print(f\" N_RETRIEVED: {r.metadata['n_retrieved']}\")\n", " print(f\" N_RELEVANT: {r.metadata['n_relevant']}\")\n", " print(f\" N_AMBIGUOUS: {r.metadata['n_ambiguous']}\")\n", " print(f\" N_IRRELEVANT: {r.metadata['n_irrelevant']}\")\n", " print(f\" RELEVANCE_FRACTION: {r.metadata['relevance_fraction']:.2f}\")\n", " print(f\" ROUTE: {r.metadata['route']}\")\n", " print(f\" N_WEB: {r.metadata['n_web']}\")\n", " print(f\" FINAL_ANSWER: {r.output[:200]}\")\n", " print()" ] }, { "cell_type": "markdown", "id": "0c1e1df2", "metadata": { "papermill": { "duration": 0.006707, "end_time": "2026-05-28T02:39:54.256657+00:00", "exception": false, "start_time": "2026-05-28T02:39:54.249950+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 9 · What we just observed\n", "\n", "The cells above ran CRAG on **3 task types** (in-corpus, out-of-corpus, mixed) to exercise the grade-then-route logic.\n", "\n", "### 9.1 · Per-task retrieval, grading, and routing\n", "\n", "| Tag | Retrieved | Rel/Amb/Irr | Rel% | Route | Web docs | Final answer |\n", "|---|---|---|---|---|---|---|\n", "| `in_corpus` | 3 | 1/0/2 | 33% | `use_mixed` | 1 | The Phoenix-2 engine uses methalox propellant, which is a combination of liquid … |\n", "| `out_of_corpus` | 3 | 0/0/3 | 0% | `use_web` | 1 | I don't have enough information. |\n", "| `mixed` | 3 | 2/0/1 | 67% | `use_retrieved` | 0 | I don't have enough information. |\n", "\n", "### 9.2 · Patterns surfaced in this run\n", "\n", "- **🤔 `in_corpus` task** — relevance fraction 33%, route `use_mixed`. Expected high relevance + use_retrieved. Either the grader is over-strict or the retrieve missed the relevant doc.\n", "\n", "- **✅ `out_of_corpus` task** — corpus correctly graded irrelevant (0%), routed to web fallback.\n", "\n", "- **🤔 `mixed` task** — routed `use_retrieved` not `use_mixed`. Hybrid coverage wasn't recognised.\n", "\n", "### 9.3 · The takeaway\n", "\n", "CRAG's two columns to watch in § 9.1: **`Rel%`** (the deterministic-picker input) and **`Route`** (the Python-composed output). The architecture is working iff:\n", "\n", "1. **In-corpus tasks** show high Rel% → `use_retrieved`.\n", "2. **Out-of-corpus tasks** show low Rel% → `use_web`.\n", "3. **Mixed tasks** show moderate Rel% → `use_mixed`.\n", "\n", "If routes don't match expectations (§ 9.2 flags), the grader is mis-calibrated — usually too lenient (calls weak-matches `relevant`). Tighten the grader's prompt or add a second-pass confirm step." ] }, { "cell_type": "markdown", "id": "d6d96c0c", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:39:54.265093+00:00", "exception": false, "start_time": "2026-05-28T02:39:54.265093+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 11 · Failure modes, safety, extensions\n", "\n", "### 11.1 · Where this breaks\n", "\n", "| Failure | Mechanism | Mitigation |\n", "|---|---|---|\n", "| **Grader hallucinates `relevant`** | LLM thinks an off-topic doc relates because of surface keyword overlap | Add a second grade pass; require the grader to QUOTE the relevant sentence |\n", "| **Web fallback unreliable** | Tavily down or rate-limited | Multiple fallback sources; cache web results |\n", "| **Threshold mismatch** | `relevance_threshold=0.5` too lenient — proceeds with mostly-irrelevant docs | Tune per corpus; A/B test |\n", "| **Cost** | top_k grade calls per query | Cache grades by `(query_hash, doc_hash)`; batch grades |\n", "\n", "### 11.2 · Production safety\n", "\n", "- **Don't trust web fallback for high-stakes answers.** Tavily snippets are arbitrary web text; treat as \"loose context\" not \"authoritative source\".\n", "- **Track route distribution.** If `use_web` dominates, the corpus is failing — index more.\n", "- **Audit irrelevant-but-graded-relevant cases.** False positives leak hallucinated answers.\n", "\n", "### 11.3 · Three extensions\n", "\n", "1. **Query rewriting before web fallback.** Use a small LLM to rewrite the query for web-search-friendliness.\n", "2. **Confidence-weighted answer.** Each grade has a confidence; weight the doc's influence on the answer.\n", "3. **Per-doc citation requirement.** Force the answer to cite which doc supports each claim.\n", "\n", "### 11.4 · What to read next\n", "\n", "- [**23 · Agentic RAG**](./23_agentic_rag.ipynb) — sibling, doesn't grade.\n", "- [**25 · Self-RAG**](./25_self_rag.ipynb) — reflection-token version.\n", "- [**26 · Adaptive RAG**](./26_adaptive_rag.ipynb) — task-level routing.\n", "\n", "### 11.5 · References\n", "\n", "1. Yan, S. et al. *Corrective Retrieval Augmented Generation.* 2024. [arXiv:2401.15884](https://arxiv.org/abs/2401.15884)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" }, "papermill": { "default_parameters": {}, "duration": 62.684149, "end_time": "2026-05-28T02:39:55.168091+00:00", "environment_variables": {}, "exception": null, "input_path": "all-agentic-architectures/notebooks/24_corrective_rag.ipynb", "output_path": "all-agentic-architectures/notebooks/24_corrective_rag.ipynb", "parameters": {}, "start_time": "2026-05-28T02:38:52.483942+00:00", "version": "2.7.0" } }, "nbformat": 4, "nbformat_minor": 5 }