{ "cells": [ { "cell_type": "markdown", "id": "f1d34a83", "metadata": { "papermill": { "duration": 0.00603, "end_time": "2026-05-28T02:47:31.578902+00:00", "exception": false, "start_time": "2026-05-28T02:47:31.572872+00:00", "status": "completed" }, "tags": [] }, "source": [ "# 25 · Self-RAG — retrieve-on-demand with reflection tokens\n", "\n", "> **TL;DR.** Decide whether to retrieve; if yes, fetch top-k docs; emit **per-doc reflection tokens** (categorical `is_relevant`, `is_supported`, `is_useful`); Python composes a keep/drop boolean per doc; answer from kept docs only.\n", ">\n", "> **Reach for it when** you need finer-grained doc-level quality control than CRAG's whole-batch routing, and want explicit per-doc audit signals.\n", "> **Avoid when** retrievals are usually clean (the per-doc reflection cost is wasted) or when the corpus is small enough that reflection cost dominates.\n", "\n", "| Property | Value |\n", "|---|---|\n", "| Origin | Asai et al., *Self-RAG* (2024). [arXiv:2310.11511](https://arxiv.org/abs/2310.11511) |\n", "| Reflection tokens | 3-way categorical per doc: `is_relevant`, `is_supported`, `is_useful` |\n", "| Picker | Python composes keep/drop from token labels — deterministic-picker |\n", "| Default LLM | Llama-3.3-70B |\n", "| Cost | 1 decide + 1 retrieve + `top_k` reflect + 1 answer = `top_k + 3` calls |\n", "\n", "**Why deterministic-picker is the load-bearing pattern here.** Self-RAG's reflection tokens were originally LEARNED special tokens (the paper trains a model to emit them). We simulate them via Pydantic structured output. The key fidelity-preserving move: every token is `Literal[...]` — categorical — not a numeric score. Python composes the keep/drop boolean as `is_relevant != 'not_relevant' AND is_supported != 'no_support'`." ] }, { "cell_type": "markdown", "id": "b3b667bd", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:47:31.578902+00:00", "exception": false, "start_time": "2026-05-28T02:47:31.578902+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 2 · Architecture at a glance\n", "\n", "```mermaid\n", "flowchart TB\n", " A([task]) --> D[DECIDE_RETRIEVAL
bool: parametric or external?]\n", " D -->|False| ANS[ANSWER from parametric]\n", " D -->|True| R[RETRIEVE top-k]\n", " R --> RF[REFLECT per doc
3 categorical tokens per doc]\n", " RF --> CK[COMPOSE_KEEP
Python boolean per doc]\n", " CK --> ANS\n", " ANS --> Z([final])\n", "\n", " style RF fill:#fff3e0,stroke:#f57c00\n", " style CK fill:#fce4ec,stroke:#c2185b\n", " style ANS fill:#e8f5e9,stroke:#388e3c\n", "```" ] }, { "cell_type": "markdown", "id": "132b98c1", "metadata": { "papermill": { "duration": 0.008879, "end_time": "2026-05-28T02:47:31.587781+00:00", "exception": false, "start_time": "2026-05-28T02:47:31.578902+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 3 · Theory\n", "\n", "### 3.0 · The three reflection tokens\n", "\n", "Per Asai et al.:\n", "- **`is_relevant`** — does this doc address the question?\n", "- **`is_supported`** — would using this doc produce a well-grounded answer?\n", "- **`is_useful`** — overall usefulness signal.\n", "\n", "Each is a 3-way categorical (`fully_X` / `partially_X` / `not_X` or `no_X`). No numeric scoring.\n", "\n", "### 3.1 · How Python composes the keep/drop\n", "\n", "```python\n", "def _compose_keep(state):\n", " return [\n", " i for i, t in enumerate(state['reflection_tokens'])\n", " if t['is_relevant'] != 'not_relevant'\n", " and t['is_supported'] != 'no_support'\n", " ]\n", "```\n", "\n", "This is the deciding signal — Python AND on two categorical commitments. The LLM never emits a numeric usefulness score.\n", "\n", "### 3.2 · Where this sits in the RAG family\n", "\n", "| Pattern | Granularity of quality control |\n", "|---|---|\n", "| Plain RAG | None |\n", "| [Agentic RAG (nb 23)](./23_agentic_rag.ipynb) | None (agent just answers) |\n", "| [CRAG (nb 24)](./24_corrective_rag.ipynb) | Whole batch (route based on aggregate) |\n", "| **Self-RAG (this nb)** | **Per-document** (drop individual junk docs) |\n", "| [Adaptive RAG (nb 26)](./26_adaptive_rag.ipynb) | Per-task (no/single/multi-step routing) |" ] }, { "cell_type": "markdown", "id": "b9c6549e", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:47:31.588788+00:00", "exception": false, "start_time": "2026-05-28T02:47:31.588788+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 4 · Setup" ] }, { "cell_type": "code", "execution_count": 1, "id": "6583fdb7", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:47:31.588788Z", "iopub.status.busy": "2026-05-28T02:47:31.588788Z", "iopub.status.idle": "2026-05-28T02:47:33.494007Z", "shell.execute_reply": "2026-05-28T02:47:33.494007Z" }, "papermill": { "duration": 1.905219, "end_time": "2026-05-28T02:47:33.494007+00:00", "exception": false, "start_time": "2026-05-28T02:47:31.588788+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
LLM: meta-llama/Llama-3.3-70B-Instruct  ·  Corpus: 12 docs ────────────────────────────────────────────────────────\n",
       "
\n" ], "text/plain": [ "\u001b[1;36mLLM: meta-llama/Llama-\u001b[0m\u001b[1;36m3.3\u001b[0m\u001b[1;36m-70B-Instruct · Corpus: \u001b[0m\u001b[1;36m12\u001b[0m\u001b[1;36m docs\u001b[0m \u001b[92m────────────────────────────────────────────────────────\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from agentic_architectures import get_llm, enable_langsmith, settings\n", "from agentic_architectures.architectures import SelfRAG\n", "from agentic_architectures.data import STARDUST_CORPUS\n", "from agentic_architectures.ui import print_md, print_header\n", "\n", "enable_langsmith()\n", "llm = get_llm(provider=\"nebius\", model=\"meta-llama/Llama-3.3-70B-Instruct\", temperature=0.2)\n", "print_header(f\"LLM: {llm.model} · Corpus: {len(STARDUST_CORPUS)} docs\")" ] }, { "cell_type": "markdown", "id": "b3f05da2", "metadata": { "papermill": { "duration": 0.002008, "end_time": "2026-05-28T02:47:33.498814+00:00", "exception": false, "start_time": "2026-05-28T02:47:33.496806+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 5 · Library walkthrough" ] }, { "cell_type": "code", "execution_count": 2, "id": "507329e2", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:47:33.503312Z", "iopub.status.busy": "2026-05-28T02:47:33.503312Z", "iopub.status.idle": "2026-05-28T02:47:33.509682Z", "shell.execute_reply": "2026-05-28T02:47:33.509682Z" }, "papermill": { "duration": 0.008375, "end_time": "2026-05-28T02:47:33.509682+00:00", "exception": false, "start_time": "2026-05-28T02:47:33.501307+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--- _ReflectionTokens schema ---\n", "{\n", " \"description\": \"Per-document reflection tokens \\u2014 all categorical (deterministic-picker).\",\n", " \"properties\": {\n", " \"is_relevant\": {\n", " \"description\": \"Does this document address the question?\",\n", " \"enum\": [\n", " \"fully_relevant\",\n", " \"partially_relevant\",\n", " \"not_relevant\"\n", " ],\n", " \"title\": \"Is Relevant\",\n", " \"type\": \"string\"\n", " },\n", " \"is_supported\": {\n", " \"description\": \"If you used this document as evidence, how well-grounded would the answer be?\",\n", " \"enum\": [\n", " \"fully_supported\",\n", " \"partially_supported\",\n", " \"no_support\"\n", " ],\n", " \"title\": \"Is Supported\",\n", " \"type\": \"string\"\n", " },\n", " \"is_useful\": {\n", " \"description\": \"Ov...\n", "\n", "--- _compose_keep (Python) ---\n", " def _compose_keep(self, state: SelfRAGState) -> dict[str, Any]:\n", " \"\"\"Python decides which docs to keep — pure deterministic-picker.\"\"\"\n", " tokens = state.get(\"reflection_tokens\", [])\n", " kept: list[int] = []\n", " for i, t in enumerate(tokens):\n", " # Keep iff relevant AND has support\n", " if t[\"is_relevant\"] != \"not_relevant\" and t[\"is_supported\"] != \"no_support\":\n", " kept.append(i)\n", " return {\n", " \"kept_doc_indices\": kept,\n", " \"history\": [{\n", " \"stage\": \"compose_keep\",\n", " \"kept\": kept,\n", " \"dropped\": [i for i in range(len(tokens)) if i not in kept],\n", " }],\n", " }\n", "\n" ] } ], "source": [ "from agentic_architectures.architectures.self_rag import _ReflectionTokens, SelfRAG\n", "import json, inspect\n", "print('--- _ReflectionTokens schema ---')\n", "print(json.dumps(_ReflectionTokens.model_json_schema(), indent=2)[:700] + '...')\n", "print()\n", "print('--- _compose_keep (Python) ---')\n", "print(inspect.getsource(SelfRAG._compose_keep))" ] }, { "cell_type": "markdown", "id": "4e435e3d", "metadata": { "papermill": { "duration": 0.0, "end_time": "2026-05-28T02:47:33.509682+00:00", "exception": false, "start_time": "2026-05-28T02:47:33.509682+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 7 · Build the graph" ] }, { "cell_type": "code", "execution_count": 3, "id": "1afbdd2c", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:47:33.519444Z", "iopub.status.busy": "2026-05-28T02:47:33.519444Z", "iopub.status.idle": "2026-05-28T02:47:40.729631Z", "shell.execute_reply": "2026-05-28T02:47:40.728300Z" }, "papermill": { "duration": 7.221002, "end_time": "2026-05-28T02:47:40.730684+00:00", "exception": false, "start_time": "2026-05-28T02:47:33.509682+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPcAAAJ2CAIAAAAMnYNQAAAQAElEQVR4nOydB2DTxtvGT3L2TggZJGSxd4AwCpQVdtl7Q1vKhrILlD0LZbSFP2W1lPkBZbeMllk2BUogYWYBCRmE7OXEtvS9thPjJHaIg6f0/khd+XQ6yfaj03PvnU5mLMsSBOE0ZgRBuA6qHOE+qHKE+6DKEe6DKke4D6oc4T7GpfL01NyQy5kpCfl5ORKGIaI8lqYphmEJRdEUgQX5W3iFzLBAwf+l/xG2YBkWyPsFmmKl2xKaohTbwlqInRYUK6NwF1BKwduCwmUJFC39nyLcCmtZtkj01cyCCAS0pTXl5mMd2NrB1tGCIEYGZQzx8txM8fEtb9LfiSRiIjAjVnYCcwupriV5hBIQViJVLVUg3AL5SnUsYeXpRKpypTyUTK5sQeZi28rVXLAsB3RcmE7ksibS04MhLC09vYqoXLohW5BTjsBSeo6JhGxerkQsIrSAuHlb9J/mQxCjwfAq370sOjNVYm1H125u98lnbsTEuX7ibXhIVnY64+RKD/82gCBGgCFVfm53fERItquX+eBZvoRzHFjzMiVBXKeFXbsBHgQxKAZT+e7l0Xk5zMhFflbWAsJR3sXlHtsUZ+NAD5/nTxDDYRiV//7Da7GIHTKbg1V4SXavjHRxtewxzpsgBsIAKv91cbSlDTXsGz/CG3aviIavetQCP4IYAproF3CrVtb8kjgwaoE/hHCO/PiaIIZAryq/fTYpI1k8dK4f4R8jF/i9jckPu51GEL2jV5X/dzG9dT9XwleCOjhdO/qOIHpHfyo//r9Yc0uqdjMnwleadnE1M6f+2htPEP2iP5W/iRQ271aB8Jt6rRyjQrMJol/0pPLbZ96ZmZF6Lflbkctp3s2VZcjj2+kE0SN6Unn4f1kOFc2Jfjl8+PDixYuJ5nTs2PHNmzdEN9g7mT26hm1QvaInlWdliKvUtSH65cmTJ0Rz4uPjU1NTic6oVNUaAk0E0SN6GnkrEZOazRyJbnj58uXWrVvv378PPVz169cfOXJkYGDg2LFj//vvP1h7+vTpffv2eXt7w+utW7ciIyNdXV3btGkzYcIEKysryDBnzhyBQODp6blnz55x48Zt27YNEnv16gV51q9fT7RNtYZ2z+9mEkSP6EPlcdHS9paji04GXufn54OgmzRpsmnTJhDrjh07pk+ffvbs2e3bt48ePdrX13fp0qWQbefOnb/99tuKFSucnJwyMzO///57yDx16lRYZW5u/uLFi+zs7A0bNtSrV69WrVrTpk07efKkl5cX0QE+NWwZluRn5VvY4Uh0PaEPlacmimmdOaNXr16lpKQMGTKkZs2a8Pa7776DKlwsLm4Jhg8fHhwc7O9fMGrq4cOHN2/elKucoqi4uLi9e/fKq3Y9AF/G2ziJd3WC6Ae9OBYJRXSGj4+Ps7PzkiVLunXr1rhx4wYNGgQFBZXMBhU22BVojEK1LT8HXFxcFGtB/XqTOMBS8IeTPekPfbQ+7SvqcGytpaUluJRWrVodOHDgyy+/7N2795kzZ0pmAz8DHqZPnz4nTpy4d+/e559/XqwQokcgmOjihnfc6g99qNynui0jIaI8CdENfn5+4KT//PNPMNZVq1ZdtGjRs2fPlDNAq/To0aODBg0ClXt4SO9pAGtODERclLSVgreH6hM9RRJpAQm9qZOuEAiwnDp1ChbAcrRu3XrNmjVmZmZPnz5VziMSiXJzc93cCm63gwbr1atXiYF4EZJFYz2uX/Skcms7OvJhFtEB6enpy5Yt++GHH2JiYqAlumvXLrDd4M5hVeXKlcPCwu7evZuVlQX1PZwMsbGxaWlpkB9CjRkZGRBXKVkg5ITX8+fPw7ZEB7x6nG1jr+8BzzxHT1+3T03rtzH5RAeAoOfPnw+hQ3Aj/fr1e/DgAcTOAwKktxX37dsX4ieTJk0KDw9ftWoVVPb9+/cH4960adPJkyfD2w4dOkB0pViBEFnv0aMHFAJWnuiAzBSmVlNddR0gKtHfvUKbZ0T0mlCpcjV994AaFU9up145mjzx+6oE0SP6u3S6uJlf2J9I+M3NP1LdvPUaz0GIPufWGjrXd/P0iOSEvAoeqn9miIEkJqo4DSQSCU3TFKU66A6RQejOJDogJCQEQjcqV5V+SJcuXaJVdYNFhWUKhUz/rysTRL/o9e7mM7++iQ0Xjl1dReVaaCOW42Ds7e2JzihfwFHdIW2ZHVEryK7dIJyeRd/o+x7+XxZGVfCy7D1eJ0NEjJkjP8ZkZ4hHLcSJWQyAvkNaXy4PSIjOvXQogfCJP7e/TknMR4kbCsPMOrTj2yjv6pZdR/GiRj/xc2za2/zRi3HORINhsBnktn4TYecoGD6f49XbnhUv83KZr1aixA2JIWcD3f/dy9REca1mtsGDPQnnOLc7PvJhdkVvi4EzcJZnA2PgmZ2f3E27fPAdyxB3P4uOQ92dKpp8LPltbO71E+/io/MEZlTnkW7+dXQYAkLKiFHM0n/v75R7F1PE+dI5+a3taXtHc2s72sJKIFZze6SZgIhVDXCkpIO2qWIfSPpkCgFhJMVySp8qIZGwxTZnWapYNkKIcoEC2Iphix4MJcqX5GZJMlLEwhwJ7MjOSdCgrWPD1i4EMQ6MQuUKbp9Nin2Rm5kmkYhYqODFItXHJjCjJWJGxQoKPg8psQ1FS1UuTWZkKpb35tACiimm8hJbyyWvnCQQQJdQkdLNLeAsogXmxM7RzKeGTdPOfJ9zxggxLpXrmo0bN1asWHH48OEE4RP8GuksFovNzHBwN+9AlSPcB1WOcB9+/eQikcjcXN8T2SEGB+tyhPugyhHugypHuA+qHOE+qHKE+6DKEe6DKke4D6oc4T7YK4RwH6zLEe6DKke4D6oc4T7oyxHug3U5wn1Q5Qj3QZUj3AdVjnAf3qkcW588hEcqB4kLBDp88ihitPBI5SzL+vr6EoR/8Ejl4MijoqIIwj949OBJiqJompZIdPUIacRo4dfjVaE6F6ubYhThLqhyhPvwK5KIKucnqHKE+6DKEe6DKke4D6oc4T6ocoT7oMoR7oMqR7gPqhzhPqhyhPugyhHugypHuA+qHOE+vHh2c6dOnZKSkigZbCH169ffs2cPQXgAL0beNmnSRCAQ0DQtv5EClh0cHEaMGEEQfsALlYOgPT09lVP8/f07duxIEH7AC5XXrFmzRYsWircWFhYDBgwgCG/gy71CgwcP9vLyki9Xrly5e/fuBOENfFE5WJSWLVsSWZgFK3K+8eEYy+sX2eH/ZeYJZbkpIs8uX6ApwrBF0uXLhBTJBtA0YRj5alinWMXK7qyX5lHeSiCgJBJWUVpBCYQwhBRLlKZTLMNSJdNLFpKXJ7x37z4sNm/RgqYKT+/C45EhO44S34csMKNiF7K9U4zS+yKfnSXyjYptUrhhwVdHVH2oYt9zCVgLK6rWJ3ZevnYEKQMfUPkviyLycoi5JS3Kk2YDbbAyrRWovFC7BelUgU5ggVVOl6qcYmS/qvLvJ5sIqFDltHTjgswCwhROJ6GU//2hKooFBDQlYYqfEqS4yqXbwjFIJIz8YErqSfZG9glUfB/vTwXlXZMSYlX+7Erns4ovuVg5xY6k2PdcDCgNVJ4vZG0dBKMX+xPkQ5Sm8m1zI1y9zDqN9COIUfLHzuicVGbMiioEKRW1Kt/xbYR3NatWfbwJYsRcPBCbHJf35XIUemmobn3e+vMteAaUuPETPNQ7T8iG3UomiHpUq/x1uNDKHuf5Ng2sbM0iQ3IJoh7VUhblMIQhiKmQm4W/VmmoVjmEIliGIogpwEpYBkVeKmhLTB5WdfQTeQ+q3OSBmD1N44W3NFSrXNoziNWDqUDJerMQ9ahWOXS84UXQZGD5cCfMR4GOxeRhWKiVUOalgSo3eajCUWKIOlT3CsnuHcNvzjTA3+mDqKnL8QJoOjAYSfwQqutyliYUKt1EQMfyQdTEWCTv7xtAjBzsFfog2Po0eaAFJcAaqVTUtD4F1Ef2pv3w43effzmQlIuoqIh2wUGPHj0ouerylfOwKi0tlRiaXn2C9+zdSbTN0WMHO3RqptEmspuTUOaloVrljMSQEVgnJ+eRI8a4uXkQgxIdHTl4qNpb/QcNHFG/XkNiBEjj5WhZSsUYHYuLS4XPR48nhub5iyelrB06ZDQxDij5LeKIerQ2U0VOTs63C2d06/7ppCmf//33aeVVYrF42/afwMB81qP1N/Om3r59XbEqIzPj+3XLwYT07tthxcpvExMTSAnHsnXbj337dxo+oveu37YWm8vz3F9/TJw8uutnreD1yNEDZanSwGkcPfp/X0//CnYBe4eUx48fzflmcs9e7UaM6rvl543Z2dmQCPtas3YpHA9k+/3IfvkhwZH3H9hlzNghpKhjUVnC3Xu3YZOwsIeKXT999lhayJ0bsHzs+CHYpEfPtv0GdF62fN6buFhSfngx2eXHoDWVr1u/PDb29brvf16+dF30y8jbd95L+adNa0GCfXoPOrD/jzatgxcvnfPP1YtEpv6586a+S07asH7rlMmz3yYlzp0/tZiOT546cvLU719P/WbLlj2enl579u5QrLpw8RwIsXq1mgf2nRrz5STYxeYt6z94nObm5n+eOV61ao3v1/7Pxtom9k3MrDkThXnCzZt2wZFHRYVPnzEWjgEuJoMHjXR397h88d6A/sNgK9h2z76dYFRmzligXKC6Eho1bGJvZ3/12iVFzuvXL0NKk6DmoaEhmzZ/X6dOg2XL1s39ZmlqasrKVQtIeaEoFscklo5qlVO0ZtfAd++SoF04ZPCo2rXqgt8YN3aqpaWVfFVeXt5ff/8J1/eePfo5Ojh269oruH0XuVjhTHj6NGzShBkNA4OC23eePGlWlSrVU1KK3MJ47PjBNq07wLnhYO/QpXMPkI5i1ZkzJ+rXbzjt67nOzi6Q/vmo8SdOHAbFlH6o8MEcHBynTJoV1LiZmZnZhQtnzc3MQZ0+Pn5+fgGzZi4Mj3h+/caVklvBKwgUFF+rZh3lVepKEAgE7dp1unrtoiInKD44uAuk165db9cvh4cN/Rw+OJQ5cMBw+B7SM9JJuYB6nMFxLKWipi5niEbfW3z8G3j19Q1QpNSoUVu+8OLF0/z8/CZBnyhWBTZoDAYAftTIyHAbGxsQhzwdauUF81e4ubkrcsKF+M2bGJCOIqV69VoFB8gwYY8fKhfbsGETSHwUqiIyU4wa1Wsrlh8/flizZh1HRyf5Ww8Pz0qVvNUVUr1arZKJpZTQtm1H8Dwvwp8RWVsWrnVwhhPpXDGCuLjYefO/7t6zDXiY+QumQ2Lah85PdaDAP4ia1ifN0pr0CqVnpMErGABFirWVtXwhKysTXqd8/WWxTVJTkrOzsxRVvkrA4EokEmulYq0Ki4UzRyQS/fLrFvgrUmwZtGJhYaFYhsN79vwJSK3Ysane0NKyZGIpJcD5DNeZq1cvwgl87frlihXd6tZtAOk3bvyzYNFMqMvHjf26SpVqWVbvqAAAEABJREFU9+7fAY9OEJ2hnfHljg7Smkwon2ZORk5OtnyhgmtFeJ0541svr8rKm0Cg0MbGNjc3BypgmlZ9SbG1tYVqL0+pWMgvX7CysoLrQKeOn7VuHay8SSVPzWbXcKngWq9eYLGQjvzjfHwJ4HPAtIB7gWYDmPKOHbrJ10LDADaBRPlbeUXwMaArLx3tRBI9PCrBK8QTasgcBdSyUD9B2BuWvb18LGVVIHhQeWaobsGKgEZr1qgtFAqfv3gqd7qvX7/c8MOqKZNmK9oEsODu7gkRDFI4fadyoxZMfGZWpqJY2CkYJ2XDUxaqBFT7+/zpBvUbKc60ly+jvL19tFVC+7adjh07CMEZMOvz5y2XJ2ZkpHu4v59P/ZpSC7UcSJue2PosFbUjbzX63uTX4t9+2xoT8wqamxATVCgV1Dx61DhobkJgAWwGRFcgIgE9o7AqKKg5VPDbt/8EV3OIu0Fi0ttEX98iE/+1a9sRGm3QtIXl/zu4+8mTUMWqr76cfOPGlTNnT8LVAAqHeNyMWeNhF0QT+vcfBptDcAbONzh4iHh+MWZQVHQErAKlJie/u379CqSXrwSgTp36cOJBXDIgoKqigVG1SnX4vA9C7kEoBsKU8sSExHhSLqDpiXdRlI6avk9G4+9t3txltWrVHTt+GATF7e0dIJaiCOJCSG72rEUHDv7Wo1fbH39aA6Zi5kxp4AxCHOvWboFdLVo8G4yplbX16lU/QqJyscOHfflZt94QdwPje+v2tYkTZhBS0NUHF/3tW/dDWL1Pv45w5oDLX7F8g6Uq61wKELr5ZechaEWMmzB85Oh+IQ/vz561EGw0rGrerFW9uoELF8+6eOmv8pUgp22bjtAAbd+usyLliy8mNmvaYsHCGZ26fALNUwgmwmUNgqoQGyWIDlDdobB7+UuWofpN8yWI0fP7+mhrO7MhcyoTRA04JtHkYTBe/iE4qHLoNle36ptvlrRq2ZZwC5yn4oOoVrl0Un2TrR22bz+gbpWzkwvhHBDaEfDlwTnlRE28nJjw/SeesrAmf5AwrATnSSwVdb1C0oegEAThBNj6NHmwNvogqHIOwOKIrdJBlZs80M1M4+3NpYIqN3kwXv5BUOUmD85f/kFQ5SYP1uUfBFWOcB9UOcJ9VKvcwlrAiiUEMQXMLSlLa/TlpaF6AIS1LREKUeWmQV6O2MYJVV4aqlXebqBrbhY2aEyA/Px8cR7pNgqfJV8aqlXuWMHaw99i/+oIghg3h9a+9q1jTZBSKW3ysTt/JT24lO7hb+NVzdraxuJDRbHFh1RQxecKgZ7oUgZDw4HQ6p8NoHZbqqCHm9K0TOmANEpVMlt8yiXYsSad6DSE9lQVXOIYVRw1RcoyTk4izJa8fp719nXep31c6jTn4HBi7fKBKfZun0t6ejsrL0ciFhEtwMr0Uk4o7U6wQ1HqBhdrsiNVpxdNk/I/MrwMO4f15hYEWpyN2js3aI0S/zDGNZHk69evJ0yYcPLkyWL3OBucvLy8du3a3bx5kyAmiHGp/Pbt282bNydGSUJCQlhYWIcOHQhiahiFysVi8ejRo/ft20cQRAcYxQ2DGzduXLx4MTEFpk+ffufOHYKYFAauy48cOdK/f39iUqxatWr27NnyGc0Rk8CQKl+/fr2Pj8+AAQMIgugSw6g8Li6uUqVKT548qV27NjFBLl68GBsbO2rUKIKYAgbw5ceOHTt69CgsmKjEgeDg4OzsbIgIEcQUMEBd/tNPP02dOpUgiL7QX10eHx9/6NAhWOCMxDMzMzds2EAQo0dPKhcKhV999VX37t0Jh7C3t2/QoMGiRYsIYtzow7GEh4e7u7s7ODgQBDEEuq3LGYYZOnSojY0NtyV+5syZlJRyPuEN0QM6rMvBpYSFhcFlvUaNGoTTwCeFqMuNGzcIYpToSuU7d+7s1atXxYoVCT8AoUNjlD+f17TQiWO5cuWKSCTi1U9uZWWVl5cXExNDEONDy3V5RERE1apVIWjo6elJ+MfEiROhQ7RZs2YEMSa0WZffv39/+XLpIy35KXFgy5YtUKMzDE6ab1xosy4/e/Zs165dCb8Ri8UZGRkuLnijmhGhhbo8NTV12rRpsIASJ7KHmJ4/f37t2rUEMRq0UJdPnz59wYIFFSpUIEghly9f9vX1DQgIIIgR8FEqh96Qbt26EQQxbsrvWMaMGePs7EwQNSQmJg4aNIggRkB56nIII1haWj5//pzznZofSUhICHxLqHWDo7HK3717t2vXrtmzZxMEMRE0dixZWVl4j0zZOXfuHPSUEcSgaDyFVaVKlTBMVnauXbsGr9AfTBDDYVxza3GPGzduuLu7o8oNi8aOJT4+fs6cOQQpGy1btkSJGxyNVQ4BlsjISIKUjStXroSFhRHEoGiscvTlGnH37l1UucFBX65b7t27Z2VlVbduXYIYDvTluiUoKAglbnDQl+uWO3fuQHVOEIOCvly3hIaGgjUniEFBX65boOkJV7/GjRsTxHCgL9ctYMpR4gYHfbluefjwobyTHzEg6Mt1S3h4+PXr1wliUDQerWVhYVGlShWClI0GDRrwdkYD4wF9uW6pVq1ay5YtCWJQ0JfrlufPn//9998EMSgaRxLz8/NjYmLQtJRO165dk5KSWBkUVfAlMwwTEhJCEL2jcV2OvrwsDBs2TCAQgL5pmpa/gtCht58ghgB9uU4YPny4t7e3coq9vT0kEsQQoC/XFaBpuO4p3latWrVt27YEMQQYL9cVffr08fX1lS+D3AcPHkwQA4G+XIcoqnN/f/9OnToRxEBo3CsEvnzjxo2mW53HhucIc8QUJZC9g9AHVbCicJGlpP8K0mQBEsVKSrYgzUIotuhmslVKpcnKqVm5bVCth3Fxb7q37Rf5KFvpKIrklL2Xlql2ddGjUpdHdsDiipUtHF2sCaKExio3XV9+YmtsXKQQlCERl5pPpXzKgqoNqzsMrO5Akh+Ts0/i32dkCaXhLlSUrWp30pOXJRaWJHioe0Bde4LI4Eu8/OzuuJjnOc26ugTU5/7M4jf+SIj4L2vQLK+KlbBSl8KL8eUH173KzhQPnMGv5sTeZRHdx3v6VLMlvIf78XKJRJIcL+KbxAHPKtYX9ycShA/x8suHEy2sCA9p0NYpJxOfcCSF+/Mk5mQQihYQ/lHRw5aw5WtHcw3ujy+HiIokn4/3tkoYwjB4U68UHMfCXViCNbkc7sfLC/ps+EdhHxbCh/nLBRRF81HmLM3T07sk3PflrISFP8JDmPfjEHgO+nLOgvW4AvTlnAWjiAq478spCuLlPL1yo2GRw31fDiFjVsLPyhxNSwHoyzkLRaE3L4AHvpxiKT528JOCuz8Qfjzvk+Ltj40il8P9+z5Z8OUGGpm3eMmcmbMmEEOBvrwQPvhyHf7a0dGRg4d2V7e2devgjh27EUOBvrwQHtz3ycrqc93w/MWTUtYGt+9MDAiLlqUAHszHonmsAZzGsuXztm3/qV1w0NVrlyDl8eNHc76Z3LNXuxGj+m75eWN2tvSG/F2/bV2zdmliYgJk+/3I/qioCFi4fft6/4FdxowdQoo6lpSU5BUrv4WKv3ffDitXL4yJeQWJd+/dhk3Cwh4qdv302WNpIXduqNupLj83Z+G+L4cfm9ZwtJa5uXlUdAT8rVy+oX69hrFvYmbNmSjME27etGv50nVRUeHTZ4wVi8Wfjx4/eNBId3ePyxfvDeg/DLaCbffs2zlo4IiZMxYoFyiRSKbPHBfy8P70afN/3XnI2cll4qRRb+JiGzVsYm9nLz+R5Fy/fhlSmgQ1V7dTUmZYrMsL4b4vhx9b05sJKIpKSIhbunhtixatnZycL1w4a25mDlLz8fHz8wuYNXNheMTz6zeulNwKXkGgoPhaNesorwoNDXn9+uX8ecubNW3h4lJhwvhpDo5OR48eEAgE7dp1unrtoiInKD44uAukl3GnSFnQWOX5+fkvX74kXMfXx9/KquB20cePH9asWcfR0Un+1sPDs1Il70ehD1RuWL1arZKJoWEhUNNDzS1/C+dDYIPGDx/9B8tt23YEz/Mi/BmRtWVjY18Ht++i6U5Vg3alEI1bn56enqtXryamA0UTSqDxD25haalYzsrKfPb8Cdhl5QypKckf3FC5BJFIVKwEuErAK8jd2dnl6tWL1avVvHb9csWKbnXrNtB0pyrBYbcKeDC+nCEfOb7cpYJrvXqB4MKVEx0dnMpeQoUKrtbW1itXbFROFMjuuYZ6HUwLWJExX04CU96xQzdt7VR6FwUacxncnydRNiaRfAxVAqr9ff50g/qNaLqgoJcvo7y9fTQooUr13NxcNzcPr0oFk5rHxb9xcnSWL7dv2+nYsYMQnAHnDd5dWzsl0r4wdC1SuD8fC8uyH3kre//+wxiG2bxlvVAohAggRBi/GDMIIjCwCmSXnPzu+vUr8sigOho3atq0aYt165aDBU9PTztx8vfxE0acO3dKvrZOnfpubu4QlwwIqAoNzQ/uFNEUPsxfTlEfV6U52Dv8svOQtZX1uAnDR47uBwHB2bMWgo2GVc2btapXN3Dh4lkXL/1VeiGrV/7Qpk2HZSvmQbz82PGDHTp07dv3/Yzmbdt0hAZo+3ady7LTssLf8TvF4f48iad+jnsbmzdojj/hGZJ8sm9lxOQfqhLew4N5EhkiEfOxUkNLroAH48sFFG3Gy5kqCFIA98eXQxiR4edMFUgh3I+X01CXC/DqzWu478uhImd46cvRmCvggy8n/KzLKRateQF88OWEt74ca3M5PPDlNIsxFp7DA18uvbsZf3Few4f7PimWl0/XwfnLFfBgnkSap61PFn15IbwYX469QjyHF/d9IjyH+77c3Iw1tyS8RIKWRQ73x5fbugj4OSYxMTZHoHElxk24Px9L+4Ee+Xlsfn4+4RmPrqZYO2BlLoUX85d7+Fkc/yGG8IzE16Kek9wJwot4OSH9pvhcP5W0f3VEnRYOgW3cCKfJSs+9fSYlPjz3y6X+1na8nLm9BBrfEQeX/piYGNMyLXL+2vvm5eNcsUg2d0VpGdUHmksPQZcrQC0dU6Vub6xsrsOSm6jv7hHQFJRoZUP1nlLJpaI1QWRw/77PYkgkkpSEfFL88RTvlbN71+7mzZvVrFVT+XsBtbHs+0ygJeVBA/K1UrVSrPxOauXhgFKlMkS+oth3TVEFWaUzBFHvBxHKc0rf05TyMyUg/9IlS6dOneri4sK8/+GUC5ZUrITiLo7GKje5+Vg0gmGYQ4cODRkyhBgxW7duHT9+PEHKDPfnYyk7f/75J0VRRi5xQC7xe/fuEaRs8GE+ljIxfPjwRo0aUZTJhN7gmhMVFUWQMsA7X64OUExAQAAxKf7666/g4GAzM+z7+QB8f95nZmbmhg0bYMHkJA507twZGhKHDx8mSKnw3ZePHDly+vTpxGSBrujo6Ohnz54RRD08ipcX4/Xr1z4+mkwha8SEhYVVrlzZ0dGRIKrg/vyX1WEAABAASURBVDgWlVy/fv3KlSuEK9StWxfazevWrSOIKnjqy2/evAlehXAIBwcHLy8vtC4q4Z0vv3hR+qgqLjWgFUCk38nJCcOLJeFXvHznzp2E03h4eDg7O0+dOpUgSvArXn7hwoUOHToQrnPjxg1wL35+fgSRwRdfvmnTJnjlg8SBli1buru7Q9uDIDJ44cvnzp3btWtXwiesra3r16/fq1cvgnA+Xi6RSAQCQUJCAhhWwj9iY2OtrKxcXV0Jv+FyvBx67ydPnkxkbTLCS7y9vV1cXA4dOsQwvJxerBAu+/LVq1f//PPPhN/QNN2/f/9mzZoRHsNNXw493vC6atUqghACnu3u3btE+tTzLMJLOBgvDwkJOX36NEFKANbl1atXhH9wMF5+4MCBoUOHEkQVX3311Y4dOwjP4JQv379/P7yixEtBLvGICH496Zw7vvzkyZM2NjYEKQPQOfrgwQPCG7gTL4efrWHDhgQpGxs2bJgxYwbhB1zw5fv27Wvbti3EhgmiCRBEp2QQrsMFX37mzJns7GyCaMjmzZv37NlDeAAX5kmcP39+5cqVCaIh5ubmPBmRyt/7PhH+wIVxLNDHGRsbSxANgQqOJ+NbuODLoT8ffXk52Lt3r3zYPedBX85f0JerBX05YnKgL+cv6MvVgr6cM/zxxx/Lly8nPAB9OX9BX64W9OWIyYG+nL+gL1cL+nLOcO3atZkzZxIegL6cv6AvVwv6csTkQF/OayQSCeEB6Mv5y6NHj8aMGUN4APpy3jFgwICcnByxWAw/JdQOLVq0gBodjCiH7wTlwnwsdevWxfuay07fvn2TkpKSk5OzsrKgVQb6BpVXr16dcBf05bxjyJAhxaY2FwgEXbp0IdwFfTkfGTlyJNRWirc+Pj69e/cm3IUL87GgL9eU7t27K57iS1FU69atnZ2dCXdBX85TvvjiC/mX5uXl1b9/f8Jp0JfzlPbt21erVg0WPvnkE09PT8JpNI4kgi/fuHGjUVXn3PPl9y69C7mcJhISiZgUdE3D/4rODkSRglUUS1iqeB5pn7Z8OqESGyoyBDrMC/yUkASyeXokSz7cBf6+TDWHVCyFpqS1qI2doFW/CtXqORDDgfFyo+PFvbS759K8a1pXb+gIEpGwAiJVmPSfbD0l0zerUPl7ZCIryMlKl1hC0dLFgoxKm1CyzIXvWJpQDJFvLd1IUWLhFnL5yspUpBcekiIPHBjLKqkc3menCp/fzfz7t7dOMywqelkRA4HjWIyLv/bGvQzLGTq/KuEW+1dGNOro0LSjGzEE6MuNi8hHOR1HVSKco24bp/8uZBADgfFyI+LayUQzc6qiFwfjRQ1aubIMCQ9JJ4YA4+VGRGaKhNb4BzEZaAGVFCsihkDj1qdxxssJJxDnU+I8wlXE+SwrNsws0ujLEe6DvhzhPhgvNyIgwEzRnH0yBHw0ijLMbaboy40I6LpgGc7ebgwfrUinkR5BX45wH/TlRgQlkP4hWgd9uTHBEsJy1pdLB7wY6MOhLzcioHeQw75chmE+HfpyRF8Y7kqFvhzhPujLEX2Bvvxj4IwvNzNnBebcDrKgLy8vnPHlYhElEelw4sI9e3f2H9ilU5dPoqIi2gUHhYaGEH2CvvxjQF9eFsBq7vpta1BQ87XfbSYfR3R05OCh3YnpgOPL+UJubg68NmvaMjCwMfk4nr94QkwKnI/FtFm8ZM6y5fO2bf8JHMjVa5cg5fHjR3O+mdyzV7sRo/pu+Xmj/Cp3997tPv06wgJkBsdSrBCVm8i5desaVNvBHZuOGz/87LlTkAIXhDVrlyYmJsAefz+yn5gC6MuNCIpmKYFmztXc3DwqOgL+Vi7fUL9ew9g3MbPmTBTmCTdv2rV86bqoqPDpM8aKxeImQc2PHz0P+RctXP33uVvKJajbhMgkvnDxrC+/mPTd6p9atWq39vtlFy6e+3z0+MGDRrq7e1y+eG9A/2Gk7FBKswboF5yPxYhgGYqVaKYDiqISEuK2btlrZSWdB+LEyd/NzcxBrI6OTvB21syFQ4b1uH7jSts2HdSVcOHCWXWbQLXd+tP2HTt0hXQ4T7Kzs3JyPuJ7NqHWZ35+flpaGjEmFixYwGdf7uvjL5c4kXqPhzVr1pHrFfDw8KxUyftRaGkTk6vbhGGYyKhwWKXIOX7c1z179CPlhiKUqcTLPT09v/nmG2JM1K5dm/AYC0tLxXJWVuaz50/AMStnSE1JLmVzdZsIhUIQuqWl9qYKYomhxuhorHLj9OUjR4709vYmvMelgmu9eoFgnZUTHR2cyrGJpaUlTdPgUogWMZDM0ZcbER/f91kloNrf5083qN+ILpzy4uXLKG9vn3JsIhAIatSoHRr2vudox87N4FcnTZxByofhWp8YLzciPr7vs3//YWAzNm9ZD34jJuYVRBi/GDMIIjDl26RXj/537946dHjvg5B7J08d+b+Du/39pZdxOAeSk99dv34F8pOyY7jWJ45j4RQO9g6/7Dx08ODucROGv379EtqOs2ctrF6tZvk26dy5e0Zm+u492+FSWaGC69ivpnTr2gvSmzdrVa9uIAQZR40cO3rUWGL0UBx4ei9nfPmpbfFxETnDFnBzptU9SyMCWzu37F2B6B0cx2JESGdGJpzGVCKJ6Mt1B01TAgGn7/skOB9LeeGML5cwhJFwtzY3ob5PHMeCmBzoyxHug77ciJDOk0hx15cbDvTlRoR0nkQOR1mk5zDe91leuDO+nNt1OUtMZjZQ9OU6hGW5XJcbDvTlRgSLplw3oC83JrAe1w3oyxHug77ciKBoljbnrGmhBWDJGGII0JcbEdb2EG0zjA70ALSr7V1MJMaC87HojuadnfPzCSdJfCW9s67Bp67EEKAvNyLsnK2dXAXHNkcRznH1WIKHvwUxEOjLjYthc/1tbM0OrovIzuJIrZ70JufA6gifGnZ9J/kQA4G+3OgYMM3n/76PPrrxtZkZxYhZRqm/kJJNAc4Wte4CAZEo3yxKFdyrUNi/xEofs8kqlfB+lfStIhtNUwwjL4Al7zdhibxDli1IVKylZKukOyucgoKmibwE+RpobkJGSHH3s+gwxIMYDo3viMvPz4+JiTE208JJ7l14l51BqCI/kFxVRX4yii6u+6Kwsiu2sq7liUQkFt2+devTT1sTWa8rBSJ9XzJFCvNR71OYwneKtZRc5AV5qIJFhiU0JT29HFxJw9aG8eLK4H2f/CUlJWXQoEHnz58nXAd9OX8Ri8VmZhpbVlMEfTl/QZWrBcexcAaRSGRubk54AMbL+Qt/6nL05fwFHYta0JdzBlS5WtCXcwb05WpBX84ZJBIJ+nLVoC/nDOBYBAJuPyq6APTl/AV9uVrQl3MGjCSqBX05Z8DWp1rQl3MGdCxqQV/OGVDlakFfzhnQl6sFfTlnQF+uFvTlnAEdi1rQl3MGVLla0JdzBvTlakFfzhlQ5WpBX84ZsPWpFvTlnAF9uVrQl3MGKysre3t7wgPQl/MXsHk5OTmEB6Av5y9gV8C0EB6A8XL+gipXC/pyzgABFgizEB6Avpy/8KcuR1/OX9CxqAV9OWdAlasFfTlnQMeiFvTlnAFVrhb05ZwBHYta0JdzBlS5WtCXcwZ0LGpBX84ZsFdILejLOQM6FrWgL+cMqHK1oC/nDKBySZHn4XIW9OX8RSAQYF2uGvDlGzduNKrqHH25RgwbNiw1NZWiqPz8/MzMzO7du8OyUCjk8ONt8b5P3tG2bdvk5OTExETQOtTlCQkJUHPZ2toS7qKxyo3Tl9vY2BCkbAwePLhYpcCy7CeffEK4C/py3mFvb9+nTx9LS0tFire3N0ifcBeMl/ORgQMHgrIVb5s0aeLr60u4C/pyPgK9niB0eXXu7u4Oy4TTUODJNNoAGuYxMTHGZlqQcjBkyJDw8PDg4OA1a9YQTqOxyo0Q8OUjR45UvgTrmitHEqMeZ0nyqPw8Db49ipK+lv37pinCljk/FC7PqVj4YH6GYSAnDbshlNbLV+TUNL9GmFsSSyuqVnPbpp3cS8mG8XKNOfLDq5S3EhdvCxdXC0aiqeUDPWn0S7JlkaBGGctLwZHLdavRJpruQoMNaCY5Me/+xcz0ZLbjEA912XAci2bsXh4tETNDvkHDZlz839rIg+tfDZ6pug2NvlwDLhyIj36cO3hOAEGMj/2rIpp/5hLY2qXkKoyXa8CrJ9lu/pYEMUoc3cxCb6SpXIXxcg0QiShXT2uCGCUOFSzz1Mxtir5cA8QiVuNaAdEbEkF+LqNyDY4vR7gP+nKE+6Av1wBp/wlBz2J6oC/XAEYadGUIYmqgL0c4AiWQj1ZQAfpyzWAp3XajI+WGlRCGUd3Fib5cA6TDmkx+bBsfQV+uAaziBTEp0Jcj3Ad9uQZIPTmLvtxIoaH1KdBS65PPvlw6fBONubHCQOtTovrXQV+OcB+cj0UDwLFQGEks5Oixg8EdmxJTAH25BoBj4cBtsjwEfTnCfdCXa4DMrWjmWDIyM7Zt+/HM2ZOOjk5BjZt9NWaKu7v0JtycnJwNP6wKCbmXmZnh5xvQtWuv3r0GQPrxE4f37tu59rvN3y6cnpz8ztfXf+b0b9PSUld/t0gsETcJ+mTG9PlOTs6Qs3vPNkOHfP78+ZOr1y7Z2trWq9dw/rzl9nb2pRQO3L5z49ChPc+eP3Zxca1bt8HYMVMqVHCF9JSU5C0/bwh7/FAoFDZp8snI4WMqV9ZgHiKJRPLN3CkJifH/2/ybo4Pj48ePdu/Z/uzZY0cn50+afzpq5FjFPIzqVn27cIa5mTl83oOH9jAME+BfdfasRVWrVi/7MbAUq85Poi/XBFazPiGxWDx33tR3yUkb1m+dMnn226TEufOnyidThoW4uNjly9YfPnimdevgH39a8/TZYyKbDygrK/O3PdvWrd3yx8krIpFo1XeLzp47tXPHwf17T4aGhRw6vFdeuEBg9vuR/d2797104S6cFa9fv9y0+Xv5KnWFvwh/Nm/+1w0bNvnt1yNTp8yJjHyxZu0SItPo9JnjQh7enz5t/q87Dzk7uUycNOpNnAYmcO26ZS9ePF27ZjNIPPZNzKw5E4V5ws2bdi1fui4qKnz6jLHyT13KKjOB2YOQe7Bw7syN3b8ddangumDRDI2mVweRq/OT6Ms1Q6Oa/Pad60+fhk2aMKNhYFBw+86TJ82qUqU61JpQoYaGhsyeubBWzTpQxw8b+nm9eoFQw8m3AmVDDQdVqbW1dbOmLePj30yfNg+uAC4uFQIbNAZpKsqvWqV6k6DmUIHVrl2vV8/+V66ch21LKTwsNMTKymr4sC+gtGZNW6z//uchQ0ZDOuSHkwQuBZAIe5kwfpqDo9PRowfK+DH37N15+fLfq1b+UMnTC95euHAWamUQsY+Pn59fwKyZC8Mjnl+/caX0VUR643zeiOFj4ONAOZ+PHp+YmAAHRrSBxipPSEhYsWIFMSYyMzNzc3OJntCgNo+MDIeLDPyIYP77AAAQAElEQVSi8rfVq9VcMH+Fm5t7dHQEqM3f/31lUb1aLfAeirdgM+QLsLmzswsoT/7W2tomKztLka1q1RqKZa9KlUHiUIWXUnjdeoFgSOZ9Ow0uAlCtwjkApx+kwyUCriGNGjaR5wedwen08NF/pFQoGRcuntv121Y4Q8D/yNMfP35YU3aCyd96eHhWquT9KPRB6asAf/+qZmYFFtrbywdeX72OJmUHImBqxiRq7Mvha3rw4AExJoYNG+bl5UV0j6bhlezsLEtLq5LpYLitrIrcJQ1qzs19f2uusr8sJXapXLiVtbV8j6UUDqfZd6t/unr14vYdm7b8vLFxo6ajR40DdYJHgjOkXXCQ8lZy918KYA/AUXy3ZrF070pHAqU9e/6kWGmpKcmlrypWCJyo8o9Dyg74SUZLvUJ8Hscim1RKA89iY2ML8oK2FE0XuWZCe0soLHLxyc7Jdq1QkWiIsgiEsqsZ6Lv0wsGTwB/4gfv37xw99n/zv5127Oh5aICCO1q5YqPyVgLoMS8DM2d8C7X+d2uX7PrlMFx2IAUsNXgk2IVyNkcHp9JXFf84QiEpehp/GIrVWg8/z+PlGvnymjVqw6/1/MVT+VvwvtNmjAUbU6O6NB0sqSIn2Hc/f42/1YcP7yuWoTS43Ht5VS6l8JCQ+3f+vQkLrq4VO3fuPmnizMysTAiMQGsBLJ+bmwcYGPmfu7unsh1SB5y9Xbv0/HrKNzbWNitXLZAnVgmo9vZtQoP6jRSlQXNWbttKWQVERoWnpxdMqPJC9qUFBFQlZYel1PXwY7xcAyhKs+ZnUFBzkN327T9du3757r3bP/z4XdLbRAiWNW3aAvzohg0r4fINjdFfft0CQhw0YATRkKR3b8Fhg22A8+fP08fatetkaWlZSuEQKFyydM4ffx6D0OSTp2HHjh8EuXu4e4J1ga3WrVsODT7Q2YmTv4+fMOLcuVNlPAy4DixZshZCNId/3wdv+/cfBpevzVvWw8kWE/Nq2/afvhgzKCo6ovRVgIOD40+b1kLsFf727N0BTeT69RoSbYDxcg1gyz4FrQyoXCEguHrNokWLZ8PbTz75dPWqH+UNrBXL1m/d9gME7ODaGBBQbfmydXApJxrS/bM+EH4Ghw3L0HaEYKV8p+oKHzhgOOh78//Wbdi4Cla1b9d544bt8uNZvfKHU38cXbZi3pMnoRDe6dCha9++GjydAhz/yBFf7di5Oahxc6iAf9l56ODB3eMmDIfTD5qbs2cthAyQzcHeQd0qAGLkfn5VBg7qChrz9Ki0YtkGgaBMrumD4DyJGrB5RkTjYJe6rVyIEdCrT3C/vkNGjhhDOMHiJXOgbbp+3c+kvFw98vbV04yJ61SYHPTlGoADtYwZ8JO0tiKJfJ6/nJLqnEfzsUBkPUxNv0y3br2h84gYE2BKGAbHl3800g5+o7mL4uTxi0THzP1mqVgkUrlKsxhfGVi6RIf1JsbLNUD2KAYejbx1dHAknAB9uQZQihfE+KBolqLxvk+tgPd9GissQ7Hoyz8emWPButz0QF+uATSK3DRBX64BjKwXjSCmBvpyDZCOLMR7+I0XVt2vg75cAxiGX5FEU0Pts5/Rl2sARhJNFPTlGsDKbhQniFHC0BJ1d3OhL9cAc0uK0eSuckSviFlLWy31CvHZl1vb0rEReruNGtGM5MRcexfVDhznY9GAVn0qpMblE8T4kEgkOWnswGmqZ0pCX64BAXUcmnR22rsiIjUJa3QjIvppyoHV0R1HuKnLgOPLNSOogyvLkNPb3gjMiYWVmSi/SHuHolTceyUdlV62W7IoWQOXKhoQoymKYVmapooNnpYWKmttKRaUC1FeKH5sRVco0qkSU3Go/DgFR1XieIpuWPQjlJq5ALYgfFWmzIWYWxJhtgR+ka6jPfzr2KnLhvFyjWnSyRX+/t4Xl5EiyRcWVR5N2BLPA5XPrlgmmbMFU+cojzqSl6my5IIMRVWuDkUJ8tOGSMP/TGpKqmvFCuoOrXATtmT8VEATifonnxZTeSkHX4DivFSdmVUXwLWyoXxrWbbrX4mUCt73yV9SU1MHDBhw4cIFwnXQl/MXsVismLGN22C8nL/wR+Xoy/mLSCQyNzcnPADHsfAXdCxqQV/OGVDlakFfzhnQl6sFfTlnQF+uFvTlnAEdi1rQl3MGVLla0JdzBvTlakFfzhlQ5WpBX84Z0LGoBX05Z0CVqwV9OWdAx6IW9OWcAVWuFvTlnIE/vULoy/kL+nK1oC/nDOhY1IK+nDOgytWCvpwzoC9XC/pyzoAqVwv6cs6ArU+1gJ5A6MSYQF9ePuACqJ+Z9wyOxvOxAOnp6VAH2NraEsQ0+fPPP3fs2BEUFLRw4ULCA8rzvG1HR0cwCVFRUcQ4WLNmzatXrwhSBo4dO9atW7e7d+/+73//44nESbmfKt+sWbOdO3f+9ddfxAh48eJFamoqQUrl0KFDHTt2fPr06a5du5YuXert7U14Q3kci7Hx7NkziG86ODgQRBX79u0Df/LZZ5+NGTPGxcWF8I9y1uUKjh49avBOopo1a6LEVQLVdsuWLZOSkk6fPg2RMX5KnHy8yvv16wdVRUhICDEcYDEfP35MkEIkEsm2bduaNm0KAbGLFy9Onz7dzs6O8BgthEsXL15MDApcTLD7U45QKIT20p49e8Cc3Lp1SyAQEOTj63IFGzZsiI6OJoZg4sSJ9erVI/wmMzMTfoLg4GCI8P77779jx45FiSvQZusTInrDhg3jVePdGEhJSYH6+8yZM1999RV8/wQpARdiLLt374YGKAQ3Cc9ISEgAff/zzz/gTwYNGkQQNWjNsSiYMmVKXFwc0SPQJQS/N+ET0DkPzaEvv/yyTp0658+fR4mXjk7qcmjgDx48GLpIiV54+fKljY2Nm5sb4QHQ5Qz1N8SUwJ90796dIGWAC46FJzx//hz0Dac0+JPOnTsTpMzocOBlp06dfv/9dz3U6NAz5ezs3L59e8JRwsLCQN9v374FfXP4Y+oOHar877//PnjwYP/+/XU9iPnNmzdZWVmEizx48AA656FzB/T96aefEqRc6NyxxMfHe3p6El0CTTGKory8vAiHuHPnzi+//MIwDPhvHoaPtIs+fHlQUNC9e/cIUjZu3LgB9Te0pyGE0rhxY4J8NPpQOezi6tWrbdq0UaT06NHjjz/+IB8N2KH8/Pzc3FzYhbW1tUgkkkgkEFkjpsnly5fBf1eoUAHqb+zN1SL6uO0P7ESrVq1CQkICAwPhLSyDFo8fP96nTx/yEYwcOTI6OpqiCp5dnZaWBtf3qlWrEhME2jCgbx8fn4ULF0IPF0G0ivZ7hVQiEAj8/PyCg4NbtmwpFArz8vIuXbpEPo6hQ4dC/a2cYm5uDnF6YlKcPn0aznaoxVevXr1u3TqUuC7Qk8oBJycnsBagb+leafr169eJiYnkI+jSpUuNGjWUHRc0c3v27ElMhBMnTnz22WfQyvzxxx9B4sY2/weX0J/KmzdvDh5a8TYpKenjm6SjRo1SxOPhzOnXr59JTL0A3QjQrRMaGgpRlGXLloFRIYgu0ZPKu3XrBl4cfLMiRSumpXXr1orq3Nvbu2/fvsS42b9/f7t27SIjI2EBLLiHhwdBdI+eVH7mzJnx48fXqlXL1dWVkQGtxvDwcGgyko/jiy++gDJhoUOHDsY8ecbu3buh2Q0m7eTJk3PnzpUfM6IfdBJJPLUtNiNZJMyVLtMCipFIdyEQUBIJK5GI8vPz8vMlDAv/GDs7ewsLCxriJBRhmPfZiCwyQxUm0rQ0kCJfLnjLEkZ25FB+RloGXCYcHBxgA3gLH0lSeM2QFkxRig3lb5WLkvN+pxD3LEyEo2JKfDnmlsTahq7Twr5OcxX3UE6YMOHx48cQNlWkwNe7U8awYcMgPlisuYzoBy272Of/pV84kGRjJ7CyM7OiCgQqlxQlIKyEyGRoxUikmSmasHI5SsVXsKzID4l0oUClyizMIN8QxChXICxXtHYqsgpOXYWIlUouKIcUKUrO+50q6fz94SkhsKSEQsm14ymhNzIHz/RVXrVx48aHDx9CBEn+FiwZ2O5ff/0VOueho4cnc7UZJ9qsy/+7/O72mbS+03xs7SwI1zm0LsLexWLQ9IKGI7Qx1q5d++7dO1i2s7Pr3bv34cOHv5RBEEOjTV9+63TaoDm8kDgwaFbVzBTRmV/fwHJcXNyWLVvkEgcyMjKg//LmzZsocSNBayo/viXGxl4AJpvwhioN7GPDpY2PRYsWKc+nBzHNvXv3EsRo0JrKM5JE9k78sp41mzqJReT7779/9uwZKFt5VXJyMkGMBq3pMk9ILMUU4RMCgUAiJufPn7eyslIOnsgV37Vr17NnzxLECMCG/8eyZ88eeW+XRAZEKuE1ICCAIEYDqvxjwf5L40drKqcFhDbT36gYo4Dil0MzXbSmcrhoM2KG8AmK4PQHpoH2HAv/fnGc48NUQF+OcB+tqZyiCc03W07Ql5sGWqzLKbyAI8aJ1lTOMizLr8andFgkQUwB9OXlBwOJpgKqvPyw6MtNBO31CkHrk2c/OjZETAXt+XKWMDxTOfpyU0GbKic8a30ipgL6coT7GLIjJzo68sef1oz6vH/nri3GjR9+8tQRRXq74KCnzx4vXDQLFgYO7vbz1h8kEol87e07N6bPGNf1s1bDRvRevWZxcvK7169fQraHD/+TZ7hw8Ry8PX7isPytfO2Tp2GwfO6vPyZOHg3bwuuRowcU97wuXjJn2fJ527b/BDmvXvvYWWIQY0NrKpdOAqFhYf/bsv7u3VtfT/3mu9U/devWGxQPCiay6Q7hdf2GFcHBXf4+d+vbeSsO/77v8hXpTLYvwp/Nm/91w4ZNfvv1yNQpcyIjX6xZu8THx8/Nzf3xk0fyYsPCQtzdPZ4Uvg0NC7GztatZozaof83apdWr1Tyw79SYLyeByjdvWS/PA3uMio6Av5XLN9Sv17CMx4+RRFNBe75c89FLCxeuzsnJ9vSoBMsNA4POnTv1792bzZu1lK9t07pD2zYdYKFBg0aVPL1evHjaIbhLWGiIlZXV8GFf0DQNUgbtgjRlmzd5KqutgYeP/uvSuceZsyflb0NDQ4KCmkP+M2dO1K/fcNrXcyHR2dnl81Hj165bNnzoF7BMUVRCQtzWLXuhcFJmcLSWqaA9x8JqPiyRZY8dOzhydD/wCfD37PmTtNQUxcrq1Wsplu3s7LOyMmGhbr1AoVA479tpvx/ZH/smxtHRCU4PSG/UsMmj0AewkJ6e9vJlVM8e/cHJJCZKH48IdXmjRk0Zhgl7/LBJ0CeKMuGCAInyrQBfH3+NJI6YEAZrfYLC5s7/WiTK/2rM5MDAIHs7+ylfF5nXgVY1+Av8Btibq1cvbt+xacvPGxs3ajp61Li6dRs0btwsIyMdLDhU7dWq1nBxqVC7dr1Hj/5r2rRFXFxs0yYt8vPzRSLRL79ugT/lAlMLzysLS0uiaQQWWgAADAZJREFUIdIeAjQtpoDBVA4O+9mzx+u+3wJKladAbV3R9cPP7GzWtAX8fT56/P37d44e+7/53047dvR8hQqu/v5VwJpHRL6oV19qrMFew1taIAC3A94GUmxsbDp1/Kx162Dl0ip5lv956qyKGeYQY0SLd8RRAk06P8FawKtC1mAz4M/f7wNzeIeE3M/LzwOVu7pW7Ny5u4dHpWkzxiYkxnt7VQYHAmGWqKjw4cOl14R6dQO379wkFovBlMu3rVKlemZWptzhAFC1x8e/gWYrKS8ocVNBa76cZViJJsbczzfAzMzs0OG9GZkZ4DQ2bf6+SVBz0GvpW4G3XrJ0zh9/HktLS4Xg4LHjB0HuHu7SZ9A1CgSV35fW5XWlz3WpWzfw1atoqO8bFV4rvvpy8o0bV6BVCmYJmqQQOpwxa7zylOoIV9GeyjXs+wQX8e38FU+ehvbq3X7+gukQ2uvZsz/ESSB8XspWAwcM/6xbn83/W9enX8fpM8ba2Nhu3LBdPtEmqBlOksqVfSFmQmSTFfr5BUAK1PHybevVC9y+df+jRw9g21lzJmZnZ61YvsFSczuOmBxamw10+7woB1eLz8aU3+aaHLlZkkPfR0/5wSSf18UrsIe//OAdcaaCFu/7pAjP7vtETAUt9n2yFMuvug1H3poK2nMsDGF5FlqTdQmh0E0A9OXlhy14EhFi7KDKEe6DKi8/FN9udDVZtKZyMwvajGdz3rIMmnLTQGsqF+czYp7NeYuYCuhYEO6DKke4D6oc4T6ocoT7aE3lFta0wJxfkTUJkVA4dMcU0Nqv5FCBzk7j1x0Jz//NMMfR6aaA1lTe/SuP7HQJ4RNRD9I9/fG2fxNAayq3sLAIbOu4b2UE4QdHNkZa2gp6juXRXSOmC6XdgYRht9L+OfrOzsHMxklAMYJia2kBdBdSRW6co6WDGSkzwoqLpEjH8UoHQrHQi17wiAtKNmE4TQqfeMHK54l4/wAMWmnWXYGsAKZImQWFUNIZmaUf+n0iKx1eWFgsTVOyRzFLyy9ZAm3BCrPEGckihwrmQ+f4EsQUoLQ+XDY/N/+PHYmZaaK8nOKraIFUXsoPZpGrljYjTKHKqSKDWd+rXJ6urHJKOmmd9C0L5w7DCswEio9C07IdsUX2UrBMycaFw9bKiUrFKlT+/gRTymBhRVtak2qNbJt0rEgQE4HiwKDwixcv/vXXX2vXriUIogouxMvFYrH8Nn4EUQkXxCESieTT5CKISrAuR7gPqhzhPqhyhPugyhHug61PhPtgXY5wHy6MHMW6HCkdLqgc63KkdNCxINwHVY5wH1Q5wn1Q5Qj3QZUj3Ad7hRDug3U5wn1Q5Qj3QZUj3Ad9OcJ9sC5HuA+qHOE+qHKE+3BBHPb29qhypBS4II6cnJy8vDyCIGrggsqhIgfTQhBEDRxRuUTCr0mlEY3ggsoFAgHW5UgpoGNBuA+qHOE+qHKE+6DKEe6DKke4D6oc4T6ocoT7cCRejr1CSClgXY5wH1Q5wn1Q5Qj3QZUj3IcLMzujypHSMeFnN/fo0ePNmzdw/DRNyz8FvPr4+Jw8eZIgiBImXJcPHjzY3NwcwogURdEyoFLv2bMnQZCimLDKhw4d6u3trZwCFXmfPn0IghTFhFUOVfjIkSMtLS0VKW3atHFxcSEIUhTTbn326tWrcuXK8mUvL68BAwYQBCmBycdYRowYIa/OmzVr5unpSRCkBHqNsaQn5T+9m572ViTKZ8SiIicYTbEMSyneUrJF5UODBBb+EapoijTT02fP8vLzalavYW1rzTJSJ6P8oaAoeQBGeVsZsD+q5IcXmDEUTdk5mvnXsfWrY0cQ00cfKg+7mRryT0ZmikgiITRNKOkfzUqK7rdAs4q3FGGITISFKyiZXIvlkSbAKwMnAC0tV6boQl0rZytMVNoNVXjiFIUW0AzDQJGMbACYuRXxr2XbaSReJUwY3ar83/PvHlxMl4hZCxtzBw9bN39nYlJkvMt69zIjJzUPTgbPAMt+UyoTxATRocp3LYnOzZI4uNt5161ITJyU+IyEZ8mMmLQd6Fq3uRNBTAqdqDz+Ve6xH9/Yulj4NfYiHOJtVGpSVFqlAMs+k7BSNyW0r/J8Yf72ea+9At2c3WwJF3lyObr+p46tepj8BYo/aFnlsRE5J/4XV7eTP+E0Ty5Fu7hbDJ7lQxBTQMvx8hNb4qq35ZRLUUnt9v5pSaLTv7whiCmgTZVvnxcJgRQLCwvCA2q29Xv5JDfhdQ5BjB6tqfyPHW8kLOVTz43wBicPu5Nb4gli9GhN5a+e5nrXciV8wqtuRYmEvXr0LUGMG+2o/I/tsQJz2p6jQZVSsPewfXYvkyDGjXZU/iZS6GjEEg8JvTBrYbOs7FSibSrXdhPlsVFhGQQxYrSg8oTXuWIRqVSbX3ZFgZmV4P6FdIIYMVpQ+YNLaQIzivAVa0er1LcighgxWpipIiVBKLAQEJ1x978/b909Hp8Y4eleNbBeh08/GUzJBubuPTQferUaNehy6NiyvLwc38r1Pus82bdyXflWf57bdO/hGUsLm4b1O7u56rD7xsHNJispmyBGjBbq8txs1sJaV/O6/Pfwr0PHl3tXqjF/xvGuHSdcvXnw5JmN8lU0bfYqJvR+yNmvx/+2atE/ZuYWB48tk6+6+e/Rm/8e6fvZ7K/H7argXOn85V+IzoB4IoOTNBo3WlC5RMSaWepK5f/ePxng27Bvjzn2di7VAoI6B4+9cef3zKwU+Vqowgf1WVDBxUsgMGtUv3PSu1eQAunXbx2uXye4ft32NjYOTRp1rxoQRHQKRcVF5BLEWNGCyhlWOlUE0QEMw0S/flS9WjNFCgidZZnolyHyt24V/SwtbeTLVlb28JqTm8Gy7LuUGHe392NpvCvVJLqFZRiCGC1aqIMFNCsW6WRqK7E4XyIRnbuwFf6U0zOzC+pylWeXMC+bYSQK9QMWFtZEp7DEHqcOMGK0oHJLa1qcpxOVW1hYQfOxcWC3+nXaK6eDRSllKytLW5oWiERCRUpevg5Hm+RmSR+O7uiq4xMJ+Qi0oHIbB7N3cboKpVXyrJ4rzKwa0Fj+ViwWJae+cXJ0L2UT8E/OTp4vX4e2aVmQ8vT5DaIz0mKzCH/jqKaBFvx01Qa2EpGubGm3jhPCnv5z5/4pqUd/FbLv8Lfbdk0CJ1P6Vg3qdgh9chm6PGH50rU9r2LDiM7ITMmxd9JhIBX5eLSg8obtpJ40LVEnvdz+voHTJ+yB5uaSNV22/TYlV5j1+bDvzc0tS9+qQ5vPmzXudeLMeujYh4q8Z9dphBAd3eEqFkqq1OfdAB7TQjv3Cu1eGpUnoqu35N3dkOlJmTEh7yZvqEoQI0Y7EcC2g9zys/k4g3jCs1QXD3OCGDfa6c3xrWlnbUdF3Imt2sxbZYZHYZcOn1ypcpWNtQMEuVWuAtfRo8tUoiXA1v+yb6bKVRB5hKCkfOBAMT79ZHDn9l+p3CovK18klAydw/GbXDmANu9u3jw9omb7ymZmKs4ckSgvN1f1OGyRON/cTPVNdOYWVtZW2pzDLSPjHdEQiLVbWam23WEXo32qW/Yci7NWGDva7Jmv0cTu+T+xdYL9Sq6C9uIHm4x6wMFBa8ODo/+Lh9MZJW4SaLNnvuNQD+eK5i+uvyZcJzEyJSdFOP47bHSaBtqfdejsbwkvH2fXau9HOErcs6T0+KwJa1HiJoP2R1l1He1hX8Hs2ZWXhItE3HmTHp+NEjctdDUb6Nnd8ZEPs+0qWPg14sgkRElRKUnR6dZ2gs+XYFDFxNDhnLcikWjPshhhLmPlYOFdq6KlnanORhQT+jbrXQ7LsPXb2Lfq4U4QU0Pns/Q/u5d263RKdjpD0cTc0szCxtzC2szMkibU+7EfxWboL3J8slfZ9PqyCfmLrmJLZFP1zIn3OUvZ0fvMLCsRSfLzJBAOFwvFEjFLC6iqDWw6jcCJ+k0V/T1x5cbJxNgIYUaqRCJiQTpE5fguSiFV6eNQClJIiYetyNOUH5iiMpuiNEXOojKXnjd0ceFL79SmCU1TVja0q5dF867OLp44qta0MeFnNyNIGdHV/ZoIYjygyhHugypHuA+qHOE+qHKE+6DKEe7z/wAAAP//ns22gAAAAAZJREFUAwCQV4IxOrpIDAAAAABJRU5ErkJggg==", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import Image, display\n", "arch = SelfRAG(llm=llm, documents=STARDUST_CORPUS, top_k=4)\n", "graph = arch.build()\n", "try:\n", " display(Image(graph.get_graph().draw_mermaid_png()))\n", "except Exception as e:\n", " print(f\"(mermaid PNG render unavailable: {e}; see § 2)\")\n", " print(graph.get_graph().draw_mermaid())" ] }, { "cell_type": "markdown", "id": "de8cf6bb", "metadata": { "papermill": { "duration": 0.002284, "end_time": "2026-05-28T02:47:40.735922+00:00", "exception": false, "start_time": "2026-05-28T02:47:40.733638+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 8 · Live run — 3 tasks\n", "\n", "1. **Direct retrieval task** — answer in corpus.\n", "2. **Parametric task** — agent should skip retrieval.\n", "3. **Mismatch task** — corpus has loosely-related docs but no direct answer; reflection should drop most/all docs." ] }, { "cell_type": "code", "execution_count": 4, "id": "1ddf1e78", "metadata": { "execution": { "iopub.execute_input": "2026-05-28T02:47:40.744526Z", "iopub.status.busy": "2026-05-28T02:47:40.744526Z", "iopub.status.idle": "2026-05-28T02:48:02.746036Z", "shell.execute_reply": "2026-05-28T02:48:02.745022Z" }, "papermill": { "duration": 22.006796, "end_time": "2026-05-28T02:48:02.746872+00:00", "exception": false, "start_time": "2026-05-28T02:47:40.740076+00:00", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TASK_TAG: direct\n", " TASK: When was Stardust Aerospace founded and by whom?\n", " NEEDS_RETRIEVAL: True\n", " N_RETRIEVED: 4\n", " N_KEPT: 1\n", " KEPT_INDICES: [0]\n", " N_FULLY_RELEVANT: 1\n", " N_NO_SUPPORT: 3\n", " N_VERY_USEFUL: 1\n", " doc[0]: rel=fully_relevant sup=fully_supported use=very_useful\n", " doc[1]: rel=partially_relevant sup=no_support use=somewhat_useful\n", " doc[2]: rel=not_relevant sup=no_support use=not_useful\n", " doc[3]: rel=not_relevant sup=no_support use=not_useful\n", " FINAL_ANSWER: Stardust Aerospace was founded in 2019 by Dr. Amara Okonkwo and Jin-ho Park.\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "TASK_TAG: parametric\n", " TASK: What is 25 squared? Return only the integer.\n", " NEEDS_RETRIEVAL: False\n", " N_RETRIEVED: 0\n", " N_KEPT: 0\n", " KEPT_INDICES: []\n", " N_FULLY_RELEVANT: 0\n", " N_NO_SUPPORT: 0\n", " N_VERY_USEFUL: 0\n", " FINAL_ANSWER: 625\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "TASK_TAG: mismatch\n", " TASK: What is the boiling point of liquid nitrogen?\n", " NEEDS_RETRIEVAL: True\n", " N_RETRIEVED: 4\n", " N_KEPT: 0\n", " KEPT_INDICES: []\n", " N_FULLY_RELEVANT: 0\n", " N_NO_SUPPORT: 4\n", " N_VERY_USEFUL: 0\n", " doc[0]: rel=not_relevant sup=no_support use=not_useful\n", " doc[1]: rel=not_relevant sup=no_support use=not_useful\n", " doc[2]: rel=not_relevant sup=no_support use=not_useful\n", " doc[3]: rel=not_relevant sup=no_support use=not_useful\n", " FINAL_ANSWER: I retrieved relevant material but none of it passed the per-doc reflection check. I don't have enough verified context to answer.\n", "\n" ] } ], "source": [ "TASKS = [\n", " (\"direct\", \"When was Stardust Aerospace founded and by whom?\"),\n", " (\"parametric\", \"What is 25 squared? Return only the integer.\"),\n", " (\"mismatch\", \"What is the boiling point of liquid nitrogen?\"),\n", "]\n", "\n", "for tag, q in TASKS:\n", " r = arch.run(q)\n", " print(f\"TASK_TAG: {tag}\")\n", " print(f\" TASK: {q}\")\n", " print(f\" NEEDS_RETRIEVAL: {r.metadata['needs_retrieval']}\")\n", " print(f\" N_RETRIEVED: {r.metadata['n_retrieved']}\")\n", " print(f\" N_KEPT: {r.metadata['n_kept']}\")\n", " print(f\" KEPT_INDICES: {r.metadata['kept_indices']}\")\n", " print(f\" N_FULLY_RELEVANT: {r.metadata['n_fully_relevant']}\")\n", " print(f\" N_NO_SUPPORT: {r.metadata['n_no_support']}\")\n", " print(f\" N_VERY_USEFUL: {r.metadata['n_very_useful']}\")\n", " tokens = r.metadata['reflection_tokens']\n", " for i, t in enumerate(tokens):\n", " print(f\" doc[{i}]: rel={t['is_relevant']} sup={t['is_supported']} use={t['is_useful']}\")\n", " print(f\" FINAL_ANSWER: {r.output[:200]}\")\n", " print()" ] }, { "cell_type": "markdown", "id": "972c801b", "metadata": { "papermill": { "duration": 0.003039, "end_time": "2026-05-28T02:48:02.755177+00:00", "exception": false, "start_time": "2026-05-28T02:48:02.752138+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 9 · What we just observed\n", "\n", "The cells above ran Self-RAG on 3 task types (direct/parametric/mismatch) to exercise the per-doc reflection-token gate.\n", "\n", "### 9.1 · Per-task reflection summary\n", "\n", "| Tag | Needs retrieval? | Retrieved | Kept | Fully-relevant docs | No-support docs | Very-useful docs | Final answer |\n", "|---|---|---|---|---|---|---|---|\n", "| `direct` | True | 4 | 1 | 1 | 3 | 1 | Stardust Aerospace was founded in 2019 by Dr. Amara Okonkwo and Jin-ho Park. |\n", "| `parametric` | False | 0 | 0 | 0 | 0 | 0 | 625 |\n", "| `mismatch` | True | 4 | 0 | 0 | 4 | 0 | I retrieved relevant material but none of it passed the per-doc reflection check… |\n", "\n", "### 9.2 · Patterns surfaced in this run\n", "\n", "- **✅ `direct` task** — retrieved + kept ≥1 doc (1/4). Reflection passed real-evidence docs through.\n", "\n", "- **✅ `parametric` task** — correctly skipped retrieval (needs_retrieval=False).\n", "\n", "- **✅ `mismatch` task** — reflector correctly dropped ALL docs (0/4). Architecture correctly admits gap.\n", "\n", "### 9.3 · The takeaway\n", "\n", "Self-RAG's value lives in the **Kept / Retrieved** ratio in § 9.1. The architecture is working when:\n", "\n", "1. **Direct tasks**: Kept > 0 (real evidence let through).\n", "2. **Parametric tasks**: NEEDS_RETRIEVAL=False (no reflection burned).\n", "3. **Mismatch tasks**: Kept = 0 (architecture admits the gap rather than hallucinating).\n", "\n", "The deterministic-picker is in `_compose_keep`: pure-Python `is_relevant != 'not_relevant' AND is_supported != 'no_support'`. No LLM emits a numeric per-doc quality score — the gate is decided by two categorical commitments per doc." ] }, { "cell_type": "markdown", "id": "9b619be8", "metadata": { "papermill": { "duration": 0.002056, "end_time": "2026-05-28T02:48:02.760192+00:00", "exception": false, "start_time": "2026-05-28T02:48:02.758136+00:00", "status": "completed" }, "tags": [] }, "source": [ "## 11 · Failure modes, safety, extensions\n", "\n", "| Failure | Mitigation |\n", "|---|---|\n", "| **All docs dropped** (over-strict reflector) | Loosen the keep condition; add `partially_relevant + fully_supported` as keep |\n", "| **Junk doc kept** | Reflector too lenient on `is_relevant`; reword schema field description |\n", "| **No reflection for parametric task** | Architecture skips reflection when `needs_retrieval=False` — correct behaviour |\n", "| **Cost** | `top_k` extra calls per task | Cache reflection per (query_hash, doc_hash) |\n", "\n", "Extensions: (1) weighted combination of three tokens, (2) skip reflection if first doc is fully_relevant + fully_supported (early termination), (3) hierarchical reflection on sub-claims.\n", "\n", "References:\n", "- Asai et al., *Self-RAG.* 2024. [arXiv:2310.11511](https://arxiv.org/abs/2310.11511)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" }, "papermill": { "default_parameters": {}, "duration": 33.573079, "end_time": "2026-05-28T02:48:03.445963+00:00", "environment_variables": {}, "exception": null, "input_path": "all-agentic-architectures/notebooks/25_self_rag.ipynb", "output_path": "all-agentic-architectures/notebooks/25_self_rag.ipynb", "parameters": {}, "start_time": "2026-05-28T02:47:29.872884+00:00", "version": "2.7.0" } }, "nbformat": 4, "nbformat_minor": 5 }