"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from IPython.display import Image, display\n",
"arch = Ensemble()\n",
"graph = arch.build()\n",
"display(Image(graph.get_graph().draw_mermaid_png()))"
]
},
{
"cell_type": "markdown",
"id": "d28b0d20",
"metadata": {
"papermill": {
"duration": 0.002999,
"end_time": "2026-05-27T12:43:20.609589+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:20.606590+00:00",
"status": "completed"
},
"tags": []
},
"source": [
"## 8 · Live run — contested forecasting question\n",
"\n",
"Concrete task: a *contested* forward-looking question where reasonable people genuinely disagree. The point: see if our 3 perspectives produce 3 different answers, and how the aggregator synthesises them."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "5642baa5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-27T12:43:20.617547Z",
"iopub.status.busy": "2026-05-27T12:43:20.616547Z",
"iopub.status.idle": "2026-05-27T12:43:41.542083Z",
"shell.execute_reply": "2026-05-27T12:43:41.542083Z"
},
"papermill": {
"duration": 20.929534,
"end_time": "2026-05-27T12:43:41.542083+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:20.612549+00:00",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"Aggregated answer (llm_synth) ─────────────────────────────────────────────────────────────────────────────────────\n",
"\n"
],
"text/plain": [
"\u001b[1;36mAggregated answer \u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36mllm_synth\u001b[0m\u001b[1;36m)\u001b[0m \u001b[92m─────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Based on the perspectives provided, it is unlikely that electric vehicles will account for over 50% of new car \n",
"sales globally by 2030, due to current infrastructure and production limitations. The majority of voters, including\n",
"the Skeptical and Pragmatic ones, highlight significant hurdles such as high upfront costs, limited charging \n",
"infrastructure, and inadequate production capacity. \n",
"\n",
"There are points of agreement among the voters, including the impact of declining battery costs and the role of \n",
"government regulations in promoting electric vehicle adoption. All voters also acknowledge the importance of \n",
"factors such as global economic conditions, technological advancements, and consumer preferences in influencing \n",
"electric vehicle sales. \n",
"\n",
"However, genuine disagreements exist regarding the pace of electric vehicle adoption, with the Analytical voter \n",
"being more optimistic about the industry's ability to meet the 50% target. The Skeptical and Pragmatic voters, on \n",
"the other hand, are more cautious due to the existing limitations. Given the uncertainty and complexity of the \n",
"issue, it is recommended to continue monitoring the progress of electric vehicle technology, infrastructure \n",
"development, and government policies, as these factors will ultimately determine the feasibility of achieving the \n",
"50% target by 2030. \n",
"\n"
],
"text/plain": [
"Based on the perspectives provided, it is unlikely that electric vehicles will account for over 50% of new car \n",
"sales globally by 2030, due to current infrastructure and production limitations. The majority of voters, including\n",
"the Skeptical and Pragmatic ones, highlight significant hurdles such as high upfront costs, limited charging \n",
"infrastructure, and inadequate production capacity. \n",
"\n",
"There are points of agreement among the voters, including the impact of declining battery costs and the role of \n",
"government regulations in promoting electric vehicle adoption. All voters also acknowledge the importance of \n",
"factors such as global economic conditions, technological advancements, and consumer preferences in influencing \n",
"electric vehicle sales. \n",
"\n",
"However, genuine disagreements exist regarding the pace of electric vehicle adoption, with the Analytical voter \n",
"being more optimistic about the industry's ability to meet the 50% target. The Skeptical and Pragmatic voters, on \n",
"the other hand, are more cautious due to the existing limitations. Given the uncertainty and complexity of the \n",
"issue, it is recommended to continue monitoring the progress of electric vehicle technology, infrastructure \n",
"development, and government policies, as these factors will ultimately determine the feasibility of achieving the \n",
"50% target by 2030. \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"VOTERS: ['analytical', 'skeptical', 'pragmatic']\n",
"CONFIDENCES: [4, 4, 4]\n",
"CONFIDENCE_SPREAD: 0\n"
]
}
],
"source": [
"TASK = (\n",
" \"Will electric vehicles account for over 50% of new car sales globally by 2030? \"\n",
" \"Answer YES or NO with a 2-3 sentence rationale.\"\n",
")\n",
"\n",
"result = arch.run(TASK)\n",
"\n",
"print_header(\"Aggregated answer (llm_synth)\")\n",
"print_md(result.output)\n",
"print()\n",
"print(f\"VOTERS: {result.state['voters_used']}\")\n",
"print(f\"CONFIDENCES: {result.metadata['confidences']}\")\n",
"print(f\"CONFIDENCE_SPREAD: {result.metadata['confidence_spread']}\")"
]
},
{
"cell_type": "markdown",
"id": "96d33310",
"metadata": {
"papermill": {
"duration": 0.0,
"end_time": "2026-05-27T12:43:41.542083+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:41.542083+00:00",
"status": "completed"
},
"tags": []
},
"source": [
"### 8.0 · What just happened, briefly\n",
"\n",
"Three things to look at:\n",
"\n",
"- **Voter disagreement** — read each voter's bottom_line in §8.1. If all 3 agree, the question wasn't actually contested or the perspective prompts didn't activate distinct framings.\n",
"- **Confidence spread** — if everyone is 4/5 (flat), the LLM-as-Scorer pathology again (see Mental Loop nb 10 §9). Bottom-line *content* discrimination matters more than the confidence number on a contested question.\n",
"- **Aggregator quality** — does the synthesis preserve genuine disagreement or wash it out?"
]
},
{
"cell_type": "markdown",
"id": "dbed2e24",
"metadata": {
"papermill": {
"duration": 0.015971,
"end_time": "2026-05-27T12:43:41.558054+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:41.542083+00:00",
"status": "completed"
},
"tags": []
},
"source": [
"### 8.1 · Per-voter opinions"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "67eda73c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-27T12:43:41.558054Z",
"iopub.status.busy": "2026-05-27T12:43:41.558054Z",
"iopub.status.idle": "2026-05-27T12:43:41.609559Z",
"shell.execute_reply": "2026-05-27T12:43:41.608877Z"
},
"papermill": {
"duration": 0.051505,
"end_time": "2026-05-27T12:43:41.609559+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:41.558054+00:00",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"› === ANALYTICAL (confidence 4/5) ===\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m=== ANALYTICAL \u001b[0m\u001b[1m(\u001b[0m\u001b[1mconfidence \u001b[0m\u001b[1;36m4\u001b[0m\u001b[1m/\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1m)\u001b[0m\u001b[1m ===\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"BOTTOM LINE: YES, electric vehicles will account for over 50% of new car sales globally by 2030, driven by \n",
"declining battery costs and increasing government regulations.\n",
"\n"
],
"text/plain": [
"BOTTOM LINE: YES, electric vehicles will account for over \u001b[1;36m50\u001b[0m% of new car sales globally by \u001b[1;36m2030\u001b[0m, driven by \n",
"declining battery costs and increasing government regulations.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"The cost of electric vehicle batteries has fallen by over 80% in the last decade, making them more competitive with\n",
"internal combustion engines.\n",
"\n"
],
"text/plain": [
"The cost of electric vehicle batteries has fallen by over \u001b[1;36m80\u001b[0m% in the last decade, making them more competitive with\n",
"internal combustion engines.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Many countries have set targets for electric vehicle adoption, such as Norway's goal of 100% electric vehicle sales\n",
"by 2025.\n",
"\n"
],
"text/plain": [
"Many countries have set targets for electric vehicle adoption, such as Norway's goal of \u001b[1;36m100\u001b[0m% electric vehicle sales\n",
"by \u001b[1;36m2025\u001b[0m.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Several major automakers, including Volkswagen and General Motors, have announced plans to electrify their entire \n",
"lineups in the coming years.\n",
"\n"
],
"text/plain": [
"Several major automakers, including Volkswagen and General Motors, have announced plans to electrify their entire \n",
"lineups in the coming years.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
},
{
"data": {
"text/html": [
"› === SKEPTICAL (confidence 4/5) ===\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m=== SKEPTICAL \u001b[0m\u001b[1m(\u001b[0m\u001b[1mconfidence \u001b[0m\u001b[1;36m4\u001b[0m\u001b[1m/\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1m)\u001b[0m\u001b[1m ===\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"BOTTOM LINE: NO\n",
"\n"
],
"text/plain": [
"BOTTOM LINE: NO\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"The adoption of electric vehicles is hindered by high upfront costs and limited charging infrastructure in many \n",
"parts of the world.\n",
"\n"
],
"text/plain": [
"The adoption of electric vehicles is hindered by high upfront costs and limited charging infrastructure in many \n",
"parts of the world.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Government policies and incentives play a crucial role in promoting electric vehicle sales, but their impact can \n",
"vary greatly by region.\n",
"\n"
],
"text/plain": [
"Government policies and incentives play a crucial role in promoting electric vehicle sales, but their impact can \n",
"vary greatly by region.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Global economic conditions, technological advancements, and consumer preferences are also significant factors that \n",
"can influence the sales of electric vehicles.\n",
"\n"
],
"text/plain": [
"Global economic conditions, technological advancements, and consumer preferences are also significant factors that \n",
"can influence the sales of electric vehicles.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
},
{
"data": {
"text/html": [
"› === PRAGMATIC (confidence 4/5) ===\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m=== PRAGMATIC \u001b[0m\u001b[1m(\u001b[0m\u001b[1mconfidence \u001b[0m\u001b[1;36m4\u001b[0m\u001b[1m/\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1m)\u001b[0m\u001b[1m ===\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"BOTTOM LINE: NO, electric vehicles will not account for over 50% of new car sales globally by 2030, due to current \n",
"infrastructure and production limitations.\n",
"\n"
],
"text/plain": [
"BOTTOM LINE: NO, electric vehicles will not account for over \u001b[1;36m50\u001b[0m% of new car sales globally by \u001b[1;36m2030\u001b[0m, due to current \n",
"infrastructure and production limitations.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Current charging infrastructure is not yet widespread enough to support a majority of electric vehicles\n",
"\n"
],
"text/plain": [
"Current charging infrastructure is not yet widespread enough to support a majority of electric vehicles\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Production capacity for electric vehicles is still ramping up and may not meet demand by 2030\n",
"\n"
],
"text/plain": [
"Production capacity for electric vehicles is still ramping up and may not meet demand by \u001b[1;36m2030\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"› point\n",
"\n"
],
"text/plain": [
"\u001b[1;35m›\u001b[0m \u001b[1m point\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Many countries still lack clear policies and incentives to drive adoption of electric vehicles\n",
"\n"
],
"text/plain": [
"Many countries still lack clear policies and incentives to drive adoption of electric vehicles\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"for t in result.trace:\n",
" print_step(\n",
" f\"=== {t['voter'].upper()} (confidence {t['confidence']}/5) ===\",\n",
" f\"BOTTOM LINE: {t['bottom_line']}\"\n",
" )\n",
" for pt in t.get('key_points', []):\n",
" print_step(\" point\", pt[:200])\n",
" print()"
]
},
{
"cell_type": "markdown",
"id": "56c1e111",
"metadata": {
"papermill": {
"duration": 0.004009,
"end_time": "2026-05-27T12:43:41.618027+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:41.614018+00:00",
"status": "completed"
},
"tags": []
},
"source": [
"## 9 · What we just observed\n",
"\n",
"The cells above ran 3 voters (analytical / skeptical / pragmatic) against the same contested forecasting question, then aggregated their opinions.\n",
"\n",
"### 9.1 · Quantitative summary\n",
"\n",
"| Metric | Value |\n",
"|---|---|\n",
"| Voters run | 3 |\n",
"| Confidence values | [4, 4, 4] |\n",
"| Confidence spread | 0 |\n",
"| Voters who answered YES/likely | 2 |\n",
"| Voters who answered NO/doubt | 2 |\n",
"\n",
"### 9.2 · Per-voter bottom-line answers\n",
"\n",
"| Voter | Confidence | Bottom line |\n",
"|---|---|---|\n",
"| analytical | 4/5 | YES, electric vehicles will account for over 50% of new car sales globally by 2030, driven by declining battery costs and increasing government regulations. |\n",
"| skeptical | 4/5 | NO |\n",
"| pragmatic | 4/5 | NO, electric vehicles will not account for over 50% of new car sales globally by 2030, due to current infrastructure and production limitations. |\n",
"\n",
"### 9.3 · Patterns surfaced in this run\n",
"\n",
"- **Flat confidence scores** — all 3 voters reported 4/5 confidence. The familiar Llama-as-Scorer pathology (see Mental Loop nb 10 §9). Self-reported confidence is unreliable as a signal — focus on the CONTENT of each bottom-line instead.\n",
"\n",
"- **Genuine perspective disagreement detected.** At least one voter said YES/likely AND another said NO/doubt. This is what Ensemble is FOR — different perspectives producing different directional answers on the same question.\n",
"\n",
"- **Aggregator preserved nuance** — used hedging language ('however', 'uncertainty', etc.) in the synthesis, suggesting minority views weren't washed out.\n",
"\n",
"### 9.4 · The aggregated answer (verbatim)\n",
"\n",
"> Based on the perspectives provided, it is unlikely that electric vehicles will account for over 50% of new car sales globally by 2030, due to current infrastructure and production limitations. The majority of voters, including the Skeptical and Pragmatic ones, highlight significant hurdles such as high upfront costs, limited charging infrastructure, and inadequate production capacity. There are points of agreement among the voters, including the impact of declining battery costs and the role of government regulations in promoting electric vehicle adoption. All voters also acknowledge the impor…\n",
"\n",
"### 9.5 · The takeaway\n",
"\n",
"A *healthy* Ensemble run has:\n",
"\n",
"1. **Genuine disagreement** — at least 2 of K voters produce different directional answers.\n",
"2. **Aggregator preserves nuance** — hedging language ('however', 'on the other hand') in the synthesis.\n",
"3. **Confidence values are NOT the signal** — they're noisy. The bottom-line CONTENT discrimination matters.\n",
"4. **The synthesis is shorter than the sum of inputs** — extracts insight, doesn't just concatenate."
]
},
{
"cell_type": "markdown",
"id": "cf0e971b",
"metadata": {
"papermill": {
"duration": 0.004011,
"end_time": "2026-05-27T12:43:41.628065+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:41.624054+00:00",
"status": "completed"
},
"tags": []
},
"source": [
"## 10 · `majority_vote` mode — the deterministic-picker fix\n",
"\n",
"The default `llm_synth` mode in § 8 uses the *content* of each voter's `bottom_line`, so it works even when confidence scores are flat. But what if you need a *decisive single answer* (YES / NO / A / B) — not a balanced synthesis?\n",
"\n",
"The naive approach is `highest_confidence` mode: pick the voter with the highest self-reported confidence number. **This is broken on Llama** — confidences come back `[4, 4, 4]` and the argmax is arbitrary.\n",
"\n",
"The fix is `majority_vote` mode: voters emit a categorical answer (e.g. `\"YES\"`), and **Python tallies the votes** deterministically. Confidence numbers are ignored entirely. This is the same pattern as Mental Loop's `scoring_fn` (notebook 10) — **let the LLM predict the underlying signal, let Python compute the picker**."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "dc162679",
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-27T12:43:41.639868Z",
"iopub.status.busy": "2026-05-27T12:43:41.639868Z",
"iopub.status.idle": "2026-05-27T12:43:51.534803Z",
"shell.execute_reply": "2026-05-27T12:43:51.534039Z"
},
"papermill": {
"duration": 9.902302,
"end_time": "2026-05-27T12:43:51.536380+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:41.634078+00:00",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"Mode: majority_vote (deterministic Python picker) ─────────────────────────────────────────────────────────────────\n",
"\n"
],
"text/plain": [
"\u001b[1;36mMode: majority_vote \u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36mdeterministic Python picker\u001b[0m\u001b[1;36m)\u001b[0m \u001b[92m─────────────────────────────────────────────────────────────────\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Majority answer: YES (2/3 voters) — tally: YES=2, NO=1. \n",
"\n",
"Supporting voters' bottom lines: \n",
"\n",
" • (analytical) Electric vehicles will likely account for over 50% of new car sales globally by 2030 due to \n",
" decreasing battery costs and increasing government regulations. \n",
" • (pragmatic) Electric vehicles will likely account for over 50% of new car sales globally by 2030 due to \n",
" decreasing battery costs, government incentives, and growing consumer demand. \n",
"\n",
"Dissenting voter(s): \n",
"\n",
" • (skeptical -> NO) I do not think electric vehicles will account for over 50% of new car sales globally by 2030. \n",
"\n"
],
"text/plain": [
"\u001b[1mMajority answer: YES\u001b[0m (2/3 voters) — tally: YES=2, NO=1. \n",
"\n",
"Supporting voters' bottom lines: \n",
"\n",
"\u001b[1m • \u001b[0m(analytical) Electric vehicles will likely account for over 50% of new car sales globally by 2030 due to \n",
"\u001b[1m \u001b[0mdecreasing battery costs and increasing government regulations. \n",
"\u001b[1m • \u001b[0m(pragmatic) Electric vehicles will likely account for over 50% of new car sales globally by 2030 due to \n",
"\u001b[1m \u001b[0mdecreasing battery costs, government incentives, and growing consumer demand. \n",
"\n",
"Dissenting voter(s): \n",
"\n",
"\u001b[1m • \u001b[0m(skeptical -> NO) I do not think electric vehicles will account for over 50% of new car sales globally by 2030. \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"VOTE_TALLY (Python-computed): {'YES': 2, 'NO': 1}\n",
"CATEGORICAL_ANSWERS (LLM-supplied): ['YES', None, 'YES']\n",
"CONFIDENCES (unused for argmax — note flatness): [4, 4, 4]\n"
]
}
],
"source": [
"print_header(\"Mode: majority_vote (deterministic Python picker)\")\n",
"mv_arch = Ensemble(aggregator_mode=\"majority_vote\")\n",
"mv_result = mv_arch.run(\n",
" \"Will electric vehicles account for over 50% of new car sales globally by 2030? Answer YES, NO, or UNCERTAIN.\"\n",
")\n",
"print_md(mv_result.output[:700])\n",
"print()\n",
"print(f\"VOTE_TALLY (Python-computed): {mv_result.metadata['vote_tally']}\")\n",
"print(f\"CATEGORICAL_ANSWERS (LLM-supplied): {mv_result.metadata['categorical_answers']}\")\n",
"print(f\"CONFIDENCES (unused for argmax — note flatness): {mv_result.metadata['confidences']}\")"
]
},
{
"cell_type": "markdown",
"id": "12fec1f3",
"metadata": {
"papermill": {
"duration": 0.002138,
"end_time": "2026-05-27T12:43:51.550363+00:00",
"exception": false,
"start_time": "2026-05-27T12:43:51.548225+00:00",
"status": "completed"
},
"tags": []
},
"source": [
"## 11 · Failure modes, safety, extensions\n",
"\n",
"### 11.1 · Where this breaks\n",
"\n",
"| Failure | Mechanism | Mitigation |\n",
"|---|---|---|\n",
"| **Flat confidences** | Same Llama-as-Scorer pathology — all confidence 4/5 | **Use `aggregator_mode=\"majority_vote\"`** (see § 10) — Python tallies discrete `categorical_answer` values, sidestepping the flat-confidence noise |\n",
"| **Aggregator washout** | Synthesis blends opinions so minority view disappears | Aggregator prompt EXPLICITLY asks to preserve disagreement (we do this); still happens occasionally |\n",
"| **Hidden conformity** | All 3 voters give the same answer despite different prompts | Run higher temperature; or use genuinely-different LLMs (gpt-4o + claude + llama) |\n",
"| **Voters skip categorical_answer** | Llama leaves optional field null even after instruction | Library has keyword-fallback: scans `bottom_line` for YES/NO/UNCERTAIN if `categorical_answer` is missing |\n",
"| **Adversarial-voice winner** | Skeptical voter's \"what could go wrong\" is loudest | Counterbalance with explicit \"what could go right\" voter |\n",
"| **Cost** | K voter calls + 1 aggregator = K+1 LLM calls per task | Don't use Ensemble for cheap tasks; reserve for high-stakes decisions |\n",
"\n",
"### 11.2 · Production safety\n",
"\n",
"- **Confidence is unreliable.** Don't expose voters' self-reported confidence to users as if it were calibrated probability — it isn't.\n",
"- **Aggregator is a single point of failure.** If the aggregator LLM gets a bad seed, the whole ensemble's output is corrupted. Run the aggregator 2-3 times and pick the best (meta-ensemble).\n",
"- **Voter diversity matters more than count.** 3 *genuinely different* voters > 7 paraphrase-of-same voters.\n",
"\n",
"### 11.3 · Three extensions\n",
"\n",
"1. **Parallel voter execution.** Replace the sequential `_vote` loop with `langgraph.graph.parallel` — K× latency win.\n",
"2. **Voters are full architectures.** Instead of 3 LLM calls with different prompts, use 3 *different architectures* (ReAct + Reflection + Planning) on the same task. Higher cost, much richer ensemble.\n",
"3. **Confidence-weighted aggregation.** Replace the LLM aggregator with a weighted majority vote where each voter's contribution is weighted by their confidence. (Use after fixing self-reported confidence calibration.)\n",
"\n",
"### 11.4 · What to read next\n",
"\n",
"- [**05 · Multi-Agent**](./05_multi_agent.ipynb) — specialists with DIFFERENT sub-tasks vs Ensemble's same-task voters.\n",
"- [**07 · Blackboard**](./07_blackboard.ipynb) — distributed bidding instead of fan-out.\n",
"- [**21 · Self-Consistency**](./21_self_consistency.ipynb) — N samples + majority vote (Ensemble's simpler cousin).\n",
"- [**28 · Multi-Agent Debate**](./28_agent_debate.ipynb) — adversarial back-and-forth instead of one-shot vote.\n",
"\n",
"### 11.5 · References\n",
"\n",
"1. Surowiecki, J. *The Wisdom of Crowds.* 2004.\n",
"2. Wang, X. et al. *Self-Consistency Improves Chain-of-Thought Reasoning in Language Models.* ICLR 2023. [arXiv:2203.11171](https://arxiv.org/abs/2203.11171)\n",
"3. Du, Y. et al. *Improving Factuality and Reasoning in Language Models through Multiagent Debate.* 2023. [arXiv:2305.14325](https://arxiv.org/abs/2305.14325)\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
},
"papermill": {
"default_parameters": {},
"duration": 37.898106,
"end_time": "2026-05-27T12:43:52.663198+00:00",
"environment_variables": {},
"exception": null,
"input_path": "all-agentic-architectures/notebooks/13_ensemble.ipynb",
"output_path": "all-agentic-architectures/notebooks/13_ensemble.ipynb",
"parameters": {},
"start_time": "2026-05-27T12:43:14.765092+00:00",
"version": "2.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}