{ "run_id": "fault-tolerance-20260623", "utc": "2026-06-23", "shard_commit": "f3fba2d05b88e965ccb0a01c46df136300375873", "claim": "Mid-request fault tolerance: kill a node mid-generation UNDER LOAD and the SAME request still completes (not a restart). The engine resume primitive (coordinate_pipe resume_ids/resumable) + a control-plane heal (spare splice) preserve the committed output and continue.", "test": "mid-request-fault-tolerance", "model": "gpt-oss-120b", "nstages": 4, "victim_stage": 1, "victim_id": 42234402, "spare_id": 42234410, "kill_after_s": 25, "committed_before_drop": 189, "total_after_resume": 256, "continuation_preserved": true, "failover_s": 131.0, "output_text": "analysisNeed to produce detailed explanation >250 words.assistantfinal### Overview \n\nA distributed key\u2011value store (DKVS) is a cluster of machines that together present a single logical map **K \u2192 V**. The system must keep this map **consistent** (all clients see the same value for a given key) and **available** (clients can read or write even when some nodes are down). Achieving both goals simultaneously is a classic trade\u2011off (the CAP theorem), so real\u2011world DKVSs adopt a carefully engineered combination of **replication**, **failure detection**, **leader election**, **re\u2011replication**, and **rebalancing**. Below is a step\u2011by\u2011step walk\u2011through of how these mechanisms interact when a node fails.\n\n---\n\n## 1. Replication \u2013 the foundation of durability and availability \n\n1. **Data partitioning** \u2013 The key space is split into *shards* (also called *partitions* or *ranges*) using a deterministic hash function (e.g., consistent hashing or Rendezvous hashing). \n2. **Replica set** \u2013 For each shard the system creates **N** copies (the *replication factor*). \n * Example:", "committed_ids": [ 200005, 35644, 200008, 23483, 316, 10635, 14633, 30547, 1424, 6911, 6391, 13, 200007, 200006, 173781, 200005, 17196, 200008, 31639, 41854, 11691, 32, 7581, 2140, 50005, 1594, 4897, 350, 30283, 35688, 8, 382, 261, 19723, 328, 14656, 484, 4717, 3333, 261, 4590, 36574, 4014, 6240, 42, 15155, 631, 410, 13, 220, 623, 2420, 2804, 3357, 495, 4014, 6240, 173878, 410, 350, 586, 7485, 1921, 290, 2684, 1432, 395, 261, 4335, 2140, 8, 326, 6240, 17006, 410, 350, 68893, 665, 1729, 503, 5067, 1952, 1261, 1236, 15749, 553, 1917, 741, 220, 31526, 41907, 2973, 11366, 38328, 382, 261, 13686, 10552, 50005, 2864, 350, 3086, 37257, 177750, 936, 813, 1374, 50005, 24169, 97690, 35688, 82, 17022, 261, 18455, 67095, 14083, 328, 6240, 17407, 2389, 191888, 6240, 63591, 28385, 191888, 6240, 68083, 14254, 191888, 6240, 264, 50005, 17407, 2389, 191888, 326, 6240, 264, 13738, 12551, 410, 13, 220, 35307, 382, 261, 5983, 50005, 2345, 50005, 17509, 5229, 50005, 32099, 328, 1495, 1879, 37230, 20255, 1261, 261, 4596, 28336, 364, 103995, 877, 220, 16, 13, 10577, 2389, 1127, 290, 21915, 328, 42008, 326, 18744, 11691, 16, 13, 6240, 1186, 31512, 289, 410, 1127, 623 ], "output_ids": [ 200005, 35644, 200008, 23483, 316, 10635, 14633, 30547, 1424, 6911, 6391, 13, 200007, 200006, 173781, 200005, 17196, 200008, 31639, 41854, 11691, 32, 7581, 2140, 50005, 1594, 4897, 350, 30283, 35688, 8, 382, 261, 19723, 328, 14656, 484, 4717, 3333, 261, 4590, 36574, 4014, 6240, 42, 15155, 631, 410, 13, 220, 623, 2420, 2804, 3357, 495, 4014, 6240, 173878, 410, 350, 586, 7485, 1921, 290, 2684, 1432, 395, 261, 4335, 2140, 8, 326, 6240, 17006, 410, 350, 68893, 665, 1729, 503, 5067, 1952, 1261, 1236, 15749, 553, 1917, 741, 220, 31526, 41907, 2973, 11366, 38328, 382, 261, 13686, 10552, 50005, 2864, 350, 3086, 37257, 177750, 936, 813, 1374, 50005, 24169, 97690, 35688, 82, 17022, 261, 18455, 67095, 14083, 328, 6240, 17407, 2389, 191888, 6240, 63591, 28385, 191888, 6240, 68083, 14254, 191888, 6240, 264, 50005, 17407, 2389, 191888, 326, 6240, 264, 13738, 12551, 410, 13, 220, 35307, 382, 261, 5983, 50005, 2345, 50005, 17509, 5229, 50005, 32099, 328, 1495, 1879, 37230, 20255, 1261, 261, 4596, 28336, 364, 103995, 877, 220, 16, 13, 10577, 2389, 1127, 290, 21915, 328, 42008, 326, 18744, 11691, 16, 13, 6240, 1186, 31512, 289, 410, 1127, 623, 2140, 4918, 382, 12648, 1511, 425, 1116, 3098, 9, 350, 32038, 4358, 425, 5815, 5500, 9, 503, 425, 175054, 8162, 2360, 261, 164009, 10691, 1114, 350, 68, 1940, 4213, 20434, 185245, 503, 176358, 101896, 185245, 741, 4066, 17, 13, 6240, 129179, 920, 410, 1127, 2214, 2454, 110945, 290, 2420, 18518, 6240, 45, 410, 20138, 350, 3086, 425, 17407, 2389, 12332, 9, 741, 4066, 256, 425, 24855, 25 ], "detection_s": 4, "mechanism": "specpipe coordinate_pipe(resumable=True) returns the committed tokens on a dead edge instead of failing; phase0/heal.py (control-plane) detects it, splices the pre-warmed SPARE into the victim's slot (only the spare + the victim's predecessor are (re)launched \u2014 the other survivors auto-re-handshake their dropped links), then re-invokes the coordinator with --resume-file to re-prefill prompt+committed and continue to completion.", "topology": "N=4 even split (9 layers/box) WA->MN->NC->NJ over WAN; spare OH (Ohio) pre-holding layers [9:18]. Victim = stage1 (MN). All RTX 4090 24GB, distinct hosts/states.", "honesty": "failover ~131s is dominated by the COLD-spare reload (the spare had the weights on disk but not in VRAM, so it loaded 9 layers of the 120B; the head also reloaded as its --next changed) plus the re-prefill of prompt+189 committed tokens. A HOT pre-warmed standby (weights already in VRAM, idle in the ring) removes the reload, leaving only the re-prefill (~seconds at this short context). 'Fast re-prefill of JUST the dropped block' (vs the full prompt+committed) needs upstream activation checkpointing \u2014 identified, not built. What IS proven: a real node death mid-stream no longer fails the request; the committed output is preserved byte-continuous and generation resumes to completion." }