--- status: active owner: backend-mlx summary: The Phase 2-3 language bench — hint-contract and on-device output verification matrices, subset semantics, Speech asset prerequisites, and how to read hint_gate/output_gate verdicts. sourceOfTruth: - scripts/check_language_hints.py - scripts/check_language_output.py - config/language-bench-matrix.json --- # Language bench (Phases 2–3) Headless matrix for the Qwen3 language path: 1. **Phase 2 — hint contract:** UI hint → resolved `notes.languageHint` in engine telemetry. 2. **Phase 3 — output verification:** three-pass locale-locked on-device Speech consensus, language score, WER/CER, and exact fixed-seed WAV proof vs script. ## Config | File | Role | | --- | --- | | `config/language-bench-corpus.json` | Versioned scripts plus Custom speaker and Design delivery fixtures per language | | `config/language-bench-matrix.json` | Cells: mode, `uiHint`, `scriptLang`, `expectedHint` | | `config/language-bench-diagnostic-cohort.json` | Fixed cells and five predeclared seeds for autonomous failure diagnosis | Cells tagged `"quick": true` form the **quick** subset (English + French + negative control, 7 cells). **full** runs all 19 cells (6 languages × Custom pinned/Auto + Design explicit-language + negative). The version-2 corpus is deliberately longer than the original smoke snippets: each alphabetic script contains at least 15 normalized words and each Chinese/Japanese script contains at least 24 normalized characters. Design always receives the known target language explicitly. Custom uses a native-language speaker where the Qwen speaker contract provides one (Chinese `vivian`, Japanese `ono_anna`); the remaining languages use the contract's stable `aiden` fixture. The paired Custom pinned/Auto cells intentionally generate the same prompt with the same speaker, seed, and sampling policy. They prove that Auto resolves equivalently to the pinned hint; they are not independent audio samples. Likewise, the three sequential Speech recognitions prove that the on-device recognizer reproduced one transcript for one WAV. They do not provide three statistically independent accuracy observations. The 18 output cells remain strict per-cell multilingual smoke acceptance, not a population estimate of language quality. ## iOS (on-device) Requires Built-in Voice and Voice Design **Speed** installed on the paired iPhone. **Speech Recognition (app):** Phase 3 transcribes each output WAV in the app process. Grant **Settings → Privacy → Speech Recognition → Vocello** once before the first output-gated run. ### Phase 3 prerequisites (on-device Speech assets) Output verification runs three sequential recognitions of the exact generated WAV using **on-device Speech** in the deterministic locale of each cell. All three final transcripts must agree before WER/CER is scored. EN/FR work out of the box; **DE, ES, ZH, JA** need system dictation languages and downloaded voice assets on the phone. Authorization denied, recognizer unavailable, missing on-device support, timeout, engine error, inconsistent transcripts, or failed WER/CER are distinct machine failures; none is replaced with a fabricated score or a listening judgment. The versioned `normalized-edit-rate-v1` accuracy contract uses **WER ≤ 0.15** for languages with word boundaries and **CER ≤ 0.15** for Chinese and Japanese; both scores and both word/character edit-count decompositions remain evidence. The Python gate and history publisher independently recompute the metrics from the tracked corpus and untracked consensus transcript before accepting the Swift verdict. The host checker also enforces the app's existing **outer-edge** timing rule whenever the verification declares `sourceAudioDurationSeconds` or the record includes WAV `outputEvidence`: all three recognitions must start within `min(2.5, max(1.0, duration × 0.15))` seconds of the beginning and end within that allowance of the end, without exceeding the duration by more than 0.25 seconds. Supplied durations must be positive, finite and mutually consistent. Missing timings, invalid durations or incomplete edges cannot authorize a claimed PASS. The VLR composer classifies this as a harness/evidence gap, preserving already typed incomplete-coverage results. Legacy reports lacking both duration fields retain historical validation; they are not fresh full-WAV proof. Edge coverage does **not** establish interior completeness or waive WER/CER. Independent recognizers disagreeing on the same WAV remain diagnostic evidence; never choose the better score, substitute their transcripts, or relabel a retained failure automatically. **One-time setup (on the iPhone — Settings app, not Vocello):** 1. **Keyboards:** Settings → General → Keyboard → Keyboards → Add keyboard — e.g. Allemand, Espagnol, Japonais (Romaji), Chinois simplifié (Pinyin QWERTY). 2. **Dictation languages:** Settings → search *dictée* → **Langues de Dictée** — enable Allemand, Espagnol, Japonais, Mandarin (and any variants listed for your locale). 3. **Explicit asset bootstrap:** With the unlocked phone on Wi‑Fi, run `scripts/ios_device.sh speech-assets`. Vocello resolves the device-supported equivalents for `de_DE`, `es_419`, `ja_JP`, and `zh_CN`, creates DictationTranscriber modules, checks AssetInventory before and after one combined `downloadAndInstall()` request, and requires every resolved locale to report installed. The command also prints a separate `vocello_legacy_gate` verdict from fresh SFSpeechRecognizer instances and the same deterministic locale-selection policy used by the output verifier. 4. **Interpret both results:** `asset_inventory=PASS` proves the modern assets installed. `vocello_legacy_gate=PASS` is additionally required by the current Phase 3 verifier. If the modern gate passes but the legacy gate remains blocked, do not claim language readiness or run the full matrix as promotion evidence; preserve the local diagnostic result and investigate the OS-level legacy recognizer state. 5. **Re-run** once both are ready: `scripts/ios_device.sh lang-bench --subset full --label "lang-full-output-v3"`. Settings remains useful for confirming enabled Dictation languages, but the explicit command owns asset installation and machine verification. This is an operational prerequisite, not a subjective audio review. Vocello UI expectations remain documented in [`ios-ui-reference.md`](ios-ui-reference.md). Language-benchmark labels are opaque privacy-safe identifiers matching `[A-Za-z0-9][A-Za-z0-9._-]{0,95}`; they are not free-form notes. ```sh scripts/ios_device.sh lang-bench --subset quick --label "lang-smoke" scripts/ios_device.sh lang-bench --subset full --label "lang-full" scripts/ios_device.sh lang-bench --diagnostic-cohort ``` Skip output verification (hint gate only): ```sh QVOICE_LANG_BENCH_SKIP_OUTPUT=1 scripts/ios_device.sh lang-bench --subset quick ``` Per cell the driver sets: - `QVOICE_IOS_DEVICE_RUN_ID` — shared run id (`ios-lang-bench-…`) - `QVOICE_MAC_BENCH_CELL` — matrix cell id - `QVOICE_IOS_DEVICE_DIAGNOSTICS_LANGUAGE` — language hint (`english`, `french`, …; omitted for Auto) - `QVOICE_IOS_DEVICE_DIAGNOSTICS_SPEC` — `mode:speed: