World-Class Operator Runbook
A single operating page for collecting the remaining human and external evidence. It coordinates action, but does not accept evidence or change the ledger.
Fast Path
- Run the real external or human work for one evidence item.
- Generate and fill the matching submission draft.
- Validate intake and inspect the submission review queue.
- Refresh the ledger and run the claim guard before making any completion claim.
Coordination Plan
These steps tell the assistant what to run and where the user must supply real provider, reviewer, client, or telemetry input. Every row is operational guidance only and keeps counts_as_completion false.
| Step | Evidence | Owner | User | User action | Assistant action | Command | Pass condition |
|---|---|---|---|---|---|---|---|
prepare-evidence-session | all | assistant + user | true | Confirm provider access, reviewer availability, target client path, and telemetry client path before collection starts. | Run preflight and prepare submission drafts without accepting them as evidence. | python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self && python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts --self | Preflight lists the same pending evidence keys and no credential values are printed. |
collect-provider-holdout | provider-holdout | assistant + operator with provider credentials | true | Provide DEEPSEEK_API_KEY through the environment for the fixed Flash+Pro matrix. | Run evidence-build, verify 40 governed calls and the private blind-review material boundary, then prepare the evidence packet. | python3 scripts/yao.py evidence-build . --run-id <PROVIDER_RUN_ID> --self | reports/provider_output_evaluation.json records 40/40 model calls, zero failures, and total_tokens <= 250000. |
collect-human-adjudication | human-adjudication | human reviewer + assistant | true | Have reviewer-a, reviewer-b, and reviewer-c independently complete the same 20-pair role-neutral pack through controlled submissions. | Verify the three packet identities and commitments, then finalize the source provider run without rerunning the matrix. | python3 scripts/yao.py evidence-finalize-review . --source-run <PROVIDER_RUN_ID> --decisions <A.json> --decisions <B.json> --decisions <C.json> --reviewer-registry <registry.json> --self | reports/provider_output_adjudication.json has reviewer_count == 3, pair_count == 20, and failure_count == 0. |
collect-native-permission-enforcement | native-permission-enforcement | target client or installer integrator + assistant | true | Select a real target client or external installer guard that can enforce declared capabilities instead of metadata-only fallback. | Run runtime permission probes, package verification, install simulation, and prepare the native enforcement evidence packet. | python3 scripts/yao.py runtime-permissions . --package-dir dist --self | reports/runtime_permission_probes.json has native_enforcement_count > 0 and failure_count == 0. |
collect-native-client-telemetry | native-client-telemetry | real client integrator + assistant | true | Install the native host manifest in a real Browser, Chrome, IDE, or provider client and trigger a metadata-only event. | Generate native host assets, import the external event JSONL, refresh adoption drift, and prepare the telemetry evidence packet. | python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --source external --self | reports/adoption_drift_report.json has source_types.external > 0 and adoption_sample_count > 0. |
review-and-release-gate | all | assistant + ledger reviewer | true | Approve only validated evidence packets and confirm the release wording after the claim guard passes. | Run intake, submission review, ledger, claim guard, benchmark, evidence consistency, Review Studio, and CI before final publish. | python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self && python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions --self && python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions --self && python3 scripts/yao.py world-class-claim-guard . --self && make ci-test | Ledger ready_to_claim_world_class, benchmark public_claim_ready, claim guard violation_count == 0, Review Studio has no blockers, and CI passes. |
Phase Queue
- Unblock access
unblock-access: 4 / 4Assign three independent controlled reviewer identities before claiming human adjudication. - Collect source
collect-source: 3 / 3Telemetry must include adoption outcome evidence.
Evidence Items
Provider Holdout
No evidence packet has been submitted for review.
- Owner
- operator with provider credentials
- Ledger
pending- Blocked
0- Queue
1- Submission
evidence/world_class/submissions/provider-holdout.json
Phase Queue
- Unblock access
unblock-access: 1 / 1Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.
Source Runbook
- Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.
- python3 scripts/yao.py evidence-build . --run-id <PROVIDER_RUN_ID> --self
- Keep the generated private answer key and role-neutral review materials inside .yao/runs/<PROVIDER_RUN_ID>.
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
- Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --self - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions --self - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions --self - guard claim
python3 scripts/yao.py world-class-claim-guard . --self
Must Collect
- provider-backed model run
- observed timing
- observed token metadata
Success Checks
- reports/provider_output_evaluation.json summary.call_count == 40
- reports/provider_output_evaluation.json summary.model_executed_count == 40
- reports/provider_output_evaluation.json summary.failure_count == 0
- reports/provider_output_evaluation.json summary.total_tokens <= 250000
- reports/skill_os2_audit.json item provider-holdout status becomes pass
Privacy
- Do not commit provider credentials or environment dumps.
- The output execution report records output hashes and aggregate run metadata, not raw provider prompts.
Next Source Actions
- No blocked source checks.
Source Evidence Snapshot
- Provider calls
call_count: 40 / ==40Complete all 40 fixed DeepSeek calls. - Provider model runs
model_executed_count: 40 / ==40Require model identity on all 40 calls. - Provider failures
failure_count: 0 / ==0Resolve every fixed-matrix failure. - Token budget
total_tokens: 40938 / <=250000Keep the matrix within the 250000-token ceiling.
Human Adjudication
No evidence packet has been submitted for review.
- Owner
- human reviewer
- Ledger
pending- Blocked
0- Queue
1- Submission
evidence/world_class/submissions/human-adjudication.json
Phase Queue
- Unblock access
unblock-access: 1 / 1Assign three independent controlled reviewer identities before claiming human adjudication.
Source Runbook
- Give each registered reviewer an independent copy of the matching provider_review_reviewer-*.json template and the role-neutral blind pack.
- Collect all 20 A/B choices, reasons, controlled submission ids, timestamps, and truthful independent-review attestations.
- Export an access-controlled reviewer registry that binds each reviewer id to the exact packet SHA256.
- python3 scripts/yao.py evidence-finalize-review . --source-run <PROVIDER_RUN_ID> --decisions <reviewer-a.json> --decisions <reviewer-b.json> --decisions <reviewer-c.json> --reviewer-registry <registry.json> --run-id <FINAL_RUN_ID> --self
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
- Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --self - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions --self - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions --self - guard claim
python3 scripts/yao.py world-class-claim-guard . --self
Must Collect
- real reviewer identity
- blind A/B decisions
- answer key unopened until decisions exist
Success Checks
- reports/provider_output_adjudication.json summary.reviewer_count == 3
- reports/provider_output_adjudication.json summary.pair_count == 20
- reports/provider_output_adjudication.json summary.failure_count == 0
- reports/provider_output_adjudication.json evidence_binding.blind_pack_sha256 matches the source run
- reports/skill_os2_audit.json item human-adjudication status becomes pass
Privacy
- Reviewer packets contain choices, reasons, hashes, and controlled submission metadata without raw prompts or answer-key roles.
- The private answer key remains under .yao/runs and is opened by the finalizer after all controlled packets are fixed.
- The adjudication and lineage artifacts preserve blind_pack_sha256 and answer_key_sha256 commitments.
Next Source Actions
- No blocked source checks.
Source Evidence Snapshot
- Registered reviewers
reviewer_count: 3 / ==3Collect reviewer-a, reviewer-b, and reviewer-c. - Blind pairs
pair_count: 20 / ==20Complete all 20 blind pairs. - Review failures
failure_count: 0 / ==0Resolve packet, identity, or adjudication failures. - Blind pack binding
blind_pack_bound: True / trueBind adjudication to the reviewed blind pack SHA256.
Native Permission Enforcement
No evidence packet has been submitted for review.
- Owner
- target client or installer integrator
- Ledger
pending- Blocked
1- Queue
2- Submission
evidence/world_class/submissions/native-permission-enforcement.json
Phase Queue
- Unblock access
unblock-access: 1 / 1Attach a real target-client or external installer runtime guard; metadata fallback is not enough. - Collect source
collect-source: 1 / 1Collect real target-client or external runtime guard proof.
Source Runbook
- Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
- Update the generated target adapter only when the guard is actually enforced by that target.
- python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip --self
- python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation --self
- python3 scripts/yao.py runtime-permissions . --package-dir dist --self
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
- Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --self - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions --self - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions --self - guard claim
python3 scripts/yao.py world-class-claim-guard . --self
Must Collect
- real target client or external installer runtime guard
- native enforcement flag or externally accepted guard proof
- residual risk retained for fallback targets
Success Checks
- reports/runtime_permission_probes.json summary.native_enforcement_count > 0
- reports/runtime_permission_probes.json summary.failure_count == 0
- reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
- reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
Privacy
- Do not mark native_enforcement true for metadata-only fallbacks.
- Keep residual risks visible for targets that still rely on operator enforcement.
Next Source Actions
- Collect real target-client or external runtime guard proof.
Source Evidence Snapshot
- Native enforcement
native_enforcement_count: 0 / >0Collect real target-client or external runtime guard proof. - Probe failures
failure_count: 0 / ==0Runtime permission probes must stay clean. - Installer support
installer_enforcement_ready: True / trueInstaller enforcement is supporting evidence, not native proof.
Native Client Telemetry
No evidence packet has been submitted for review.
- Owner
- Browser/Chrome/IDE/provider client integrator
- Ledger
pending- Blocked
2- Queue
2- Submission
evidence/world_class/submissions/native-client-telemetry.json
Phase Queue
- Unblock access
unblock-access: 1 / 1Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events. - Collect source
collect-source: 2 / 2Telemetry must include adoption outcome evidence.
Source Runbook
- python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
- Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
- python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --self
- python3 scripts/yao.py skill-atlas --workspace-root . --self
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
- Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --self - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions --self - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions --self - guard claim
python3 scripts/yao.py world-class-claim-guard . --self
Must Collect
- real external client source
- metadata-only event
- local-first import path
Success Checks
- reports/adoption_drift_report.json summary.source_types.external > 0
- reports/adoption_drift_report.json summary.adoption_sample_count > 0
- reports/skill_os2_audit.json item native-client-telemetry status becomes pass
Privacy
- Telemetry must remain metadata-only and local-first.
- Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.
Next Source Actions
- Import at least one metadata-only event from a real client.
- Telemetry must include adoption outcome evidence.
Source Evidence Snapshot
- External events
external_source_events: 0 / >0Import at least one metadata-only event from a real client. - Adoption sample
adoption_sample_count: 0 / >0Telemetry must include adoption outcome evidence. - Raw content blocked
raw_content_allowed: False / falseTelemetry must stay metadata-only.
Release Gate
Run make ci-test in a clean worktree and verify GitHub Actions before converting the PR out of Draft.
| Check | Current | Expected | Status | Artifact |
|---|---|---|---|---|
| World-class ledger ready | evidence-pending | ready_to_claim_world_class == true | blocked | reports/world_class_evidence_ledger.json |
| Claim guard clean | violations 0; ledger ready False | violation_count == 0 and ledger_ready_to_claim_world_class == true | blocked | reports/world_class_claim_guard.json |
| Benchmark public claim ready | public_claim_ready False | public_claim_ready == true | blocked | reports/benchmark_reproducibility.json |
| Review Studio clean | blockers 0; warnings 5 | blocker_count == 0 and warning_count == 0 | blocked | reports/review-studio.json |
| Evidence consistency clean | consistent | decision == consistent and fail_count == 0 | pass | reports/evidence_consistency.json |
Boundary
- Planned work, draft packets, metadata fallback, pending human decisions, and local command runners do not count as completion.
- Valid intake means ready for submission review; ledger review still requires passing source evidence.
- The world-class ledger and claim guard remain the source of truth.