Evidence Collection

World-Class Evidence Preflight

This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.

Decisioncollection-preflight-blocked

Pending4

Ready1

Blocked3

Source11/14

Repairs7/7

Phases2/2

Submission Kit

Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied. Artifact prefill is convenience data only.

submissions directory: evidence/world_class/submissions
drafts count as evidence: false
artifact prefill counts as evidence: false
preflight accepts evidence: false

Submission refs

submission-ref

6/7 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

31/33 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

submission-ref rows are the paths expected in artifact_refs; supporting-evidence rows stay available for audit context.

Phase Queue

Repair rows are grouped into execution phases so operators can clear the first blocker before moving to later source collection. Phase queue rows remain procedural and never count as accepted evidence.

#20 · blocked

Unblock access

Phase: unblock-access
Rows: 4 / 4 blocked
Owners: Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator
Evidence: human-adjudication, native-client-telemetry, native-permission-enforcement, provider-holdout
Next: Assign three independent controlled reviewer identities before claiming human adjudication.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Counts: does not count as completion

#40 · blocked

Collect source

Phase: collect-source
Rows: 3 / 3 blocked
Owners: Browser/Chrome/IDE/provider client integrator, target client or installer integrator
Evidence: native-client-telemetry, native-permission-enforcement
Next: Telemetry must include adoption outcome evidence.
Verify: python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --self && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Counts: does not count as completion

Evidence Queue

external · blocked

Provider Holdout

Evidence: provider-holdout
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/provider-holdout.json

Next Action

Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.

python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --self

python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts --self

Artifact Roles

Submission refs

submission-ref

1/1 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

7/8 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Output eval cases

Current: present
Status: pass
Action: Keep output holdout cases available before provider execution.

file Provider runner

Current: present
Status: pass
Action: Use the provider runner instead of the local command runner for model-backed evidence.

env Provider credential

Current: not-set
Status: missing
Action: Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.

Source Checks

call_count Provider calls

Current: 40
Expected: ==40
Status: pass
Action: Complete all 40 fixed DeepSeek calls.

model_executed_count Provider model runs

Current: 40
Expected: ==40
Status: pass
Action: Require model identity on all 40 calls.

failure_count Provider failures

Current: 0
Expected: ==0
Status: pass
Action: Resolve every fixed-matrix failure.

total_tokens Token budget

Current: 40938
Expected: <=250000
Status: pass
Action: Keep the matrix within the 250000-token ceiling.

Repair Rows

precheck provider-api-key

Priority: 20
Phase: unblock-access
Owner: operator with provider credentials
Status: blocked
Reason: Required env precheck is missing.
Action: Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

Runbook

Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.
python3 scripts/yao.py evidence-build . --run-id <PROVIDER_RUN_ID> --self
Keep the generated private answer key and role-neutral review materials inside .yao/runs/<PROVIDER_RUN_ID>.
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

human · ready-for-human-review

Human Adjudication

Evidence: human-adjudication
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/human-adjudication.json

Next Action

Assign three independent controlled reviewer identities before claiming human adjudication.

python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --self

python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts --self

Artifact Roles

Submission refs

submission-ref

1/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

7/8 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Blind review kit

Current: present
Status: pass
Action: Use the provider run's role-neutral pack and finalizer for three controlled reviews.

file Decision template

Current: present
Status: pass
Action: Collect three exact 20-pair reviewer packets with integrity and independent-review attestations.

file Decision importer

Current: present
Status: pass
Action: Keep the fixed reviewer count and promotion thresholds unchanged.

human Human reviewer

Current: external-human-action
Status: human-required
Action: Assign three independent controlled reviewer identities before claiming human adjudication.

Source Checks

reviewer_count Registered reviewers

Current: 3
Expected: ==3
Status: pass
Action: Collect reviewer-a, reviewer-b, and reviewer-c.

pair_count Blind pairs

Current: 20
Expected: ==20
Status: pass
Action: Complete all 20 blind pairs.

failure_count Review failures

Current: 0
Expected: ==0
Status: pass
Action: Resolve packet, identity, or adjudication failures.

blind_pack_bound Blind pack binding

Current: True
Expected: true
Status: pass
Action: Bind adjudication to the reviewed blind pack SHA256.

Repair Rows

precheck human-reviewer

Priority: 20
Phase: unblock-access
Owner: human reviewer
Status: blocked
Reason: Required human precheck is human-required.
Action: Assign three independent controlled reviewer identities before claiming human adjudication.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

Runbook

Give each registered reviewer an independent copy of the matching provider_review_reviewer-*.json template and the role-neutral blind pack.
Collect all 20 A/B choices, reasons, controlled submission ids, timestamps, and truthful independent-review attestations.
Export an access-controlled reviewer registry that binds each reviewer id to the exact packet SHA256.
python3 scripts/yao.py evidence-finalize-review . --source-run <PROVIDER_RUN_ID> --decisions <reviewer-a.json> --decisions <reviewer-b.json> --decisions <reviewer-c.json> --reviewer-registry <registry.json> --run-id <FINAL_RUN_ID> --self
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

external · blocked

Native Permission Enforcement

Evidence: native-permission-enforcement
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/native-permission-enforcement.json

Next Action

Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --self

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts --self

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

11/11 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Permission policy

Current: present
Status: pass
Action: Keep approved high-permission capabilities explicit.

file Runtime probes

Current: present
Status: pass
Action: Refresh runtime permission probes after packaging changes.

external Native guard

Current: external-integration-required
Status: external-required
Action: Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

Source Checks

native_enforcement_count Native enforcement

Current: 0
Expected: >0
Status: blocked
Action: Collect real target-client or external runtime guard proof.

failure_count Probe failures

Current: 0
Expected: ==0
Status: pass
Action: Runtime permission probes must stay clean.

installer_enforcement_ready Installer support

Current: True
Expected: true
Status: pass
Action: Installer enforcement is supporting evidence, not native proof.

Repair Rows

precheck native-guard

Priority: 20
Phase: unblock-access
Owner: target client or installer integrator
Status: blocked
Reason: Required external precheck is external-required.
Action: Attach a real target-client or external installer runtime guard; metadata fallback is not enough.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

source-check native_enforcement_count

Priority: 40
Phase: collect-source
Owner: target client or installer integrator
Status: blocked
Reason: Current value 0 does not satisfy '>0'.
Action: Collect real target-client or external runtime guard proof.
Verify: python3 scripts/yao.py runtime-permissions . --package-dir dist --self && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

Runbook

Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
Update the generated target adapter only when the guard is actually enforced by that target.
python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip --self
python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation --self
python3 scripts/yao.py runtime-permissions . --package-dir dist --self
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

external · blocked

Native Client Telemetry

Evidence: native-client-telemetry
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/native-client-telemetry.json

Next Action

Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --self

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts --self

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Native telemetry host

Current: present
Status: pass
Action: Use the native host to receive metadata-only client events.

file Hook recipes

Current: present
Status: pass
Action: Refresh telemetry hook recipes before external client installation.

external External client

Current: external-integration-required
Status: external-required
Action: Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

Source Checks

external_source_events External events

Current: 0
Expected: >0
Status: blocked
Action: Import at least one metadata-only event from a real client.

adoption_sample_count Adoption sample

Current: 0
Expected: >0
Status: blocked
Action: Telemetry must include adoption outcome evidence.

raw_content_allowed Raw content blocked

Current: False
Expected: false
Status: pass
Action: Telemetry must stay metadata-only.

Repair Rows

precheck external-client

Priority: 20
Phase: unblock-access
Owner: Browser/Chrome/IDE/provider client integrator
Status: blocked
Reason: Required external precheck is external-required.
Action: Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

source-check adoption_sample_count

Priority: 40
Phase: collect-source
Owner: Browser/Chrome/IDE/provider client integrator
Status: blocked
Reason: Current value 0 does not satisfy '>0'.
Action: Telemetry must include adoption outcome evidence.
Verify: python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --self && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

source-check external_source_events

Priority: 40
Phase: collect-source
Owner: Browser/Chrome/IDE/provider client integrator
Status: blocked
Reason: Current value 0 does not satisfy '>0'.
Action: Import at least one metadata-only event from a real client.
Verify: python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --self && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions --self
Evidence: does not count as completion

Runbook

python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --self
python3 scripts/yao.py skill-atlas --workspace-root . --self
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

Safety Boundary

Environment variables are displayed only as set or not-set; secret values are never printed.
Human-required and external-required states are operator work, not accepted evidence.
The world-class ledger remains the only source of truth for ready_to_claim_world_class.