Task Onboarding

1. Minimal extension point

A new task normally does not require changes to LNR, ESTRA, resume, resource management, monitoring, or global merge. Add a task package that declares the artifact and metric contract and provides a system-side evaluator capable of validating one workspace candidate.

tasks/<family>/<task-id>/
├── task.yaml
├── description_lite.md
├── evaluator.py
└── optional problem assets

2. Task manifest contract

id: example-task
category: opt_solver
profile: opt_solver
description: description_lite.md
artifact:
  path: artifacts/best_solution.json
  kind: json_solution
metric:
  name: objective
  lower_is_better: false
  type: benchmark
evaluator:
  backend: task_package
  entrypoint: evaluator.py:evaluate
  timeout_sec: 300
  visible_to_agent: false
gate:
  policy: default

provider is optional. profile selects the prompt/artifact profile, while the evaluator and Gate remain domain-neutral. A task may select a trusted alternative Gate policy, but workspace code cannot register arbitrary policies.

3. Evaluator function

def evaluate(*, artifact_path, workspace_dir, task_dir, dataset_dir, config):
    # Validate schema, instance identity, constraints, and numerical values.
    return {
        "metric": {"name": "objective", "value": 1.23},
        "valid": True,
    }

The entry point must return a JSON object. The task-package backend copies the package to a system-owned runtime directory, verifies its hash, launches it in a subprocess, and normalizes its result. Exceptions, non-zero exit status, timeouts, malformed output, non-finite metrics, and invalid candidates remain explicit evaluator states.

• Reject missing or malformed artifacts with actionable errors.
• Validate array lengths, identifiers, bounds, feasibility constraints, and finite numeric values.
• Compute the metric from the artifact and task data; never trust an agent-reported score.
• Keep lower_is_better aligned with the actual objective.
• Return valid: true only after the complete task contract passes.

4. Current packages

5. Onboarding sequence

1. Place task data outside the source tree and expose it through input_data_dir.
2. Write a concise description_lite.md that names the artifact path and constraints.
3. Add task.yaml with task identity, profile, artifact, metric, evaluator, and optional Gate policy.
4. Implement a deterministic system-side evaluator and test valid, invalid, malformed, missing, and boundary cases.
5. Create a run manifest with budget and CPU/GPU boundaries.
6. Run a short end-to-end task and inspect evaluator events, Gate decisions, Stage rows, snapshots, and final artifacts.