# Observability Read this when you need to: - debug a failed or slow run; - check who used capacity this month; - find a remote machine to SSH into for live inspection; - correlate Actions hydration with the remote workspace. Crabbox surfaces operational visibility through CLI commands, coordinator usage summaries, retained run history and logs, provider labels, GitHub Actions run links, and Worker logs. The single most useful habit is to keep the **lease ID** (`cbx_...`) and the **run ID** (`run_...`) together — most debugging paths start from one of those two identifiers. See [Identifiers](features/identifiers.md) for the full ID scheme. ## Lease state Inspect a lease with `status`, `list`, and `inspect`: ```sh crabbox status --id swift-crab crabbox list --json crabbox inspect --id swift-crab --json ``` Fields worth checking: - lease ID and slug; - owner and org; - provider and server type; - state (`active`, `released`, `expired`, `failed`); - `createdAt`, `lastTouchedAt`, `idleTimeoutSeconds`, `ttlSeconds`, `expiresAt`; - public address; - SSH user and port; - keep/delete behavior. `status` accepts `--wait` (with `--wait-timeout`, default 5m) to block until the lease reaches a stable state. Provider machines are tagged with Crabbox metadata, so a cloud console can be correlated back to the lease that owns the instance. ## Usage and cost `usage` reports monthly cost and capacity summaries from the coordinator: ```sh crabbox usage crabbox usage --scope user --user alice@example.com crabbox usage --scope org --org example-org crabbox usage --scope all --json ``` Scope is `user` (default), `org`, or `all`; `--month YYYY-MM` selects the reporting month (defaults to the current UTC month). Reports include lease count, active lease count, elapsed runtime, estimated elapsed cost, reserved worst-case cost, and breakdowns by owner, org, provider, and server type. `usage` requires a configured coordinator. See [Cost & usage](features/cost-usage.md) for how the Worker computes rates and enforces budget caps. ## Run history and logs Coordinator-backed `crabbox run` creates a durable run record before leasing begins, appends lifecycle events as the CLI progresses, and finishes the run with exit code, timing, and retained command output. Inspect those records with: ```sh crabbox history crabbox history --lease cbx_... crabbox history --owner alice@example.com --json crabbox logs run_... crabbox events run_... crabbox attach run_... crabbox results run_... ``` The CLI chooses the run ID before admission. If an admission response is lost, it can recover the same record using that ID and the original request. The coordinator accepts recovery only for the same initiating owner, organization, and request; it does not append another `run.started` event. Lease replacement uses the existing acknowledged attribution flow after admission, rather than changing the original create request. Recovery never replays the remote command. The CLI refuses execution without a validated, still-starting run handle, and cancellation stops admission recovery. An older coordinator that lacks the caller-known admission route must be updated; the CLI does not fall back to anonymous record creation. Run IDs are single-invocation identities, and recovery relies on the existing record's retention. A new CLI invocation always uses a new ID. - **`history`** lists recorded runs. Filter with `--lease`, `--owner`, `--org`, `--state`, and `--limit` (default 50). It is intended for command debugging, not unbounded log archival. - **`logs `** prints the retained remote stdout/stderr capture for a run. Use `--tail N` for the last N lines. Logs are stored in 64 KiB chunks with an 8 MiB cap, so very long output is truncated. - **`events `** prints the ordered phase and output events for a run. Filter with `--type`, `--phase`, `--after N`, and `--limit` (default 500). - **`attach `** follows events for a still-active run, polling every `--poll` interval (default 1s), so you can watch a run another CLI started. - **`results `** prints structured test-result summaries. See [Test results](features/test-results.md). The run ID is `run_`; the lease ID is `cbx_<12 hex>`. `history`, `logs`, `events`, `attach`, and `results` all accept the run ID as a positional argument or via `--id`. ### Capturing run output locally By default, coordinator logs and events go to the broker and live progress goes to your terminal. To redirect streams into local files: - `run --capture-stdout ` writes remote stdout to the file and leaves coordinator logs/events plus lifecycle progress on stderr. - `run --capture-stderr ` does the same for remote stderr. Failed runs write a local failure bundle to `.crabbox/captures/*.tar.gz` by default. POSIX SSH-backed runs bundle only the current run's uploaded script file when still available, redacted env/config summaries, timing JSON, command stdout/stderr, common test/report/log paths, and a generic gateway log tail when present. Automatic capture excludes the retained `.crabbox/scripts` store from general report/log discovery: earlier uploads and arbitrary neighbors, including logs and XML reports, are not included. A run without an uploaded script includes no files from that store. Explicit artifact and download selections are independent, including `--download-on-failure` for eligible failures. Native Windows capture stays local-only. Blacksmith delegated runs bundle stdout/stderr plus timing and redacted env/config metadata. The stdout/stderr files captured inside automatic failure bundles are size-capped — pass `--capture-stdout` / `--capture-stderr` when you need a complete local stream file. Remote archive entries are confined to the bundle subtree; unsafe links and special files are omitted. `--capture-on-fail` is still accepted as a compatibility alias; failure bundles are saved automatically on non-zero exit regardless. If the project capture destination is unwritable (permission denied or a read-only filesystem), Crabbox saves the bundle under its per-user state directory instead: `$XDG_STATE_HOME/crabbox/captures/` when `XDG_STATE_HOME` is set, otherwise `/crabbox/state/captures/`. The `failure-bundle local=...` line reports the actual path. These fallback bundles persist outside the project; remove them when no longer needed. Unsafe destinations, including symlinks, and archive/read errors do not trigger this fallback. If both destinations fail, the warning includes both paths and the remote command's exit status is preserved. Explicit `--capture-stdout` and `--capture-stderr` destinations are never moved. The bundle directory stays open and private through file creation, publication, and writing, so a concurrent replacement of its parent path cannot redirect the capture. Failed archive writes remove the incomplete bundle through that same directory. Crabbox does **not** redact captured files. Treat every bundle and capture file as secret-bearing until you have reviewed it. On Unix-like hosts, Crabbox creates local captures, downloads, proofs, and failure bundles with owner-only file permissions (`0600`); new output directories use `0700`. To pull successful-run artifacts back without routing file bytes through the coordinator log, use `--download remote=local` (repeatable): ```sh crabbox run --id swift-crab \ --download dist/report.xml=./report.xml \ -- pnpm test ``` Test results are stored as structured summaries when `--junit`, `--results-auto`, `results.junit`, or `results.auto` is configured. Optional `--fail-on-test-failures` / `results.failOnFailures` policy makes parsed JUnit failures turn an otherwise successful command into a failed run. Successful Blacksmith runs additionally support `--emit-proof `: when requested, Crabbox writes bounded stdout/stderr, timing, metadata, and the generated proof block as local run artifacts. ### Timing and phases `--timing-json` prints final timing as JSON, including sync phases and command phases. Failed runs add `blockedStage` and `retryLikely` when Crabbox can classify the likely blocker; the human-readable run summary prints the same values as `blocked_stage` and `retry_likely`. `runnerTotalMs` is the CLI's observed wall time through route cleanup. `runnerPhases` is a timing-only breakdown whose accepted durations never exceed that total; Crabbox fills any remainder with `unattributed` or a delegated opaque phase. Coordinator phase vectors accept at most one positive integer duration for each of `request`, `network_ready`, `bootstrap`, and `unattributed`. Any malformed vector is discarded as a unit, after which valid legacy startup scalars can supply the breakdown. These fields are unsigned local telemetry: receipt v2 and its signing contract are unchanged. Automatic run cleanup adds `leaseStopped`: true means the release owner confirmed the end of the recoverable lease, even when a terminal receipt remains locally. Retained resources and accepted but pending, failed, retry-scheduled, or otherwise unconfirmed cleanup report false and preserve failure recovery guidance. False does not certify a running or reachable resource. `leaseStopError` reports cleanup errors separately and may be present even after confirmed removal, for example when local finalization fails. Run finalization emits timing after cleanup and the failure digest. Its terminal order is timing record, timing JSON, local receipt persistence, then coordinator finish. Timing sink failures are terminal and are reflected in the local receipt and process exit; a failing CLI invocation can append its normal exit diagnostic. Timing `artifacts` lists only files already committed when the timing payload is emitted. Terminal receipt metadata is intentionally excluded because persistence happens afterward; successful persistence prints a separate `artifact kind=receipt path=... bytes=...` confirmation. Commands can define their own phases by printing marker lines to stdout or stderr: ```sh echo CRABBOX_PHASE:install pnpm install --frozen-lockfile echo CRABBOX_PHASE:build pnpm build echo CRABBOX_PHASE:test pnpm test ``` ### Local benchmark ledger `--timing-record=default` and `--timing-record ` append the final `TimingReport` to a local JSONL benchmark store. The default store is `/timings.jsonl`, which uses `$XDG_STATE_HOME/crabbox` when available and otherwise falls back to the user config directory's `crabbox/state` directory. The ledger is explicit local state. Plain `crabbox run` does not write it. Recorded rows can include repo paths, remote workdirs, command display text, labels, artifact paths, and lease metadata because they preserve the timing payload. Use [`crabbox bench report`](commands/bench.md) to aggregate local observations, and treat insufficient sample counts as a prompt to collect more local evidence rather than as a provider ranking. Timing rows, receipts, captures, and failure bundles are sensitive local correlation artifacts. They may contain repository and filesystem paths, workdirs, labels, artifact paths, lease IDs, and run IDs. Keep them private and review them before sharing. The benchmark ledger records observed timing; it is not a deterministic budget gate. The future deterministic metric contract lives in [Deterministic perf evidence](features/deterministic-perf-evidence.md) and intentionally separates reproducible fuel/instruction-style counters from noisy wall-clock timings. ### Forwarding live secrets When local `CRABBOX_ENV_ALLOW` is set, `run` prints the variable names selected for forwarding plus safe metadata (whether secret-looking names are set and their value length). Values are never printed. Delegated Testbox providers report that forwarding is unsupported, because secrets belong in the provider workflow. For one-off live secrets, avoid hand-written `source` boilerplate: ```sh crabbox run \ --env-from-profile ~/.project-live.profile \ --allow-env API_TOKEN \ --preflight \ -- ./scripts/live-smoke.sh ``` Crabbox parses simple `export NAME=value` and `NAME=value` profile lines without executing the profile. Only names selected by `--allow-env`, `env.allow`, or `CRABBOX_ENV_ALLOW` are forwarded, and summaries show presence/length metadata for secret-looking names rather than values. On native Windows, the profile handoff file is uploaded as UTF-8 and imported with PowerShell UTF-8 decoding, so non-ASCII token material and paths do not depend on the system code page. See [Env forwarding](features/env-forwarding.md). ### Running scripts instead of argv Use `--script ` or `--script-stdin` when the remote command is more than a small argv: ```sh crabbox run \ --script ./scripts/provider-smoke.sh \ --env-from-profile ~/.project-live.profile \ --allow-env API_TOKEN \ --timing-json ``` The script is uploaded under `.crabbox/scripts/` in the remote workdir; only that run's uploaded file is selected for automatic failure bundles, not its directory or earlier uploads. POSIX SSH providers support this path; delegated providers reject it before reading stdin because they own command transport. Native Windows targets upload scripts too and run them through Windows PowerShell — use `--shell` for short snippets and `--script ` for longer runs. Crabbox writes Windows scripts as UTF-8 with a byte-order mark when the input has none, which keeps Windows PowerShell 5.1 from mojibaking non-ASCII script source: ```sh crabbox run \ --provider ssh \ --target windows \ --windows-mode normal \ --static-host win-dev.local \ --preflight \ --script ./scripts/windows-smoke.ps1 \ -- -Mode smoke ``` For PR debugging that should not inherit local dependency churn, run against a fresh remote checkout: ```sh crabbox run \ --fresh-pr example-org/my-app#123 \ --script ./scripts/e2e-smoke.sh ``` Add `--apply-local-patch` only when the local diff should be applied on top of the PR checkout. PR URLs must be on `github.com`; non-GitHub and GitHub Enterprise PR URLs are rejected rather than rewritten to a public clone. The numeric shorthand uses the current repository's GitHub origin. ## Remote debugging SSH in for live process and filesystem inspection: ```sh crabbox ssh --id swift-crab crabbox inspect --id swift-crab --json ``` Useful remote checks: ```sh crabbox-ready test -f /var/lib/crabbox/bootstrapped df -h free -h ps aux --sort=-%cpu | head ``` A lease created with `--keep` stays reachable over SSH until `crabbox stop`, idle expiry, or the TTL cap removes it. For one-shot E2E debugging, add `--keep-on-failure`: Crabbox releases successful runs normally, but on failure it prints the inspect, SSH, and stop commands for the exact failed lease and leaves idle/TTL expiry to clean it up later. ### Preflight capability snapshot For a concise pre-command capability snapshot, add `--preflight`: ```sh crabbox run --id swift-crab --preflight -- pnpm test:changed ``` Preflight prints a target-specific capability snapshot from the same workdir the command will run in. It sources the Actions handoff env file when present and marks the workspace as raw or Actions-hydrated. A raw workspace that has Actions hydration configured prints the exact hydrate command suggestion and whether the selected provider/target supports hydration. Preflight is a probe layer, not an installer. Missing tools print `tool=missing`; Crabbox does not run `apt install`, `corepack prepare`, `bun install`, or host toolchain setup. Install toolchains through Actions hydration, a prebaked image, a devcontainer/Nix/mise/asdf setup, or the uploaded script/command itself. For `npm`, `pnpm`, and `yarn`, probe-local Corepack controls disable networking, latest-version lookup, automatic project pinning, and download prompts. Cached project versions remain selectable; unavailable versions must be hydrated separately. The workload keeps its original environment. This is not a universal network sandbox or a guarantee that external tools never touch cache files. The pnpm probe additionally uses `PNPM_CONFIG_PM_ON_FAIL=ignore` to suppress secondary version/environment-lockfile management on versions supporting that setting. Corepack's project-version selection is retained; the workload keeps its original pnpm policy. Standalone legacy wrappers may have other behavior. The built-in probes cover common toolchains and target-specific capabilities. Use [`crabbox preflight-tools [--json]`](commands/preflight-tools.md) for the complete names, selector aliases, default membership, and target support in the installed binary. This inspection is offline and executes no probes. Override the probe list per run: ```sh crabbox run --preflight --preflight-tools python,python3 -- python3 -m pytest crabbox run --preflight --preflight-tools default,cmake -- cmake --build build crabbox run --preflight --preflight-tools default,python3-venv -- python3 -m pytest crabbox run --preflight --preflight-tools raw_socket -- ./packet-tests ``` The opt-in CMake probe invokes the literal `cmake --version` command on POSIX, WSL2, and native Windows targets. It reports only the first output line or `cmake=missing`; the result is diagnostic only, so missing CMake does not block the workload or trigger installation or upgrades. The opt-in `python3-venv` probe checks a real disposable environment on Linux, macOS and WSL2, not merely the interpreter version or whether `venv` imports. It reports `remote preflight python3-venv= cleanup=`. `ready` means the fresh environment's Python and pip succeeded, the probe owner confirmed process quiescence and scratch removal, and the caller confirmed exact transport-stage retirement. Capability results distinguish `missing-python3`, `venv-unavailable` and `pip-unavailable`; other states are `worker-failed`, `timed-out`, `canceled` and `unavailable`. `unavailable` means no reliable capability result. Cleanup is independent: an operational transport, setup or envelope error may remain after confirmed cleanup, yielding `unavailable cleanup=confirmed`. Unknown quiescence, scratch removal or stage retirement instead reports `cleanup=unconfirmed`; uncertainty is never successful cleanup. Capability absence, broken capability, worker failure or probe timeout does not block the workload when cleanup is confirmed. Operational transport, setup, envelope, ownership or cleanup errors prevent the later workload regardless of the cleanup flag. `unavailable cleanup=unconfirmed` also fails the run rather than continuing with unresolved ownership. Caller cancellation prevents continuation. The 90-second probe allowance and independent 30-second cleanup reserve exclude separately bounded transport setup, so they are not a total-command deadline. No host packages are installed or upgraded, and no project environment is activated or reused; pip is seeded only inside the disposable environment. See [run preflight](commands/run.md#preflight) for selection and target support. `raw_socket` reports `direct` when the execution user can open and immediately close `socket(AF_INET, SOCK_RAW, IPPROTO_RAW)`, `sudo` when only the same bounded probe succeeds through `sudo -n`, `unavailable` otherwise, and `probe_missing` when neither `python3` nor `python` exists. It never sends a packet or elevates the workload. This kernel capability is distinct from Python, Scapy, tcpdump, libpcap, or packet-capture-tool availability, and unsupported targets skip the probe. Or per repository: ```yaml run: preflightTools: - default - cmake ``` ## Actions hydration `crabbox actions hydrate` populates a lease's workspace by driving the repo's configured workflow setup over SSH by default, waiting for a ready marker. The marker path is the key local correlation point; `--github-runner` instead registers the box as a self-hosted runner and reports the workflow run URL. ```sh crabbox actions hydrate --id swift-crab --workflow .github/workflows/hydrate.yml crabbox inspect --id swift-crab --json ``` A hydrated run writes non-secret handoff data for later `crabbox run --id swift-crab` commands. Secrets and OIDC tokens stay workflow-step scoped unless the workflow intentionally writes its own short-lived handoff. See [Actions hydration](features/actions-hydration.md). ## Live provider debugging For live provider or end-to-end test runs, prefer an Actions-hydrated lease when tests need Node, pnpm, Docker services, repository secrets, or GitHub OIDC: ```sh crabbox warmup --provider aws --class beast --keep crabbox actions hydrate --id swift-crab --workflow .github/workflows/hydrate.yml mkdir -p .crabbox/logs CRABBOX_ENV_ALLOW=OPENAI_API_KEY,OPENAI_BASE_URL \ crabbox run --id swift-crab \ --preflight \ --timing-json \ --capture-stdout .crabbox/logs/live-provider.stdout.log \ --capture-stderr .crabbox/logs/live-provider.stderr.log \ --keep-on-failure \ --shell 'echo CRABBOX_PHASE:install; pnpm install --frozen-lockfile; echo CRABBOX_PHASE:test; pnpm test:live' ``` For Blacksmith Testbox comparison runs, keep secrets in the Testbox workflow environment. Crabbox reports that `CRABBOX_ENV_ALLOW` forwarding is unsupported, because Blacksmith owns command execution: ```sh CRABBOX_ENV_ALLOW=OPENAI_API_KEY \ crabbox run --provider blacksmith-testbox \ --blacksmith-workflow .github/workflows/ci-check-testbox.yml \ --blacksmith-job test \ --preflight \ -- pnpm test:live:providers ``` ## Worker logs When the coordinator path fails before SSH is established, check Worker logs and Durable Object errors. The symptoms usually fall into a few groups: - auth failure; - cost limit rejection; - provider quota or capacity rejection; - provider API failure; - Durable Object alarm or state-transition bug. Keep the lease ID, owner, org, provider, class, and request time on hand when comparing CLI output to Worker logs. See [How it works](how-it-works.md) and the [coordinator](features/coordinator.md) reference for the request flow. ## Gaps Crabbox observability is sufficient for maintainer operations but is not yet a full analytics product. Notably missing: - alerting on budget or failure-rate thresholds; - a dashboard UI.