Benchmark Mode Issues

Benchmark Mode failures fall into two categories: API-mode jobs that fail at the provider call level, and agentic jobs that fail inside a worktree spawn. The root cause is usually visible in the per-run result row.

API-mode jobs fail immediately

API-mode benchmark rows call provider APIs directly. If a row shows a failure straight away, check:

Provider credentials. The model row is using a key stored in the Vault. Run orbit_vault_list or open Settings → Vault to confirm the key exists and has the right name for that provider.
Model not available. Some model IDs are experimental or require specific API access. Confirm the model ID is valid for your account with the provider before adding the row.

Agentic job rows fail with a known failure class

Each agentic run row in the Results panel displays a failure class when the run doesn’t complete cleanly. These map directly to the outcome codes used internally:

Failure class	What it means	How to fix
`runner_missing`	The Claude or Codex CLI was not found on this machine when the worktree spawn ran.	Install the CLI and confirm it’s on `PATH`. The agent row will show: “The [runner] CLI could not be found on this machine, so the agent never started.”
`auth_missing`	The CLI binary was found but is not signed in.	Sign into the CLI (`claude` or `codex` login command) and re-run the job.
`nonzero_exit`	The agent process exited with a non-zero code before finishing.	Check the transcript attached to the run row for the last output before exit.
`tool_loop_exhaustion`	The agent exhausted its maximum tool-call turns without finishing the task.	Simplify the prompt or split it into smaller benchmark jobs.
`timeout`	The run exceeded the per-run time limit before completing.	Reduce scope or increase the timeout if the job is legitimately long-running.
`killed`	The job was cancelled and the worktree was removed.	This is expected on manual cancellation. Re-queue if needed.

Agentic harness errors stop the whole job

Three specific conditions cause the harness to reject a run immediately rather than mark it failed:

COMPLETION_LOG.md changed during benchmark spawn harness run
COMPLETION_LOG.archive.md changed during benchmark spawn harness run
ORCHESTRATOR.md changed during benchmark spawn harness run

These errors mean the agent modified a file it is explicitly forbidden from touching during a benchmark run. The protection exists because benchmark runs are isolated — changes to these shared files would corrupt your main project state. Adjust the agent’s prompt or file-lock these files before rerunning.

Apply Winner does nothing after selecting a run

After clicking Apply Winner, Orbit presents the winning run’s diff for review before writing anything to your project. If you don’t see the diff:

Confirm the run shows a terminal outcome (done / CLEAN_COMPLETION) and not a partial or failed outcome. Apply Winner is only available on runs that completed cleanly.
If the run had no net diff (the agent ran but made no changes), Apply Winner reports nothing to apply. This is expected.

Context mode produces unexpected results

Agentic benchmark runs support two context modes:

Workspace context — the agent sees your current project working tree and CLAUDE.md, as it would in a normal queue run.
Clean context — the agent starts with no project-specific context.

If a Clean-context run produces output that looks too project-specific, verify that ORBIT_CONTEXT_MODE was correctly set to clean in the spawn environment — visible in the run’s configuration metadata. If a Workspace-context run produces output that looks surprisingly generic, check that the project’s CLAUDE.md is complete and reachable from the worktree path.

Memtrace on vs. off comparisons

Each run records whether Memtrace was enabled or disabled. If you don’t see a difference between the two when comparing runs, confirm that the Memtrace index is current (visible in Memtrace panel → last indexed time). A stale or empty index produces the same output as a disabled index.