Orbit

Did your AI agent actually finish? Check the commit, the diff and the tests

Chris Rupp

First, check the work yourself

You can verify an AI coding task with Git and your project’s checks. Start with the original request beside you, then follow these steps before accepting the result.

  1. Confirm the commit exists. Ask for the commit hash and run git show --stat <commit>. Check that it contains implementation files. A chat message, an uncommitted edit or a commit that only updates a completion log does not establish delivery. Fetch the remote and run git branch -r --contains <commit> to see which remote branches contain it. A commit on an agent’s branch still needs integration into your intended publication branch.
  2. Read the diff against the request. Identify the task’s base commit, then use git diff <base> <commit> and git diff --name-status <base> <commit>. For a task with several commits, review the whole range. Map each requirement to the code that implements it. Follow changed callers and error paths, rather than stopping at a plausible function name.
  3. Run the tests yourself. Check out the exact revision in a review checkout and use the project’s documented test runner, build and lint commands. Record the commands, exit results and revision. Run an acceptance check for the behavior the user requested, including the failure case. If a check cannot run, record why and leave verification pending.
  4. Check for changes outside the task. Review the file list and full diff for unrelated edits, removed tests, generated files, dependency changes or weakened validation. Inspect git status --short in the agent’s checkout too: a clean commit can still leave required files uncommitted or untracked.

If the agent said tests pass but the code is wrong, turn the original requirement into an observable example. For a duplicate-submission fix, submit the same request twice and check that only one record is created. A passing suite may never have tested that case. Add a regression check that fails on the old behavior and passes on the proposed fix, using your project’s test conventions.

Keep delivery and correctness separate

“The commit exists,” “the change reached the target branch,” “the tests passed” and “the feature works in the deployed app” are different claims. Keep evidence for each one. A local test run does not prove a deployment includes the commit; a successful deployment does not prove the requested behavior works.

For a small manual ledger, record the task, implementation commit, target branch, checks and results, missing scope and any runtime observation still needed. This answers “what did my agents actually ship?” more reliably than searching terminal summaries.

What Orbit automates: the inspector and verdicts from Git

Orbit brings these records into a shared workflow. Its inspector checks the resulting commit diff after completion. The documented validation outcomes include Validated (Auto), Ghost Ship, Wrong Files and Auto Skipped. These labels help draw your attention to missing or unrelated changes; they do not prove the implementation is correct.

A ghost ship is a run that claims success without committing a real change. Orbit’s troubleshooting guide explicitly recommends checking git show <hash> --stat yourself. A completion entry alone is insufficient evidence.

Orbit’s verdicts from Git connect delivery claims to repository evidence. The Pipeline and History reference describes integration and publication status backed by remotely confirmed commits or equivalent applied patches, with publication failures and merge conflicts kept explicit. Review that evidence alongside the agent’s report and test results.

The completion ledger gives the work a permanent record. Use it to return to a task’s commit and notes rather than trusting the final line of a conversation. The queue reference explains how exact verification commands retain bounded evidence, while prose checks that need an operator remain visible for review.

For Sweep review-and-fix work, the Convergence Gate adds a bounded correction loop. It persists the final diff, stop condition and iteration history, then presents a human-merge card. Read why the loop stopped: it can end because the reviewer is clear, but also because progress stalled, findings oscillated or the iteration bound was reached.

Before accepting the task

Require a real implementation diff, evidence that it reached the intended branch, meaningful checks for the requested behavior, and an explicit list of anything still unverified. Orbit collects delivery signals and history; your acceptance criteria determine whether the result solves the problem.

Next, add an approval step before an AI agent merges code, or explore Orbit’s features.