Learn · ~7 min

How you know it worked

'It ran' is not 'it worked.' What this site's tested label means — and what it doesn't.

The site's central argument is this: anyone can schedule an agent. The hard part is trusting the result. This guide makes that argument concrete — and documents, honestly, what this site's own "verified" status means, including what it does not mean.

For agents: - After installing a loop, run the manifest's verification.commands and check verification.required before reporting success. - Never mark your own work verified — that flip belongs to a separate checker or a human. - Badge semantics (verified / draft / stale) are defined by the registry index; read them from /registry/index.json. - A "verified" status carries a date and auto-decays to "stale" after 30 days without a green run. - Prefer deterministic checks (tests, schema, link checks) over a model's self-grade.

Scheduling is easy; trust is the bottleneck

The gap between "it ran" and "it worked" is where loop projects die. Loop builders can usually schedule an agent. They struggle to trust it. The verifier is the product: tests for code loops, source citations for research loops, diff summaries for content loops, schema checks for extraction loops, external monitors for ops loops, and independent reviewers for judgment-heavy tasks. Each loop type has a verification shape, and the manifest declares which one applies.

Deterministic checks first

Tests, schema validation, link checks, and diff gates are cheap and honest. They should be silent on success and verbose on failure. A deterministic check that pings on every green run trains humans to ignore the loop; a deterministic check that speaks only when something breaks earns attention. In the manifest this is verification.commands (REQUIRED) — the executable checks — and verification.required (REQUIRED) — the declarative acceptance criteria (spec). A loop whose verification block is empty has no honest claim to "done."

The maker/checker split

The agent that produced work must not be its only verifier. The self-evaluation trap is agents over-crediting their own output. The mitigation is structural: a separate evaluator, a deterministic check, or a human (anthropic-claude-cookbooks-agent-patterns).

This is why the evaluator–optimizer pattern (/learn/glossary#evaluator-optimizer) carries the risk "self-evaluation is too lenient if the same agent grades." The fix is not a better prompt; it is a different agent, or a deterministic check that the producing agent cannot influence.

Clean-room testing

The strongest doc/template verification is a clean-room run: a fresh environment, an agent given only the published docs, a committed transcript. False passes come from dirty environments — an agent that "verified" the docs by leaning on context it already had, rather than context the docs actually provide.

This is the pattern behind Loopmaster's own agent-usability-test loop: an agent follows a template's install docs in a fresh checkout and reports friction. If the agent succeeds, the docs are agent-ready; if it fails, the failure is evidence against the template, not against the agent.

Verification by loop type

Different loop types verify differently:

  • Code → tests, lint, type checks.
  • Research → source citations + dedupe against prior state.
  • Content → source-backed diffs + a human publish gate.
  • Extraction → schema checks on the output shape.
  • Ops → external monitors (uptime, error rate).
  • Judgment → independent review.

Tooling exists for the eval layer: inspect-ai, promptfoo, and deepeval each cover a slice of model-output evaluation. They are backends to the verification contract, not replacements for it — the manifest declares *what* must pass; the tooling *runs* the check.

What "verified" means on this site

A template on this site is verified only after a clean-room agent run following only its own docs, with the transcript committed. The status carries a date and auto-decays to stale after 30 days without a green run. The status is computed from evidence files; it is never hand-set.

Concretely:

  • verified — a clean-room run, dated, transcript linked, within the freshness window.
  • draft — has not passed a clean-room run; install at your own risk. Draft says draft, loudly.
  • stale — was verified, but the freshness window has lapsed without a green re-run.

This is the propose/publish distinction applied to our own claims: a loop that opens a PR *proposes* a fix; it does not *fix* until merged. A template that has not been clean-room-run *claims* agent-readiness; it does not *have* it until the transcript says so. Loops rot: maintenance is part of the design shows how the maintenance roster keeps these statuses honest over time.

Where to go next

Sources