Agent Usability Test Loop

The agent-usability-test loop verifies that another agent can follow Loopmaster template or site docs from a clean environment without relying on hidden project context. It produces a transcript, a friction report, and a rubric-backed pass/fail decision.

This directory is a draft template. "Draft" is registry/site operational metadata for the template directory, not a hidden loop.yaml field; the manifest itself uses only the fields defined by spec/loop-spec-v0.1.md.

Files

• loop.yaml — agent-readable manifest for the template.
• AGENT-INSTALL.md — exact setup and execution guide for cron, GitHub Actions, and agent-kanban backends.
• verification/clean-room-protocol.md — normative protocol used by LM-VER-01 and LM-VER-02.
• verification/transcript-template.md — transcript format every run must fill in.
• verification/pass-fail-rubric.md — scoring rubric and pass/fail thresholds.
• verification/check_clean_room_packet.py — local completeness check for the protocol packet.
• verification/dry-run-checklist.md — dry-read verification for this template revision.

The seven human questions

1. What loop do I need?

Use this loop when you want evidence that a template, repo entry point, or install guide is actually agent-usable. It matches the agent usability loop in docs/research/loop-landscape.md §2 and the LM-TPL-04 requirement in the F2 plan: fresh checkout/container, agent receives only template/site docs, transcript format, and pass/fail rubric.

LM-VER-01 uses this protocol for the repo surface: an agent gets the repo URL and expected entry points only, then tries to reach and run a chosen template. LM-VER-02 uses the same protocol for the public site/agent entry points once those artifacts exist.

2. What can it safely automate?

The manifest permits creating a clean checkout, reading explicitly listed docs, running documented commands, writing a transcript, writing a friction report, and creating follow-up tasks. It forbids unlisted project context, private credentials, source-repo mutation without approval, unevidenced verified claims, and public verified-badge flips without approval. These claims map to permissions.allowed_actions, permissions.forbidden_actions, and human_gates.required_for in loop.yaml.

3. What do I need to connect?

Minimum setup needs a target repo or site URL, a disposable clean working directory or container, the exact docs packet the test agent may read, Python 3 for the packet completeness check, and a place to write transcript/report artifacts. No credential is required for a read-only public repo or local checkout. If the target is private, provide only read-scoped repo access through the backend's secret store; never place secret values in the prompt, transcript, or template files.

4. What can go wrong?

The environment may not be clean, the test agent may receive hidden context, docs may omit commands or expected outputs, the manifest may promise capabilities the prose does not explain, or the reviewer may over-credit a partial run. The clean-room protocol requires an allowlist of docs, environment proof, full command transcript, and a rubric that distinguishes pass, conditional pass, and fail.

5. How do I know it worked?

A successful run has two durable artifacts: a filled transcript and a friction report. The report must score every rubric section, cite transcript evidence for each score, and list zero unresolved blockers for a pass. The local verification command python3 templates/agent-usability-test/verification/check_clean_room_packet.py confirms the protocol packet itself is present and internally checkable, but it is not a substitute for a real clean-room run.

6. How do I stop or roll it back?

For cron, remove the scheduled packet-check or task-creation command. For GitHub Actions, disable or delete the usability-test workflow. For agent-kanban, pause or remove the recurring reviewer task. Rollback means: do not mark the target verified, do not flip a public badge, keep the failed transcript/report as evidence, and create follow-up work for every unresolved friction point.

7. What should I improve after the first run?

Improve the docs where the transcript shows hesitation, missing prerequisites, ambiguous commands, unsafe assumptions, or prose↔manifest drift. Then rerun from a fresh environment; do not reuse a successful transcript after changing the docs. Review this protocol weekly because agent harnesses, tooling, and site entry points change quickly.

Clean-room protocol summary

A conforming run follows verification/clean-room-protocol.md:

1. Start from a fresh checkout, disposable container, or equivalent clean working directory.
2. Record environment proof before reading target docs.
3. Give the test agent only the prompt and allowlisted docs packet.
4. Capture the transcript using verification/transcript-template.md.
5. Score the result with verification/pass-fail-rubric.md.
6. Create a task or PR comment for each unresolved friction point.
7. Require human approval before the first public verified status flip.

Backend behavior

• cron can run the packet completeness check and create/remind a reviewer task. It must not self-certify a template.
• github_actions can create a clean checkout/container, run deterministic checks, and archive transcript/report artifacts when paired with an agent runner. Badge/status flips remain gated.
• agent_kanban is the recommended backend for LM-VER-01 and LM-VER-02 because it preserves maker/checker split and stores the reviewer handoff in the task board.

Agent install guide — Agent Usability Test Loop

Run commands from the repository root unless a step says otherwise. This template defines a verification protocol; the deterministic local command checks the protocol packet, while a real pass/fail decision requires an actual clean-room run and transcript.

0. Preflight

python3 --version
uv run --with check-jsonschema --with pyyaml check-jsonschema \
  --schemafile spec/loop.schema.json \
  templates/agent-usability-test/loop.yaml
python3 templates/agent-usability-test/verification/check_clean_room_packet.py

Expected output:

Python 3.x.x
ok -- validation done
CLEAN_ROOM_PACKET_OK

Failure branches:

• python3: command not found — install Python 3 or use a backend image that includes it.
• No such file or directory — confirm you are at repo root and templates/agent-usability-test/ exists.
• schema validation failure — fix loop.yaml before scheduling any usability run.
• CLEAN_ROOM_PACKET_FAILED — restore or fix the missing protocol, transcript, rubric, README, install guide, or dry-run checklist before using this template.

1. Define the target and allowed packet

Create a run-specific packet before starting the test agent. Example for a template verification:

Target: templates/research-watch at commit <SHA>
Environment: fresh checkout of <repo> into /tmp/loopmaster-usability/repo
Allowed docs:
- README.md
- spec/loop-spec-v0.1.md
- templates/research-watch/loop.yaml
- templates/research-watch/README.md
- templates/research-watch/AGENT-INSTALL.md
- templates/agent-usability-test/verification/clean-room-protocol.md
- templates/agent-usability-test/verification/transcript-template.md
- templates/agent-usability-test/verification/pass-fail-rubric.md
Forbidden context:
- prior kanban handoffs
- implementation chat logs
- private credentials or unlisted local files

Expected result: the packet is explicit enough that a reviewer can tell whether every file read by the test agent was allowed.

Failure branches:

• target commit is unknown — stop and record a failed setup; do not test a moving target.
• allowed docs are vague, such as "all docs" — rewrite the packet as exact paths/URLs.
• credentials are needed — record their scope and backend storage, but never expose secret values to the prompt or transcript.

2. Create a clean environment

For a fresh checkout:

TARGET_REPO="https://github.com/OWNER/REPO.git"
TARGET_SHA="<commit-sha>"
WORKDIR="/tmp/loopmaster-usability-$TARGET_SHA"
rm -rf "$WORKDIR"
mkdir -p "$WORKDIR"
git clone "$TARGET_REPO" "$WORKDIR/repo"
cd "$WORKDIR/repo"
git checkout "$TARGET_SHA"
git status --short

Expected output: git status --short prints nothing.

For a disposable container, use an image that matches the target docs and record the image name before starting the test agent. Install only prerequisites named by the docs.

Failure branches:

• checkout cannot reach the target commit — stop; the target version is not reproducible.
• git status --short is not clean before the test — reset or recreate the environment.
• local env files or prior state are needed to pass — count this as friction unless the docs explicitly tell the agent how to create them safely.

3. Run the test agent

Give the test agent only the standard prompt from verification/clean-room-protocol.md plus the run-specific allowed packet. The test agent should fill verification/transcript-template.md and then write a friction report using verification/pass-fail-rubric.md.

Required transcript destination for template runs:

templates/<template-id>/verification/YYYY-MM-DD-agent-usability-transcript.md
templates/<template-id>/verification/YYYY-MM-DD-agent-usability-report.md

Expected result: transcript records every file read, command run, exit code, output summary, ambiguity, and final answer.

Failure branches:

• agent asks for context outside the packet — do not provide it; record hidden-context friction.
• command would print a secret — stop or redact before recording output; record the redaction.
• agent mutates the source repo outside documented install steps — stop and mark fail unless prior approval explicitly allowed it.

4. Backend A — cron

Cron is useful for recurring reminders and packet checks, not for autonomous certification.

Install a weekly packet check that opens or notifies a reviewer task in your own scheduler/task system:

REPO_ROOT=$(pwd)
CRON_LINE="0 10 * * 1 cd $REPO_ROOT && /usr/bin/env python3 templates/agent-usability-test/verification/check_clean_room_packet.py >> .loopmaster/state/agent-usability-test.cron.log 2>&1"
( crontab -l 2>/dev/null | grep -v 'agent-usability-test/verification/check_clean_room_packet.py' ; printf '%s\n' "$CRON_LINE" ) | crontab -
crontab -l | grep 'agent-usability-test/verification/check_clean_room_packet.py'

Expected output: one crontab line containing check_clean_room_packet.py.

Failure branches:

• crontab: command not found — this backend is unavailable; use GitHub Actions or agent-kanban.
• cron check fails — inspect .loopmaster/state/agent-usability-test.cron.log; do not start usability runs until the packet is fixed.
• cron attempts to mark a target verified — remove that automation; verified status requires transcript evidence and the human gate declared in loop.yaml.

Stop cron:

crontab -l | grep -v 'agent-usability-test/verification/check_clean_room_packet.py' | crontab -

5. Backend B — GitHub Actions

GitHub Actions can validate the protocol packet and archive clean-room artifacts. It can also run a deterministic target dry-run when the target docs use only normal CLI commands. It must not flip verified status by itself.

Example workflow body:

name: Agent usability packet check
on:
  workflow_dispatch:
  pull_request:
    paths:
      - 'templates/agent-usability-test/**'
      - 'templates/**/README.md'
      - 'templates/**/AGENT-INSTALL.md'
      - 'templates/**/loop.yaml'
jobs:
  packet:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv run --with check-jsonschema --with pyyaml check-jsonschema --schemafile spec/loop.schema.json templates/agent-usability-test/loop.yaml
      - run: python3 templates/agent-usability-test/verification/check_clean_room_packet.py

Expected result: packet check passes before usability artifacts are trusted.

Failure branches:

• workflow lacks the target docs in paths — update triggers before relying on it.
• packet passes but no transcript/report is produced — the target has not been agent-usability verified; only the protocol packet was checked.
• artifact contains secrets or private environment details — delete/redact the artifact before publication and route for human review.

6. Backend C — agent-kanban

Agent-kanban is the recommended backend for LM-VER-01 and LM-VER-02 because it creates a durable reviewer task and keeps maker/checker split.

Create a reviewer task with a self-contained body like this:

Run the Loopmaster agent-usability-test protocol against <target> at <commit-or-site-version>. Use only the allowed input packet below. Start from a fresh checkout/container, record environment proof, give the test agent the standard prompt from templates/agent-usability-test/verification/clean-room-protocol.md, capture the transcript using transcript-template.md, score with pass-fail-rubric.md, and create one follow-up task or PR comment for every unresolved friction point. Do not flip any public verified status; recommend only.

Allowed input packet:
<exact path/URL list>

Using Hermes native task creation from an agent session:

kanban_create(title="Clean-room usability run for <target>", assignee="reviewer", body="<self-contained body above>", workspace_kind="worktree", workspace_path="/absolute/path/to/repo")

Expected result: the reviewer task completes with transcript path, report path, pass/fail decision, scores, and follow-up ids/comments.

Failure branches:

• reviewer profile is unavailable — block and ask an operator to choose a real reviewer profile; do not assign maker to self-certify.
• task body omits the allowed packet — block before running; hidden context would invalidate the result.
• report recommends first public verified badge flip — route to human approval before changing registry/site status.

7. Final verification checklist

Before calling this template installed or ready for use, record:

uv run --with check-jsonschema --with pyyaml check-jsonschema \
  --schemafile spec/loop.schema.json \
  templates/agent-usability-test/loop.yaml
python3 templates/agent-usability-test/verification/check_clean_room_packet.py

Expected success:

ok -- validation done
CLEAN_ROOM_PACKET_OK

Then dry-read the clean-room procedure:

• Can a reviewer identify the exact target version before running?
• Is the allowed packet exact, not implied?
• Does the protocol forbid hidden context and private credentials?
• Does the transcript template capture commands, exit codes, and output summaries?
• Does the rubric force fail on missing commands, unsafe credentials, and manifest/prose drift?
• Does every backend state that public verified status remains human-gated?

If any answer is no, the install is not verified.