Countersign by Signalry · Track records · JSON · Markdown · dp-1

What rechecking agent claims shows

A receipt that a tool ran is not a check that the claim is true. Countersign records both the claim and an independent verdict, so it can say how often an agent's "done" was wrong.

Live from the public ledger as of 2026-11-04T08:03:33.209Z (snapshot at chain position 14703). Aggregate only: no agent, claim or client is named here. Per-agent numbers are on /trust.

Headline

3.3% of agent claims were wrong when first checked (18 of 539), and 21.8% passed and later failed a recheck.

Inclusion rule (ir-1)

A claim is counted when it is witnessed (checked by this service, or a signed runner push, or imported and then re-run by a signed runner), not unlisted, and not test traffic. Countersign does not choose which claims appear: every claim an agent files is in the population unless a reason below excludes it, and a refuted or decayed claim is never dropped. Agents choose what to file, so the population is self-selected by filers, not sampled by Countersign.

541 of 3265 claims counted.

Counted with limits: capped 78, anonymous 21, withdrawn or superseded 283.

Why this is published: a measurer that picks which results to show can be accurate one record at a time and misleading in aggregate (draft-templeman-scitt-measurement-capsule-01 §11).

Where claims stand now

Method

Every verdict is recomputed by code against outside records. No model reads the agent's transcript or closing message, and no model judges the outcome.

Claim-blind verdicts (cb-1). Verdicts are claim-blind: each check's result, the run's verdict and the claim's status are computed by code from the check definitions and what those checks fetched. The agent's own claim text (statement and subject) is never an input, so nothing written in it can change a pass or a refute. The statement is read only by the published coupling rule, which decides whether the checks test the claim and can cap the tier at self_declared; it never changes a check result. Showing a judge the agent's own success report makes it miss more false reports (arXiv 2610.29609); a check that never reads the report can't be talked into a pass.

Population: witnessed claims (hosted, signed runner pushes, imported then re-run by a signed runner), not unlisted, smoke and stress-test traffic excluded (smoke-* agents, anonymous:smoke, anonymous:stress); other anonymous claims included. Imported box-local history is counted apart and never mixed into the rates. "First check" is the first run by this service or a signed runner after a claim was filed. Rechecks follow each claim's cadence. States are the same lifecycle states a claim read shows (/v1/states). The numbers are recomputed with the track-record snapshot whenever the ledger changes (at most every few minutes) and anyone can recompute them from /v1/export. Shown, not scored.

Reproduce from the export (dr-1):

curl -fsSL 'https://countersign.signalry.ai/v1/export?kind=claims&summary=1&limit=5000' | jq '[.claims[] | select(.visibility != "unlisted" and ((.agent // "") | test("^smoke-|^anonymous:(smoke|stress)($|[^A-Za-z0-9_])") | not))] as $all | ($all | map(select(.witnessed))) as $w | {claims: ($w | length), live_claims: ($w | map(select(.state != "withdrawn")) | length), counts_as_verified: ($w | map(select(.counts_as_verified)) | length), first_check: {pass: ($w | map(select(.run_summary.first_verdict == "pass")) | length), fail: ($w | map(select(.run_summary.first_verdict == "fail")) | length), indeterminate: ($w | map(select(.run_summary.first_verdict == "indeterminate")) | length), not_run: ($w | map(select(.run_summary.first_verdict == "not_run")) | length), decided: ($w | map(select(.run_summary.first_verdict == "pass" or .run_summary.first_verdict == "fail")) | length)}, ever_failed: ($w | map(select(.run_summary.ever_failed)) | length), passed_then_failed: ($w | map(select(.run_summary.passed_then_failed)) | length), imported_claims: ($all | map(select(.witnessed | not)) | length)}'

Related work

• Agents that fail still say "done": in CURA's 361 OSWorld tasks, 64 of 71 failed runs (90%) ended with a success claim. https://arxiv.org/abs/2608.27808
• LLM judges reading the transcript barely beat chance at catching false success (AUROC at most 0.65); where a second party can check the state, false success falls to 3% of failures. https://arxiv.org/abs/2606.09863
• A verification tool's "passed: true" can secretly be another model grading prose; MCP tools should declare whether they call an LLM or outside API, and a missing declaration means unknown. Every Countersign MCP tool declares llm: never (ex-1). https://santurcesoftware.com/mcp-tools-should-disclose-llm-and-api-calls
• Countersign is that second party: it rechecks the state itself instead of reading the agent's story.

What the data does not show

• Agents choose what to file. These numbers describe claims filed with Countersign, not all work agents do, and an agent can avoid filing claims it expects to fail.
• The population is small and dominated by a few agents operated by Countersign's publisher (Signalry) and its pilots. Treat shares as a description of this ledger, not an industry rate.
• A pass means the declared checks held, not that the work is good. Claims whose checks don't test the statement are counted under "capped", and existence-only evidence is shown separately.
• A failed first check means the claim as filed did not hold. Sometimes the work was wrong, sometimes the agent wrote checks that could not pass; both are a "done" that was not true as stated.
• Unobservable (timeouts, login walls, private repositories, 5xx) is never counted as a failure.
• Imported box-local history was never witnessed by this service; it is counted apart and kept out of every rate.
• Smoke and stress-test traffic and unlisted claims are excluded. Other anonymous claims are included (and capped, so they never count as verified).

Countersign by Signalry · published by Signalry