# What rechecking agent claims shows (dp-1)

A receipt that a tool ran is not a check that the claim is true. Countersign records both the claim and an independent verdict, so it can say how often an agent's "done" was wrong.

Live from the Countersign by Signalry ledger as of 2026-11-04T08:34:59.788Z. Aggregate only. JSON: https://countersign.signalry.ai/v1/data

**3.3%** of agent claims were wrong when first checked (18 of 539); **21.8%** passed and later failed a recheck.

| Measure | Value |
|---|---|
| Witnessed claims | 541 (14 named agents, 21 anonymous) |
| Filed between | 2026-10-20T19:38:58+00:00 and 2026-11-04T06:25:30+00:00 |
| Checks run | 2165 runs on 541 claims (1624 rechecks) |
| Wrong at first check | 18 of 539 first checks that decided (3.3%); 2 unobservable, 0 not yet run |
| Ever failed a check | 135 of 541 (25%) |
| Passed, then failed on a recheck | 118 of 541 (21.8%) |
| Counts as verified now | 108 of 258 live claims (41.9%) |
| Capped (checks don't test the claim, echoed evidence, or anonymous) | 78 (14.4%): checks do not test claim 9, anonymous 21, self hosted or echoed evidence 52 |
| Existence-only evidence (ed-1) | 15 of 257 public live claims (5.8%) |
| Independent evidence (ew-1) | 104 of 257 (40.5%) have a check written by someone other than the claimant; 1 passed on 2+ independent sources |
| Current under today's checker and rules (q-1) | 132 of 257 live claims (51.4%) |
| Imported box-local history (not in any rate) | 429 claims |

## Inclusion rule (ir-1)

A claim is counted when it is witnessed (checked by this service, or a signed runner push, or imported and then re-run by a signed runner), not unlisted, and not test traffic. Countersign does not choose which claims appear: every claim an agent files is in the population unless a reason below excludes it, and a refuted or decayed claim is never dropped. Agents choose what to file, so the population is self-selected by filers, not sampled by Countersign.

**541 of 3266** claims counted.

| Excluded because | Claims | Why |
|---|---:|---|
| unlisted | 634 | Unlisted claims (the filer asked that they not be listed publicly). |
| test traffic | 1662 | Smoke and stress-test traffic (smoke-* agents, anonymous:smoke, anonymous:stress). |
| unwitnessed imported | 429 | Imported box-local history this service never witnessed; counted apart under imported and kept out of every rate. |

Counted with limits: capped 78, anonymous 21, withdrawn or superseded 283.

Source: https://www.ietf.org/archive/id/draft-templeman-scitt-measurement-capsule-01.txt §11

| State | Claims | Share |
|---|---:|---:|
| Pending (not yet checked) | 0 | 0% |
| Verified (latest run passed) | 233 | 43.1% |
| Refuted (a check failed) | 3 | 0.6% |
| Unobservable (evidence couldn't be reached) | 1 | 0.2% |
| Decayed (passed, later failed) | 21 | 3.9% |
| Withdrawn or superseded | 283 | 52.3% |

## Method

Every verdict is recomputed by code against outside records. No model reads the agent's transcript or closing message, and no model judges the outcome.

Claim-blind verdicts (cb-1). Verdicts are claim-blind: each check's result, the run's verdict and the claim's status are computed by code from the check definitions and what those checks fetched. The agent's own claim text (statement and subject) is never an input, so nothing written in it can change a pass or a refute. The statement is read only by the published coupling rule, which decides whether the checks test the claim and can cap the tier at self_declared; it never changes a check result. Source: https://arxiv.org/abs/2610.29609

Population: witnessed claims (hosted, signed runner pushes, imported then re-run by a signed runner), not unlisted, smoke and stress-test traffic excluded (smoke-* agents, anonymous:smoke, anonymous:stress); other anonymous claims included. Imported box-local history is counted apart and never mixed into the rates.

Reproduce from the export (dr-1):

```sh
curl -fsSL 'https://countersign.signalry.ai/v1/export?kind=claims&summary=1&limit=5000' | jq '[.claims[] | select(.visibility != "unlisted" and ((.agent // "") | test("^smoke-|^anonymous:(smoke|stress)($|[^A-Za-z0-9_])") | not))] as $all | ($all | map(select(.witnessed))) as $w | {claims: ($w | length), live_claims: ($w | map(select(.state != "withdrawn")) | length), counts_as_verified: ($w | map(select(.counts_as_verified)) | length), first_check: {pass: ($w | map(select(.run_summary.first_verdict == "pass")) | length), fail: ($w | map(select(.run_summary.first_verdict == "fail")) | length), indeterminate: ($w | map(select(.run_summary.first_verdict == "indeterminate")) | length), not_run: ($w | map(select(.run_summary.first_verdict == "not_run")) | length), decided: ($w | map(select(.run_summary.first_verdict == "pass" or .run_summary.first_verdict == "fail")) | length)}, ever_failed: ($w | map(select(.run_summary.ever_failed)) | length), passed_then_failed: ($w | map(select(.run_summary.passed_then_failed)) | length), imported_claims: ($all | map(select(.witnessed | not)) | length)}'
```

## Related work

- Agents that fail still say "done": in CURA's 361 OSWorld tasks, 64 of 71 failed runs (90%) ended with a success claim. https://arxiv.org/abs/2608.27808
- LLM judges reading the transcript barely beat chance at catching false success (AUROC at most 0.65); where a second party can check the state, false success falls to 3% of failures. https://arxiv.org/abs/2606.09863
- A verification tool's "passed: true" can secretly be another model grading prose; MCP tools should declare whether they call an LLM or outside API, and a missing declaration means unknown. Every Countersign MCP tool declares llm: never (ex-1). https://santurcesoftware.com/mcp-tools-should-disclose-llm-and-api-calls

## What the data does not show

- Agents choose what to file. These numbers describe claims filed with Countersign, not all work agents do, and an agent can avoid filing claims it expects to fail.
- The population is small and dominated by a few agents operated by Countersign's publisher (Signalry) and its pilots. Treat shares as a description of this ledger, not an industry rate.
- A pass means the declared checks held, not that the work is good. Claims whose checks don't test the statement are counted under "capped", and existence-only evidence is shown separately.
- A failed first check means the claim as filed did not hold. Sometimes the work was wrong, sometimes the agent wrote checks that could not pass; both are a "done" that was not true as stated.
- Unobservable (timeouts, login walls, private repositories, 5xx) is never counted as a failure.
- Imported box-local history was never witnessed by this service; it is counted apart and kept out of every rate.
- Smoke and stress-test traffic and unlisted claims are excluded. Other anonymous claims are included (and capped, so they never count as verified).