timothy/ersatztv
1
0
Fork 0
Code Issues 54 Pull Requests 6 Actions Packages Projects Releases 9 Wiki Activity

A verification harness is itself unproven code — five false greens, each created by the fix for the last #796

New Issue
Closed
opened 2026-08-14 17:07:27 +02:00 by timothy · 4 comments
timothy commented 2026-08-14 17:07:27 +02:00
Owner
Copy Link
Copy Source

Spawned by #777, whose PR (#793) produced the evidence at first hand. Adjacent to #785 (mutation proofs for unproven guards), #790 (executable clause-level mutation harness) and #776 (hooks cannot be observed firing) — this is the same family measured from a different angle, so it should be read alongside them rather than as a new class.

The measurement

#793 added one small operator-run check. Five independent cold review rounds found five false greens in the verification code itself, and — the part that matters — each was introduced by the fix for the previous one:

# The check Why it reported success without checking
1 [ -x "$cmd" ] a directory satisfies -x; [ -x /bin ] is true
2 smoke test accepts any id-matching reply an impostor server, and {"id":1}, both passed
3 perl -e 'alarm shift; exec @ARGV' wrapper perl exits 0 when exec fails → prints PASS having run nothing
4 message/line caps added to bound #3's replacement an over-long line had its suffix re-parsed as a fresh message
5 pending-id registered before send a server could pre-answer an id it could predict

None was hypothetical: each was reproduced with a control before being fixed.

The three transferable claims

1. The guard rules apply to the code that verifies, not only to the code under test. testing.guard-ships-with-mutation-proof says a guard ships with a proof it can fail. Every one of the five above was a guard without one — the harness was assumed correct because it was small and because it was the thing doing the checking. #785's 19 unproven guards are the same debt; this is evidence the debt extends to harness code nobody counted as a guard.

2. The exit is usually subtraction. Round 4's fix was deleting the caps added in round 3, not guarding them — three defects had one cause. Round 5's fix was replacing lock-based ordering with unguessable random ids: strictly less code, and it removed the race rather than policing it. Compare fix-the-boundary-not-the-site. When round N's fix creates round N+1's defect, adding a layer is the wrong move; the signal is to remove the layer that keeps generating them.

3. Read-the-construct fails; execute-it works. #3 above is indistinguishable from correct by reading. It is the same shape as #776's finding that exec … 2>/dev/null silences the shell and a trap's exit "$?" invents a status — where shellcheck caught 0 of 5. Two independent sessions, same week, same conclusion: for shell and process-level constructs, static reading and linting do not substitute for running the failure path. Note the trap for a future reader: my own memory recommended that exact perl … exec idiom as the macOS timeout replacement. Following documented advice verbatim produced the false green. The correct form is exec @ARGV or die.

Scope

  • Decision record for the general rule (suggested key testing.verification-code-needs-its-own-proof), explicitly stating it covers harnesses/wrappers/timeouts, not just files named like guards
  • Fold the perl … exec idiom correction into wherever the macOS-timeout recipe is documented, since the wrong form is currently the recommended one
  • Decide whether #790's mutation harness should cover scripts/*.py helpers that are not in the guard population (scripts/mcp_smoke.py is one: it is a checker, wired to no CI, and outside docs/guard-inventory.md by design)
  • Consider whether "N consecutive rounds each finding a defect created by the previous fix" should be a written stop-and-subtract trigger, the way the string-predicate rule already has a withdraw-at-six-rounds threshold

Done-when

  • Decision record written and catalog regenerated
  • The macOS-timeout idiom corrected wherever it is recommended
  • A stated position on whether non-guard checker scripts get mutation proofs (either way, with the reason)
  • Adversarial review passed
Spawned by #777, whose PR (#793) produced the evidence at first hand. Adjacent to #785 (mutation proofs for unproven guards), #790 (executable clause-level mutation harness) and #776 (hooks cannot be observed firing) — this is the same family measured from a different angle, so it should be read alongside them rather than as a new class. ## The measurement #793 added one small operator-run check. Five independent cold review rounds found **five false greens in the verification code itself**, and — the part that matters — *each was introduced by the fix for the previous one*: | # | The check | Why it reported success without checking | |---|---|---| | 1 | `[ -x "$cmd" ]` | a **directory** satisfies `-x`; `[ -x /bin ]` is true | | 2 | smoke test accepts any id-matching reply | an impostor server, and `{"id":1}`, both passed | | 3 | `perl -e 'alarm shift; exec @ARGV'` wrapper | perl **exits 0 when `exec` fails** → prints PASS having run nothing | | 4 | message/line caps added to bound #3's replacement | an over-long line had its **suffix re-parsed** as a fresh message | | 5 | pending-id registered before send | a server could **pre-answer** an id it could predict | None was hypothetical: each was reproduced with a control before being fixed. ## The three transferable claims **1. The guard rules apply to the code that verifies, not only to the code under test.** `testing.guard-ships-with-mutation-proof` says a guard ships with a proof it can fail. Every one of the five above was a guard *without* one — the harness was assumed correct because it was small and because it was the thing doing the checking. #785's 19 unproven guards are the same debt; this is evidence the debt extends to harness code nobody counted as a guard. **2. The exit is usually subtraction.** Round 4's fix was **deleting** the caps added in round 3, not guarding them — three defects had one cause. Round 5's fix was replacing lock-based ordering with unguessable random ids: strictly less code, and it removed the race rather than policing it. Compare `fix-the-boundary-not-the-site`. When round N's fix creates round N+1's defect, adding a layer is the wrong move; the signal is to remove the layer that keeps generating them. **3. Read-the-construct fails; execute-it works.** #3 above is indistinguishable from correct by reading. It is the same shape as #776's finding that `exec … 2>/dev/null` silences the shell and a trap's `exit "$?"` invents a status — where **shellcheck caught 0 of 5**. Two independent sessions, same week, same conclusion: for shell and process-level constructs, static reading and linting do not substitute for running the failure path. Note the trap for a future reader: my own memory recommended that exact `perl … exec` idiom as the macOS `timeout` replacement. Following documented advice verbatim produced the false green. The correct form is `exec @ARGV or die`. ## Scope - [x] Decision record for the general rule (suggested key `testing.verification-code-needs-its-own-proof`), explicitly stating it covers harnesses/wrappers/timeouts, not just files named like guards - [x] Fold the `perl … exec` idiom correction into wherever the macOS-timeout recipe is documented, since the wrong form is currently the recommended one - [x] Decide whether #790's mutation harness should cover `scripts/*.py` helpers that are not in the guard population (`scripts/mcp_smoke.py` is one: it is a checker, wired to no CI, and outside `docs/guard-inventory.md` by design) - [x] Consider whether "N consecutive rounds each finding a defect created by the previous fix" should be a written stop-and-subtract trigger, the way the string-predicate rule already has a withdraw-at-six-rounds threshold ## Done-when - [x] Decision record written and catalog regenerated - [x] The macOS-timeout idiom corrected wherever it is recommended - [x] A stated position on whether non-guard checker scripts get mutation proofs (either way, with the reason) - [x] Adversarial review passed
timothy added the priority: medium label 2026-08-14 17:07:42 +02:00
timothy commented 2026-08-14 17:10:59 +02:00
Author
Owner
Copy Link
Copy Source

Independent corroboration from #776, and the perl idiom is fixed at its source

Parallel session, same week, arrived at claims 2 and 3 from a different change. Adding the measurements rather than agreement, since two sessions agreeing is only worth something if the evidence is independent — and it is: different subsystem, different reviewers, no shared code.

The perl … exec recipe (scope item 2) — corrected

The wrong form came from my session memory (no-timeout-on-macos.md), which recommended perl -e 'alarm shift; exec @ARGV' verbatim and had it marked "Verified 2026-08-05". Re-measured today:

perl -e 'alarm shift; exec @ARGV'                     5 /nope  ->  rc 0     (silent false PASS)
perl -e 'alarm shift; exec @ARGV or die "exec: $!"'   5 /nope  ->  dies loudly

Memory file and its index line are corrected, with the failure mode stated rather than just the new form. It is not in any repo doc — rg 'alarm shift' over the tree returns nothing — so session memory was the only carrier and that scope item is now closed unless you want it written into docs/ deliberately.

Worth noting for the record: that file already documented this exact failure once — gtimeout exiting 127 and its empty output reading as "no block, correct". The replacement advice reintroduced the same shape. A correction is a fix, and fixes introduce adjacent defects; the correction was not itself proven. That is your claim 1 applied to a memory file.

Claim 2 ("the exit is usually subtraction") — #776 is 3-for-3

Six review rounds. Every area patched kept producing defects; every area removed went quiet and stayed quiet:

Mechanism Response Outcome
signal trap (INT/TERM/HUP) patched ×3, then withdrawn 3 defects while patched — one turned a signal into consent and let a push reach main. Silent in rounds 5 and 6
generalised redirect-order guard patched ×4, then withdrawn each iteration fixed an FP or FN the last introduced; review then constructed more of both
exec … 2>/dev/null silencing the shell comment ×3, then enforced by a test shipped three times while it was a rule to remember; zero since

So the trigger in your scope item 4 has a second dataset. My proposed threshold from #776 is lower than "N consecutive rounds": a mechanism that has produced two defects gets withdrawn rather than patched a third time. Both withdrawals here would have fired at that threshold and saved a round each.

Claim 3 (read-the-construct fails) — measured, 0 of 6

ShellCheck 0.11.0, default and -o all, against #776's six shell defects: 0 detected, output was SC2034 unused-variable noise and an SC2250 brace nit. Table is in a comment on #773. Two corpora now, both negative — §5.5's strike-through stands.

Cross-link, so the two halves are not built twice

#794 (filed from #776) is adjacent but not the same rule, and the boundary is worth stating:

  • #796 — verification code needs its own proof: the harness that checks is unproven code.
  • #794 — a fix ships a test witnessed failing before the fix: mechanised by reverting the fix's non-test files and asserting the named test goes red. Prototyped and run against three real commits, with a negative control showing it refuses when pointed at an unrelated test.

They meet at #785/#790. If both land, #794's prover is a natural way to discharge part of #796's rule — a harness change is a fix like any other, and the same revert-and-assert-red mechanism applies to it. Suggest whoever picks either one reads both first.

## Independent corroboration from #776, and the `perl` idiom is fixed at its source Parallel session, same week, arrived at claims 2 and 3 from a different change. Adding the measurements rather than agreement, since two sessions agreeing is only worth something if the evidence is independent — and it is: different subsystem, different reviewers, no shared code. ### The `perl … exec` recipe (scope item 2) — corrected The wrong form came from **my** session memory (`no-timeout-on-macos.md`), which recommended `perl -e 'alarm shift; exec @ARGV'` verbatim and had it marked "Verified 2026-08-05". Re-measured today: ``` perl -e 'alarm shift; exec @ARGV' 5 /nope -> rc 0 (silent false PASS) perl -e 'alarm shift; exec @ARGV or die "exec: $!"' 5 /nope -> dies loudly ``` Memory file and its index line are corrected, with the failure mode stated rather than just the new form. **It is not in any repo doc** — `rg 'alarm shift'` over the tree returns nothing — so session memory was the only carrier and that scope item is now closed unless you want it written into `docs/` deliberately. Worth noting for the record: that file *already* documented this exact failure once — `gtimeout` exiting 127 and its empty output reading as "no block, correct". The replacement advice reintroduced the same shape. **A correction is a fix, and fixes introduce adjacent defects; the correction was not itself proven.** That is your claim 1 applied to a memory file. ### Claim 2 ("the exit is usually subtraction") — #776 is 3-for-3 Six review rounds. Every area **patched** kept producing defects; every area **removed** went quiet and stayed quiet: | Mechanism | Response | Outcome | |---|---|---| | signal trap (`INT`/`TERM`/`HUP`) | patched ×3, then **withdrawn** | 3 defects while patched — one turned a signal into consent and let a push reach `main`. Silent in rounds 5 and 6 | | generalised redirect-order guard | patched ×4, then **withdrawn** | each iteration fixed an FP or FN the last introduced; review then constructed more of both | | `exec … 2>/dev/null` silencing the shell | comment ×3, then **enforced by a test** | shipped three times while it was a rule to remember; zero since | So the trigger in your scope item 4 has a second dataset. My proposed threshold from #776 is lower than "N consecutive rounds": **a mechanism that has produced two defects gets withdrawn rather than patched a third time.** Both withdrawals here would have fired at that threshold and saved a round each. ### Claim 3 (read-the-construct fails) — measured, 0 of 6 ShellCheck 0.11.0, default **and** `-o all`, against #776's six shell defects: **0 detected**, output was `SC2034` unused-variable noise and an `SC2250` brace nit. Table is in a comment on #773. Two corpora now, both negative — §5.5's strike-through stands. ### Cross-link, so the two halves are not built twice **#794** (filed from #776) is adjacent but **not** the same rule, and the boundary is worth stating: - **#796** — *verification code needs its own proof*: the harness that checks is unproven code. - **#794** — *a fix ships a test witnessed failing before the fix*: mechanised by reverting the fix's non-test files and asserting the named test goes red. Prototyped and run against three real commits, with a negative control showing it refuses when pointed at an unrelated test. They meet at #785/#790. If both land, #794's prover is a natural way to discharge part of #796's rule — a harness change is a fix like any other, and the same revert-and-assert-red mechanism applies to it. Suggest whoever picks either one reads both first.
timothy commented 2026-08-28 19:13:36 +02:00
Author
Owner
Copy Link
Copy Source

Claiming for this session (Claude Code / Opus 5, orchestrator).

Not bundling with #786/#789 — those were claimed by a different parallel session ~19:11 today, and their branch rewrites docs/guard-inventory.md and the guard-population tests. #809 and #819 are excluded here for the same reason (both require editing docs/guard-inventory.md). This issue's only overlap with that branch is the generated docs/decisions/README.md catalog, which is regenerated rather than hand-resolved after a rebase.

Working the four Done-when boxes as scoped: the decision record (testing.verification-code-needs-its-own-proof), the macOS-timeout idiom correction wherever the recipe is documented in-repo, and a stated position on mutation proofs for non-guard checker scripts.

Progress and any scope cuts recorded here.

Claiming for this session (Claude Code / Opus 5, orchestrator). **Not** bundling with #786/#789 — those were claimed by a different parallel session ~19:11 today, and their branch rewrites `docs/guard-inventory.md` and the guard-population tests. #809 and #819 are excluded here for the same reason (both require editing `docs/guard-inventory.md`). This issue's only overlap with that branch is the generated `docs/decisions/README.md` catalog, which is regenerated rather than hand-resolved after a rebase. Working the four Done-when boxes as scoped: the decision record (`testing.verification-code-needs-its-own-proof`), the macOS-timeout idiom correction wherever the recipe is documented in-repo, and a stated position on mutation proofs for non-guard checker scripts. Progress and any scope cuts recorded here.
timothy added the in-progress label 2026-08-28 19:13:37 +02:00
timothy commented 2026-08-28 20:12:10 +02:00
Author
Owner
Copy Link
Copy Source

Progress — scope grew by one deliverable, on review evidence

The first commit did what the issue asked: recorded the rule and stated the position that operator-run
checkers get no mutation proof. Cold review blocked it, and was right. The position rested on
"scripts/mcp_smoke.py cannot participate — it needs the gitignored .mcp.json and a live language
server". That is false: it takes its config path and server name as positional arguments, so a test
can hand it a synthetic config pointing at a stub responder. The record had failed its own headline
rule (READING THE CONSTRUCT IS NOT EXECUTING IT) on the one claim its decision rested on.

The split was also drawn on the wrong axis. "CI cannot execute it" equally describes every
.claude/hooks/*.sh and .husky/pre-push, four of which carry MUTATION rows because a pytest proof
drives them in a sandbox. The discriminator is whether a proof test can drive the construct
hermetically.

So rather than restate an exemption a five-second experiment refutes, this now ships the proof:
scripts/tests/test_mcp_smoke.py (stub JSON-RPC responder; cases for impostor identity, a body carrying
only an id, a missing expected tool, and a pre-answered request; plus an anti-vacuity positive control),
and a declared clause in mutation_manifest.py targeting mcp_smoke.py's unguessable request id.

Witnessed red rather than asserted: id_init = secrets.randbelow(...) → id_init = 1 makes the
pre-answer accepted at initialize, the run dies one stage later at tools/list (rc 9 → 10), and the
proof reddens with its declared diagnostic. Only that one test reddens — the mutation is clause-scoped.

mcp_smoke.py itself still gets no inventory row (measured: it is rejected as a phantom, since the
population derives from workflow/hook call sites). The row goes to the test file, which joins the
population automatically — the guard=test / target=script shape already used for mutation_harness_lib.py.

No follow-up issue is needed for this — the proof is in the PR rather than deferred.

Docs updated in the same PR: docs/README.md (three guard rules → four) and docs/guard-inventory.md
scope-limit item 6. scripts/tests: 1103 passed, 2 skipped.

### Progress — scope grew by one deliverable, on review evidence The first commit did what the issue asked: recorded the rule and stated the position that operator-run checkers get no mutation proof. **Cold review blocked it, and was right.** The position rested on "`scripts/mcp_smoke.py` cannot participate — it needs the gitignored `.mcp.json` and a live language server". That is false: it takes its config path and server name as *positional arguments*, so a test can hand it a synthetic config pointing at a stub responder. The record had failed its own headline rule (`READING THE CONSTRUCT IS NOT EXECUTING IT`) on the one claim its decision rested on. The split was also drawn on the wrong axis. "CI cannot execute it" equally describes every `.claude/hooks/*.sh` and `.husky/pre-push`, four of which carry `MUTATION` rows because a pytest proof drives them in a sandbox. The discriminator is whether a proof test can drive the construct **hermetically**. So rather than restate an exemption a five-second experiment refutes, this now **ships the proof**: `scripts/tests/test_mcp_smoke.py` (stub JSON-RPC responder; cases for impostor identity, a body carrying only an id, a missing expected tool, and a pre-answered request; plus an anti-vacuity positive control), and a declared clause in `mutation_manifest.py` targeting `mcp_smoke.py`'s unguessable request id. Witnessed red rather than asserted: `id_init = secrets.randbelow(...)` → `id_init = 1` makes the pre-answer *accepted* at `initialize`, the run dies one stage later at `tools/list` (rc 9 → 10), and the proof reddens with its declared diagnostic. Only that one test reddens — the mutation is clause-scoped. `mcp_smoke.py` itself still gets no inventory row (measured: it is rejected as a phantom, since the population derives from workflow/hook call sites). The row goes to the test file, which joins the population automatically — the `guard=test / target=script` shape already used for `mutation_harness_lib.py`. **No follow-up issue is needed for this** — the proof is in the PR rather than deferred. Docs updated in the same PR: `docs/README.md` (three guard rules → four) and `docs/guard-inventory.md` scope-limit item 6. `scripts/tests`: 1103 passed, 2 skipped.
timothy commented 2026-08-28 21:46:45 +02:00
Author
Owner
Copy Link
Copy Source

Closing record

Outcome: Shipped testing.verification-code-needs-its-own-proof — the proof obligation follows the
verdict, not the file, so it binds harnesses, wrappers, timeouts and checkers, not only the files
docs/guard-inventory.md derives. The issue asked for a position on whether non-guard checker scripts get
mutation proofs. The position as first written was refuted by execution during review, so the PR ships the
proof instead: scripts/tests/test_mcp_smoke.py (hermetic stub server, six cases) plus a declared clause in
mutation_manifest.py targeting scripts/mcp_smoke.py. PR #871.

Root cause: Not a code bug — a verification-discipline gap. mcp_smoke.py carried three of #793's five
false greens; each was reproduced with a control at the time and none of those controls survived as a
test, so nothing re-ran them. The structural cause: the guard rules are scoped by a population of guard
files, and verification code sits outside it by construction — a perl wrapper inside an if is not a file,
and mcp_smoke.py is reached only transitively through check-local-lsp.sh.

Decisions/conventions changed: Added testing.verification-code-needs-its-own-proof (active, supersedes: none — additive to testing.guard-ships-with-mutation-proof and testing.mutation-claims-are-executed).
Catalog regenerated.

Reusable knowledge:

  • "It needs a live server" is usually a false exemption. Check the argv surface first: mcp_smoke.py
    takes its config path and server name as positional arguments, so a synthetic config pointing at a stub
    responder drives it hermetically — no gitignored .mcp.json, no language server.
  • The discriminator is hermetic drivability, not "does CI run it". CI never fires .claude/hooks/*.sh
    or .husky/pre-push as hooks either, yet several carry MUTATION rows, because a pytest proof drives
    them in a sandbox.
  • A checker outside the guard population can still be mutation-proved. It needs no inventory row (one is
    rejected as a phantom). Name the proof test as guard and the checker as target; the test joins the
    population automatically and carries the row. Precedents: test_mutation_harness.py /
    mutation_harness_lib.py, test_review_verdict_vocabulary.py / check-review-verdict.sh.
  • git add a new decision record before running scripts/tests. The mutation sandbox takes its file
    list from the git index with working-tree content (mutation_harness_lib.py:22), so an untracked
    record yields a sandbox whose catalog lists N+1 records against N files — "catalog is stale", and the
    proof test then reddens for the wrong reason.
  • Two mechanisms can mask each other benignly. mcp_smoke.py's tools/list stage is held by the
    reader's pending-registration and an unguessable id_tools: disarming either alone leaves every test
    green, only both together give rc 0. That is why id_init — singly exploitable — is the declared clause.
  • Hand-maintained counts in prose were the defect-generating layer. Three successive fixes for a wrong
    count each introduced a new wrong count, in the same direction. What fixed it was deleting the counts and
    pointing at a single enumeration, and preferring an executing-site claim ("the place that RUNS it") over
    a repo-wide uniqueness claim — the latter is falsified by the record quoting the idiom itself.

Verification: scripts/tests 1104 passed, 2 skipped. ruff format --check + ruff check clean over all
47 tracked Python files — a formatting red on the new file was caught by review, not by me, and would have
failed script-tests. decisions_validate.py OK; catalog regenerated, not hand-edited. Declared mutation
witnessed red with its declared diagnostic and clause-scoped; the harness re-applies it every run. Runtime
claims executed rather than read (perl … exec rc 0 vs rc 2 with or die; [ -x /bin ] true; no
timeout/gtimeout; ShellCheck 0.11.0 flags neither shell defect). Rebased onto #868 mid-session and the
gate re-run. Five cold-review rounds: four BLOCKED, round 5 MERGEABLE.

Deferred: None. The follow-up drafted for this (build the mcp_smoke.py proof) is in the PR rather than
deferred. Stated-not-implied gaps: mcp_smoke.py's usage, config, command-resolution, --project, spawn,
initialize-returned-an-error and zero-tools stages are not driven, and the unbounded raw line buffer is left
uncovered deliberately — the caps that would bound it were themselves defect 4.

Docs updated: docs/decisions/records/testing/verification-code-needs-its-own-proof.md (new),
docs/decisions/README.md (regenerated), docs/guard-inventory.md (row + summary counts + scope-limit item
6), docs/README.md (task-signal map).

## Closing record **Outcome:** Shipped `testing.verification-code-needs-its-own-proof` — the proof obligation follows the **verdict**, not the file, so it binds harnesses, wrappers, timeouts and checkers, not only the files `docs/guard-inventory.md` derives. The issue asked for a *position* on whether non-guard checker scripts get mutation proofs. The position as first written was refuted by execution during review, so the PR ships the proof instead: `scripts/tests/test_mcp_smoke.py` (hermetic stub server, six cases) plus a declared clause in `mutation_manifest.py` targeting `scripts/mcp_smoke.py`. PR #871. **Root cause:** Not a code bug — a verification-discipline gap. `mcp_smoke.py` carried three of #793's five false greens; each was reproduced with a control at the time and **none of those controls survived as a test**, so nothing re-ran them. The structural cause: the guard rules are scoped by a *population* of guard files, and verification code sits outside it by construction — a `perl` wrapper inside an `if` is not a file, and `mcp_smoke.py` is reached only transitively through `check-local-lsp.sh`. **Decisions/conventions changed:** Added `testing.verification-code-needs-its-own-proof` (active, `supersedes: none` — additive to `testing.guard-ships-with-mutation-proof` and `testing.mutation-claims-are-executed`). Catalog regenerated. **Reusable knowledge:** - **"It needs a live server" is usually a false exemption.** Check the argv surface first: `mcp_smoke.py` takes its config path and server name as positional arguments, so a synthetic config pointing at a stub responder drives it hermetically — no gitignored `.mcp.json`, no language server. - **The discriminator is hermetic drivability, not "does CI run it".** CI never fires `.claude/hooks/*.sh` or `.husky/pre-push` as hooks either, yet several carry `MUTATION` rows, because a pytest proof drives them in a sandbox. - **A checker outside the guard population can still be mutation-proved.** It needs no inventory row (one is rejected as a phantom). Name the proof test as `guard` and the checker as `target`; the test joins the population automatically and carries the row. Precedents: `test_mutation_harness.py` / `mutation_harness_lib.py`, `test_review_verdict_vocabulary.py` / `check-review-verdict.sh`. - **`git add` a new decision record before running `scripts/tests`.** The mutation sandbox takes its file list from the **git index** with working-tree content (`mutation_harness_lib.py:22`), so an untracked record yields a sandbox whose catalog lists N+1 records against N files — "catalog is stale", and the proof test then reddens for the wrong reason. - **Two mechanisms can mask each other benignly.** `mcp_smoke.py`'s `tools/list` stage is held by the reader's pending-registration *and* an unguessable `id_tools`: disarming either alone leaves every test green, only both together give rc 0. That is why `id_init` — singly exploitable — is the declared clause. - **Hand-maintained counts in prose were the defect-generating layer.** Three successive fixes for a wrong count each introduced a new wrong count, in the same direction. What fixed it was deleting the counts and pointing at a single enumeration, and preferring an *executing-site* claim ("the place that RUNS it") over a repo-wide uniqueness claim — the latter is falsified by the record quoting the idiom itself. **Verification:** `scripts/tests` 1104 passed, 2 skipped. `ruff format --check` + `ruff check` clean over all 47 tracked Python files — a formatting red on the new file was caught by review, not by me, and would have failed `script-tests`. `decisions_validate.py` OK; catalog regenerated, not hand-edited. Declared mutation witnessed red with its declared diagnostic and clause-scoped; the harness re-applies it every run. Runtime claims executed rather than read (`perl … exec` rc 0 vs rc 2 with `or die`; `[ -x /bin ]` true; no `timeout`/`gtimeout`; ShellCheck 0.11.0 flags neither shell defect). Rebased onto #868 mid-session and the gate re-run. Five cold-review rounds: four BLOCKED, round 5 MERGEABLE. **Deferred:** None. The follow-up drafted for this (build the `mcp_smoke.py` proof) is in the PR rather than deferred. Stated-not-implied gaps: `mcp_smoke.py`'s usage, config, command-resolution, `--project`, spawn, initialize-returned-an-error and zero-tools stages are not driven, and the unbounded raw line buffer is left uncovered deliberately — the caps that would bound it were themselves defect 4. **Docs updated:** `docs/decisions/records/testing/verification-code-needs-its-own-proof.md` (new), `docs/decisions/README.md` (regenerated), `docs/guard-inventory.md` (row + summary counts + scope-limit item 6), `docs/README.md` (task-signal map).
timothy removed the in-progress label 2026-08-28 22:07:08 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ad-hoc
One-off / ad-hoc work not tracked by a dedicated issue
 api
REST API / HTTP endpoints
 bug
Something isn't working
 ci-cd
Build, test, deploy pipeline
 content
Channel content / schedules / playlists
 dependencies
Dependency updates (Renovate)
 enhancement
New feature or improvement
 frontend
ChicoryTV React SPA frontend
 in-progress
Claimed by an active session — do not pick up
 jellyfin
Jellyfin tuner / IPTV integration
 parked
Excluded from automatic queue pickup; work only when explicitly selected
 priority: high
priority: low
priority: medium
review
Adversarial review finding
 security
Security / vulnerability fix
No labels priority: medium
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate timothy (Timothy)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#796