docs(796): verification code is code under test — and the proof it was claiming an exemption from #871

Merged
timothy merged 9 commits from fix/796-verification-code-proof into main 2026-08-28 22:06:46 +02:00
7 changed files with 356 additions and 4 deletions
+1 -1
View File
@@ -27,7 +27,7 @@ doc below, or that changes which sections a task signal points to.**
| Concurrency / optimistic-locking work | `docs/api-conventions.md` §7a/b/c + `docs/decisions/optimistic-concurrency.md` |
| Auth / security-surface work | `docs/decisions/api-auth-security.md` |
| CI / release pipeline work | `docs/ci-cd.md` + `docs/decisions/release-ci-governance.md` |
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible), then the three rules every guard must satisfy: `docs/decisions/records/testing/guard-derives-population-from-source.md`, `…/guard-ships-with-mutation-proof.md` and `…/mutation-claims-are-executed.md` (a `MUTATION` grade carries a DECLARED clause mutation that is re-run every suite) |
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible), then the rules every guard must satisfy: `docs/decisions/records/testing/guard-derives-population-from-source.md`, `…/guard-ships-with-mutation-proof.md` and `…/mutation-claims-are-executed.md` (a `MUTATION` grade carries a DECLARED clause mutation that is re-run every suite) — plus `…/verification-code-needs-its-own-proof.md`, which extends the same obligation BEYOND guards to the harness, wrapper or checker doing the checking, and says where its proof lives when the checker holds no row |
| Adding or bounding a consequential numeric config field (an FFmpeg profile tunable, a pipeline knob) | `docs/api-conventions.md` §3d — reject out of range with a 422 naming the bound and its consequence, never accept-then-rewrite; validate against the constants the renderer reads, keep the render-time clamp for pre-existing rows, and let an UNCHANGED legacy value through on update. Then `api.ffmpeg-profile-numeric-bounds` |
| Testing a surface gated by config / an env var / a credential | `docs/decisions/records/testing/deny-path-at-production-config-value.md` — cover the setting absent, at its production value, and each opt-out, and assert the DENY branch |
| Touching a full-replace write path or a hand-built request object | `docs/decisions/records/testing/full-replace-asserts-field-list.md` — derive the field list from the DTO and assert set equality; reconcile by id where child state exists. In the SPA the same rule is enforced by the type system: `docs/spa-conventions.md` §4b — build the body as `Complete<T>`, annotating BOTH the wrapper parameter and every construction site |
+1
View File
@@ -218,6 +218,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
| `testing.playwright-mcp-download-and-recovery` | In Playwright-MCP E2E, fetch file-download endpoints with curl — never a browser tab or `window.open` — and if browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session. | 2026-07-21 | [link](records/testing/playwright-mcp-download-and-recovery.md) |
| `testing.scripted-playout-golden-deferred` | The `PlayoutBuildGoldenTests` in-memory golden net covers Sequential (YAML) as of #381. Scripted's *end-to-end pipeline* is excluded — `ScriptedPlayoutBuilder` runs a user-authored external program that drives the engine over HTTP loopback, which the in-memory harness can't pin — so that full-pipeline (integration) harness is deferred to #563. But the scheduling *behavior* those scripts drive lives entirely in the in-process `SchedulingEngine` (the `ScriptedScheduleController` is a 1:1 pass-through to it), which IS directly unit/golden-testable; the earlier "Scripted is un-golden-able by construction" framing overstated the constraint by conflating transport with engine. #395 extracts that shared switch to `ContentEnumeratorBuilder` and adds a direct regression net (`ContentEnumeratorBuilderTests`) over it. | 2026-07-22 | [link](records/testing/scripted-playout-golden-deferred.md) |
| `testing.troubleshoot-path-cannot-test-branding` | Verify logo/watermark/bug changes through a real channel playout — a green troubleshoot run proves nothing about branding. | 2026-07-21 | [link](records/testing/troubleshoot-path-cannot-test-branding.md) |
| `testing.verification-code-needs-its-own-proof` | VERIFICATION CODE IS CODE UNDER TEST. The obligation in `testing.guard-ships-with-mutation-proof` attaches to whatever emits a PASS/FAIL somebody acts on — a smoke test, a timeout wrapper, an operator-run checker, the harness that applies the mutations — and NOT only to files sitting in the guard population or named like guards; every one of the five defects measured below was in code nobody had counted as a guard. THE DISCRIMINATOR IS WHETHER A PROOF TEST CAN DRIVE THE CONSTRUCT HERMETICALLY, NOT WHETHER CI RUNS IT, and the difference is load-bearing rather than pedantic: `.claude/hooks/*.sh` and `.husky/pre-push` are operator-run and CI never fires them AS HOOKS, yet several of them carry `MUTATION` rows — among them `pretooluse-bom-guard.sh`, `pretooluse-worktree-guard.sh` and `.husky/pre-push` — because a pytest proof drives them in a sandbox. A does-CI-run-it test would sort those already-proven guards into the exempt branch. (Most of the remaining hooks are graded `NONE`; that is unproven debt, not evidence for the other discriminator.) "IT NEEDS A REAL SERVER" IS AN EXEMPTION CLAIM AND IT IS USUALLY FALSE: `scripts/mcp_smoke.py` takes its config path and server name as POSITIONAL ARGUMENTS, so a test hands it a synthetic config in a tmpdir pointing at a stub responder and drives it with no gitignored `.mcp.json` and no language server, which is how it is proven in `scripts/tests/test_mcp_smoke.py`. That coverage is a SUBSET and no count of it is kept here: the file's own docstring enumerates the cases it drives and names what it does not, so there is a list rather than a number to keep in step with it. A checker is never excused because its CALLER cannot run in CI. WHERE THE PROOF LIVES: the checker itself needs NO `docs/guard-inventory.md` row, because that population is derived from workflow and hook call sites and a transitively-reached script is rejected as a phantom; the proof goes in an ordinary `scripts/tests/test_*.py`, which joins the population automatically and carries the row, while the manifest entry names THAT test as `guard` and the checker as `target`. That shape already exists for `mutation_harness_lib.py` and needs no change to any population. ONLY WHAT NO PROOF TEST CAN DRIVE HERMETICALLY falls back to a WITNESSED NEGATIVE CONTROL — the failure path executed once, the false-green form reproduced, both recorded in the PR, and its decay stated because nothing re-runs it. `scripts/check-local-lsp.sh` is the genuine case: it asserts developer-machine installs (a global `csharp-ls`, a `host/fxr` under `DOTNET_ROOT`, a root `node_modules`) that no runner has. STOP-AND-SUBTRACT, adopted here as this record's own threshold: when a review round's finding was CREATED by the previous round's fix and that happens TWICE CONSECUTIVELY, delete the layer generating them rather than guarding it. READING THE CONSTRUCT IS NOT EXECUTING IT: ShellCheck 0.11.0 flagged NEITHER of the two shell defects below, and each is indistinguishable from correct by reading. | 2026-08-28 | [link](records/testing/verification-code-needs-its-own-proof.md) |
## Review due
@@ -0,0 +1,85 @@
---
key: testing.verification-code-needs-its-own-proof
title: '2026-08-28 — verification code is code under test: the obligation follows the VERDICT, and the exempt branch is what no proof test can drive hermetically (#796)'
status: active
since: '2026-08-28'
supersedes: none
superseded-by: none
rule: 'VERIFICATION CODE IS CODE UNDER TEST. The obligation in `testing.guard-ships-with-mutation-proof` attaches to whatever emits a PASS/FAIL somebody acts on — a smoke test, a timeout wrapper, an operator-run checker, the harness that applies the mutations — and NOT only to files sitting in the guard population or named like guards; every one of the five defects measured below was in code nobody had counted as a guard. THE DISCRIMINATOR IS WHETHER A PROOF TEST CAN DRIVE THE CONSTRUCT HERMETICALLY, NOT WHETHER CI RUNS IT, and the difference is load-bearing rather than pedantic: `.claude/hooks/*.sh` and `.husky/pre-push` are operator-run and CI never fires them AS HOOKS, yet several of them carry `MUTATION` rows — among them `pretooluse-bom-guard.sh`, `pretooluse-worktree-guard.sh` and `.husky/pre-push` — because a pytest proof drives them in a sandbox. A does-CI-run-it test would sort those already-proven guards into the exempt branch. (Most of the remaining hooks are graded `NONE`; that is unproven debt, not evidence for the other discriminator.) "IT NEEDS A REAL SERVER" IS AN EXEMPTION CLAIM AND IT IS USUALLY FALSE: `scripts/mcp_smoke.py` takes its config path and server name as POSITIONAL ARGUMENTS, so a test hands it a synthetic config in a tmpdir pointing at a stub responder and drives it with no gitignored `.mcp.json` and no language server, which is how it is proven in `scripts/tests/test_mcp_smoke.py`. That coverage is a SUBSET and no count of it is kept here: the file''s own docstring enumerates the cases it drives and names what it does not, so there is a list rather than a number to keep in step with it. A checker is never excused because its CALLER cannot run in CI. WHERE THE PROOF LIVES: the checker itself needs NO `docs/guard-inventory.md` row, because that population is derived from workflow and hook call sites and a transitively-reached script is rejected as a phantom; the proof goes in an ordinary `scripts/tests/test_*.py`, which joins the population automatically and carries the row, while the manifest entry names THAT test as `guard` and the checker as `target`. That shape already exists for `mutation_harness_lib.py` and needs no change to any population. ONLY WHAT NO PROOF TEST CAN DRIVE HERMETICALLY falls back to a WITNESSED NEGATIVE CONTROL — the failure path executed once, the false-green form reproduced, both recorded in the PR, and its decay stated because nothing re-runs it. `scripts/check-local-lsp.sh` is the genuine case: it asserts developer-machine installs (a global `csharp-ls`, a `host/fxr` under `DOTNET_ROOT`, a root `node_modules`) that no runner has. STOP-AND-SUBTRACT, adopted here as this record''s own threshold: when a review round''s finding was CREATED by the previous round''s fix and that happens TWICE CONSECUTIVELY, delete the layer generating them rather than guarding it. READING THE CONSTRUCT IS NOT EXECUTING IT: ShellCheck 0.11.0 flagged NEITHER of the two shell defects below, and each is indistinguishable from correct by reading.'
signals: 'the verification harness is itself unproven code · a false green in the checker rather than the checked · `[ -x "$cmd" ]` is true for a DIRECTORY · perl alarm/exec exits 0 without `or die` · macOS has no `timeout` and no `gtimeout` · smoke test accepts an impostor server · a server that pre-answers a guessable id · each fix created the next round''s defect · stop and subtract rather than add a layer · hermetically drivable is the test, not does-CI-run-it · a checker is not exempt because its CALLER cannot run in CI · the checker needs no inventory row — name the proof test as `guard` and the checker as `target` · a transitively-reached script is rejected as a PHANTOM · witnessed negative control for what cannot be driven · paths: `scripts/mcp_smoke.py`, `scripts/tests/test_mcp_smoke.py`, `scripts/check-local-lsp.sh`, `scripts/tests/mutation_manifest.py`, `docs/guard-inventory.md` · issues: #796, #793, #790, #785, #777, #776'
mechanics: 'THE macOS TIMEOUT RECIPE, recorded here because the WRONG form was the recommended one and a session that followed it verbatim produced a false green (#793): `perl -e ''alarm shift; exec @ARGV or die "exec failed: $!\n"'' <secs> <cmd...>`. The `or die` is load-bearing — without it perl exits 0 when `exec` fails, so a missing or misspelt binary reports SUCCESS having run nothing (measured 2026-08-28: `perl -e ''alarm shift; exec @ARGV'' 5 /nope` exits 0; the `or die` form exits 2 and says why). macOS has neither `timeout` (GNU coreutils) nor `gtimeout` on this fleet, and an absent `gtimeout` exits 127 with empty output, which reads as a clean pass to any caller grepping stdout for a failure string. Two traps travel with it: `[ -x "$cmd" ]` does not establish that a command is runnable, because a directory satisfies `-x` (`[ -x /bin ]` is true); and whatever wrapper is used, its EXIT CODE must be checked rather than its stdout. The place that RUNS this recipe is `scripts/check-local-lsp.sh`; the other in-repo occurrences of the idiom are in this record. The `-x` trap appears there only in a comment recording what it deliberately does not do.'
---
**The measurement.** #793 added one small operator-run check — a smoke test that starts the
`csharp-lsp` MCP server and lists its tools. Five independent cold-review rounds found five false
greens *in the verification code itself*, each after the first introduced by the fix for the one before, and each
reproduced with a control before being fixed:
| # | The check | Why it reported success without checking |
|---|---|---|
| 1 | `[ -x "$cmd" ]` | a **directory** satisfies `-x`; `[ -x /bin ]` is true |
| 2 | smoke test accepts any id-matching reply | an impostor server, and `{"id":1}`, both passed |
| 3 | `perl -e 'alarm shift; exec @ARGV'` wrapper | perl **exits 0 when `exec` fails** → prints PASS having run nothing |
| 4 | message/line caps added to bound #3's replacement | an over-long line had its **suffix re-parsed** as a fresh message |
| 5 | pending-id registered before send | a server could **pre-answer** an id it could predict |
**Why this is a separate rule and not an application of #775.** `testing.guard-ships-with-mutation-
proof` is scoped by a population — the guard files `docs/guard-inventory.md` derives and grades.
Defects 1 and 3 are a `[ -x ]` test and a `perl` wrapper inside `scripts/check-local-lsp.sh`, which
no workflow or hook invokes; 2, 4 and 5 live in `scripts/mcp_smoke.py`, reached only through that
script and so excluded by the transitive-calls scope limit (item 6). Neither file is in the population. That scoping is what keeps the inventory from becoming a chore that gets
rubber-stamped; what was missing is the claim that the obligation follows the *verdict*, not the *file*.
**The discriminator, stated precisely because the obvious one is wrong.** It is not "does CI run
this": CI never fires `.claude/hooks/*.sh` or `.husky/pre-push` as hooks either, yet several of them
are graded `MUTATION`, because a pytest proof drives them in a sandbox. The question is whether a
proof test can drive the construct *hermetically*, and "it needs a live server" usually fails that
test rather than passing it. `mcp_smoke.py` takes its config path and server name as positional
arguments, so `scripts/tests/test_mcp_smoke.py` hands it a synthetic config pointing at a stub
responder and drives it with no server to start. What it covers is a subset, enumerated in that
file's docstring rather than counted here. `check-local-lsp.sh` genuinely
cannot be driven that way: it asserts a global `csharp-ls` on PATH, a `host/fxr` under
`DOTNET_ROOT`, and a root `node_modules`, which is developer-machine state no runner has. That, and
not "it is operator-run", is what puts it in the exempt branch. Note what that concedes: two of
the five defects above (the `-x` test and the `perl` wrapper) were in `check-local-lsp.sh` itself,
so the exempt branch is not defect-free — it is the part where a recorded control is the best
available proof. Of the three in `mcp_smoke.py`, defects 2 and 5 are pinned by tests; defect 4's
territory is deliberately left uncovered, because the message caps that would bound it WERE that
defect, and a test pressing on it would argue for reintroducing them.
**Where the proof lives, and why the checker needs no row.** `docs/guard-inventory.md`'s population
is derived from workflow and hook call sites, so a row for `mcp_smoke.py` is rejected outright —
measured, not assumed:
AssertionError: guard-inventory.md lists ['scripts/mcp_smoke.py'], which no longer exist or are
no longer invoked by any workflow or hook. A row for a guard that does not run reads as coverage
and is not.
The row is therefore never the route. The proof goes in `scripts/tests/test_*.py`, which joins the
population automatically and carries the row, and the manifest entry names that test as `guard` with
the checker as `target` — the shape `test_mutation_harness.py` already uses for
`mutation_harness_lib.py`, itself outside the population under scope limit item 7. No population
derivation changes; the declared clause is re-applied every run like any other.
**The residual, for the exempt branch only.** A witnessed negative control recorded in a PR is
weaker than a mutation re-run every suite: nothing re-executes it, and a later edit can silently
undo what it proved. The alternative there is not a stronger check but a green one that runs
nowhere — this repo's most expensive recurring failure (#631, #719, #751).
**Stop-and-subtract.** Rounds 3→4 are the clearest case: round 3 replaced the broken wrapper, round
4 found that the caps round 3 added re-parsed an over-long line's suffix, and the fix was **deleting
those caps** rather than guarding them — the layer was the cause, not any clause inside it. Round 5
likewise replaced lock-based ordering with
unguessable random ids: strictly less code, and it removed the race rather than policing it. Two
consecutive rounds of fix-creates-the-next-defect is adopted here as the threshold; the six-round
text-comparison case behind `release.verdict-vocabulary-shared` is precedent for the shape, not its
source, and it concluded in a categorical ban rather than a count.
**Read-the-construct fails; execute-it works.** Defect 3 is indistinguishable from correct by
reading. ShellCheck 0.11.0 over reconstructed forms of the two shell defects returns only style nits
(SC2292, SC2250) and an unused-variable warning — nothing about either; the remaining three are
Python and outside its reach, so the honest figure is two of two, not zero of five. #776 reached the
same conclusion the same week from a different subsystem, with no shared code: `exec … 2>/dev/null`
silences the shell, and a trap's `exit "$?"` invents a status. The idiom behind defect 3 came from
written project memory recommending the form without `or die` — correct advice, written down, was
still a single point of failure, because the recipe had never had its own failure path executed.
+14 -2
View File
@@ -69,7 +69,18 @@ inventory shipped with were sitting in the gap:
5. **Non-`.sh` hooks** — the hook pattern is `*.sh` only. (`.yaml` workflows are no longer in this
gap: the caller scan matches `*.yml` and `*.yaml` since #806, because Gitea accepts both.)
6. **Transitive calls** — a script invoked only by another script, rather than by a workflow or
hook, is not discovered.
hook, is not discovered. Being outside this population is NOT an exemption from proof:
`testing.verification-code-needs-its-own-proof` (#796) puts such a script in the ordinary branch
whenever a proof test can drive it hermetically. A row here is not the route — the population is
derived from the call sites, so one is rejected as a phantom — but the manifest can still name
the checker, as `target`, with its proof test as `guard`. `scripts/mcp_smoke.py` is the worked
example: no row of its own, proven by `scripts/tests/test_mcp_smoke.py`, which carries the row.
That row is `GUARD` citing itself while its declared `target` is a different file — an ESTABLISHED
shape here, not a new one, and deliberately not counted: `test_review_verdict_vocabulary.py`
targets `scripts/check-review-verdict.sh` and `test_mutation_harness.py` targets
`mutation_harness_lib.py`. Worth noting only because the self-citation carve-out in
`test_every_proof_ref_points_at_a_row_marked_PROOF` is worded for a checker guarding a repo
invariant, whereas these guard another file.
7. **Non-`test_` modules under `scripts/tests/`** — the pattern is `test_*.py`, so `conftest.py`,
`mutation_harness_lib.py`, `mutation_manifest.py` and `tracked_files.py` are outside the
population and hold no rows. They are not guards (they assert nothing on their own), but the
@@ -211,6 +222,7 @@ recorded as unexamined rather than as cleared.
| `scripts/tests/test_guard_populations_derive_from_git.py` | the `script-tests` job, on a guard whose file population admits a file git does not track | GUARD | MUTATION | `test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file` |
| `scripts/tests/test_hook_fire_log.py` | the `script-tests` job, on a hook that stops reporting that it fired, or whose reporting changes what the harness sees | GUARD | MUTATION | `test_hook_fire_log.py::test_a_hook_that_LOSES_its_instrumentation_is_DETECTED` |
| `scripts/tests/test_jq_preflight.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_mcp_smoke.py` | the `script-tests` job, when the MCP smoke checker accepts a server it should refuse — an impostor identity, a reply carrying only an id, a missing expected tool, or an `initialize` pre-answered before the request (the `tools/list` twin is held by two mechanisms jointly and is covered behaviourally, not by the declared clause) | GUARD | MUTATION | `test_mcp_smoke.py::test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUESSABLE` |
| `scripts/tests/test_merge_consent_base_change.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_merge_consent_exemption.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_merge_consent_required_check.py` | the `script-tests` job | PROOF | NONE | — |
@@ -247,7 +259,7 @@ clause-provable and the entry is regraded.
## What the numbers say
42 guards, 6 tooling scripts, 21 proof files. **20 guards carry a mutation proof; 6 are
43 guards, 6 tooling scripts, 21 proof files. **21 guards carry a mutation proof; 6 are
behaviour-only; 16 have none.** These figures are asserted against the table by
`test_the_summary_counts_match_the_table` — they were wrong in the first draft (28/4/6/3/19 against
a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it,
+1 -1
View File
@@ -22,7 +22,7 @@ deliberately share a stage code, so read the message, not the number.
ACCEPTED LIMIT: a server that writes a gigabyte with no newline can still exhaust
memory before the timeout fires. Guarding that needs the very frame-capping reader
whose caps caused three defects in an earlier round, and the input here is our OWN
whose caps were themselves a defect in an earlier round, and the input here is our OWN
configured server on a developer machine — not an adversary. Stated rather than
silently unhandled.
+28
View File
@@ -310,6 +310,34 @@ MUTATIONS: tuple[Mutation, ...] = (
"one checkout, so the clause is unambiguous there; the trailing comment line is part of it "
"only to make the match unique within the file.",
),
Mutation(
guard="scripts/tests/test_mcp_smoke.py",
target="scripts/mcp_smoke.py",
clause=" id_init = secrets.randbelow(2**31 - 1000) + 1000",
replacement=" id_init = 1",
proof="test_mcp_smoke.py::test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUESSABLE",
granularity=CLAUSE,
expect="a pre-answered id was ACCEPTED",
why="The target is not the guard for the same structural reason as the harness entry above: "
"`scripts/mcp_smoke.py` is a checker reached only transitively, through "
"`scripts/check-local-lsp.sh`, so it holds no inventory row of its own and the clause has to "
"be disarmed in it directly (`testing.verification-code-needs-its-own-proof`). The clause is "
"the unguessable request id, which AT THE `initialize` STAGE is the only thing refusing a "
"server that answers before it is asked: the pending-registration cannot help there, because "
"that id is already in flight when the pre-answer arrives, which is why #793 round 5 replaced "
"the lock rather than tightening it. Disarmed, the stub's pre-answer is "
"accepted at `initialize` and the run dies one stage later at `tools/list`, so the proof "
"asserts the STAGE (rc 9 and the initialize diagnostic) rather than mere failure: the mutant "
"still fails, and a test checking only that would stay green over a real vulnerability. THE "
"`id_tools` TWIN IS DELIBERATELY NOT THIS CLAUSE, measured rather than assumed: replacing "
"`id_tools` alone leaves every case green, because the reader keeps only the reply matching "
"the id in flight, so a `tools/list` frame emitted while `initialize` is pending is dropped "
"whatever its id. Disarming that retention clause alone is green too. Only BOTH together "
"produce the false green (rc 0), so the tools stage is held by two mechanisms that mask each "
"other and neither is singly detectable — #685's shape. `id_init` is the one place a "
"pre-answer IS singly exploitable, which is why it is the declared clause; the tools stage is "
"covered behaviourally in the same file.",
),
)
+226
View File
@@ -0,0 +1,226 @@
"""`scripts/mcp_smoke.py` answers its own question correctly (ersatztv#796).
WHY THIS FILE EXISTS AND WHY IT IS NOT A ROW FOR `mcp_smoke.py`. That script is the checker that
carried three of #793's five false greens: it accepted any reply carrying the right id, its
message/line caps re-parsed an over-long line's suffix as a fresh message, and it registered a
pending id before sending so a server could pre-answer a predictable one. Each was reproduced with a
control at the time and then fixed, and NONE of those controls survived as a test — so nothing
re-ran them and a later edit could silently undo what they proved. `testing.verification-code-needs-
its-own-proof` is the rule that says a checker owes an executed proof; this is that proof.
`mcp_smoke.py` itself gets no `docs/guard-inventory.md` row: the inventory's population is derived
from workflow and hook call sites and this script is reached only transitively, through
`scripts/check-local-lsp.sh` (the transitive-calls scope limit, item 6), so a row for it is rejected as a phantom. THIS
file joins the population as an ordinary `scripts/tests/test_*.py` and carries the row, and the
manifest entry names it as `guard` with `scripts/mcp_smoke.py` as `target` — the same shape
`test_mutation_harness.py` uses for `mutation_harness_lib.py`.
HERMETIC BY CONSTRUCTION, which is what makes the proof possible at all. `mcp_smoke.py` takes its
config path and server name as POSITIONAL ARGUMENTS, so nothing here touches the gitignored
`.mcp.json` or starts a language server: every case below drives it against a synthetic config in a
tmpdir pointing at the stub in `_STUB`. "It needs a real server" exempts the INTEGRATION —
`check-local-lsp.sh` driving the actual csharp-lsp — never this protocol logic.
WHAT IS COVERED, stated as a list rather than as "every branch", because the honest figure is a
subset: the success path, and the refusals for a pre-answered `initialize` (9), a pre-answered
`tools/list` (10), an `initialize` reply with no `result` (12), a wrong `serverInfo.name` (13) and a
missing expected tool (14). `mcp_smoke.py` has many other `fail()` sites — among them the usage,
config, command-resolution, `--project` and spawn stages, an `initialize` that returns an error,
zero well-formed tools, and further malformed shapes — and none of those is driven here.
NOT TESTED, DELIBERATELY: the unbounded raw line buffer. `mcp_smoke.py` states that as an ACCEPTED
LIMIT, because the caps that would bound it were themselves defect 4. A test pushing toward
reintroducing them would drive the file back into the failure it already paid for.
"""
from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
SMOKE = REPO_ROOT / "scripts" / "mcp_smoke.py"
# The stub speaks just enough JSON-RPC to exercise each case. The two `preanswer` arms are the
# security cases: they emit replies for the LOW, GUESSABLE ids a predictable client would use,
# BEFORE the corresponding request arrives, and then go silent — so they can only be accepted by a
# client whose ids can be guessed. Each drops a MARKER file once it has emitted, because otherwise
# a stub that silently emitted NOTHING would satisfy the same assertion: "no answer" and "the
# pre-answer was refused" are the same observation from outside, and only the marker separates them.
_STUB = r"""
import json, sys, time
mode, marker = sys.argv[1], sys.argv[2]
def emit(obj):
sys.stdout.write(json.dumps(obj) + "\n")
sys.stdout.flush()
def mark():
with open(marker, "w") as fh:
fh.write("emitted\n")
def init_result(name="stub-mcp"):
return {"serverInfo": {"name": name, "version": "0.1"}, "capabilities": {}}
def tools_result(names):
return {"tools": [{"name": n} for n in names]}
if mode == "preanswer_init":
emit({"jsonrpc": "2.0", "id": 1, "result": init_result()})
mark()
time.sleep(30)
sys.exit(0)
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
msg = json.loads(line)
except ValueError:
continue
mid, method = msg.get("id"), msg.get("method")
if method == "initialize":
if mode == "impostor":
emit({"jsonrpc": "2.0", "id": mid, "result": init_result("impostor-mcp")})
elif mode == "no_result":
emit({"jsonrpc": "2.0", "id": mid})
else:
emit({"jsonrpc": "2.0", "id": mid, "result": init_result()})
if mode == "preanswer_tools":
# answer initialize honestly, then pre-answer the tools/list id we hope to guess
emit({"jsonrpc": "2.0", "id": 2, "result": tools_result(["alpha", "beta"])})
mark()
elif method == "tools/list":
if mode == "preanswer_tools":
continue # already "answered" it above, at a guessed id
names = ["alpha"] if mode == "missing_tool" else ["alpha", "beta"]
emit({"jsonrpc": "2.0", "id": mid, "result": tools_result(names)})
"""
def _run(tmp_path: Path, mode: str, *extra: str, timeout: int = 5):
"""Drive `mcp_smoke.py` against the stub in `mode`. Returns (rc, output, marker_path)."""
stub = tmp_path / "stub_server.py"
stub.write_text(_STUB)
marker = tmp_path / "emitted.marker"
cfg = tmp_path / "synthetic.mcp.json"
cfg.write_text(
json.dumps({"mcpServers": {"stub": {"command": sys.executable, "args": [str(stub), mode, str(marker)]}}})
)
proc = subprocess.run(
[sys.executable, str(SMOKE), str(cfg), "stub", str(timeout), *extra],
capture_output=True,
text=True,
timeout=timeout + 40,
)
return proc.returncode, proc.stdout + proc.stderr, marker
# ------------------------------------------------------------------------------------------------
# ANTI-VACUITY AS A DEPENDENCY, NOT A CONVENTION. Every case below except the positive control itself
# asserts a REFUSAL, and a harness that could not make the checker SUCCEED would report all of those
# green while proving nothing. A
# sibling test placed first does not establish that: `-k`, a node-id selection, or the mutation
# harness running one test by id all skip it. A fixture the rejection tests DEPEND on cannot be
# skipped that way.
# ------------------------------------------------------------------------------------------------
@pytest.fixture(scope="module")
def harness_is_live(tmp_path_factory) -> None:
rc, out, _ = _run(
tmp_path_factory.mktemp("positive_control"),
"good",
"--expect-server",
"stub-mcp",
"--expect-tool",
"alpha",
)
assert rc == 0 and "OK:" in out and "stub-mcp" in out, (
f"the positive control did not pass, so every refusal asserted in this file proves nothing "
f"— the stub or the config shape is broken, not the checker (rc={rc}):\n{out}"
)
def test_the_harness_can_make_the_checker_PASS(harness_is_live) -> None:
"""The positive control, surfaced as a test so it is visible in the report as well as depended on."""
def test_an_IMPOSTOR_server_is_rejected_on_identity(harness_is_live, tmp_path) -> None:
"""#793 defect 2: 'it answered' is not 'the configured server answered'."""
rc, out, _ = _run(tmp_path, "impostor", "--expect-server", "stub-mcp")
assert rc == 13, f"an impostor server was accepted (rc={rc}): {out}"
assert "wrong server" in out, out
def test_a_reply_carrying_only_an_ID_is_rejected_as_malformed(harness_is_live, tmp_path) -> None:
"""#793 defect 2, the other half: a bare `{"id": N}` is a reply, not an answer."""
rc, out, _ = _run(tmp_path, "no_result")
assert rc == 12, f"a body with no 'result' was accepted (rc={rc}): {out}"
assert "no 'result' object" in out, out
def test_a_MISSING_expected_tool_is_reported(harness_is_live, tmp_path) -> None:
rc, out, _ = _run(tmp_path, "missing_tool", "--expect-tool", "alpha", "--expect-tool", "beta")
assert rc == 14, f"a server missing an expected tool was accepted (rc={rc}): {out}"
assert "missing expected tool" in out, out
def test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUESSABLE(
harness_is_live, tmp_path
) -> None:
"""#793 defect 5, and the declared clause mutation for this guard.
The stub answers id 1 before reading anything, then goes silent. With random request ids the
pre-answer cannot match, so `initialize` times out and the run is refused at THAT stage. Replace
`id_init = secrets.randbelow(...)` with `id_init = 1` and the pre-answer is accepted: the run
gets past identity and dies one stage later, at `tools/list`. Asserting the STAGE rather than
merely 'it failed' is what makes the mutation visible — the mutant still fails, just not here.
The marker assertion is not decoration. Without it this test is satisfied by a stub that emits
NOTHING AT ALL, since a silent server also produces rc 9 — an absence standing in for the
refusal the test claims to observe.
"""
rc, out, marker = _run(tmp_path, "preanswer_init", "--expect-server", "stub-mcp")
assert marker.exists(), (
"the stub never emitted its pre-answer, so this case did not pose the attack it narrates: "
"rc 9 below would only mean the server said nothing."
)
assert rc == 9 and "no 'initialize' response" in out, (
f"a pre-answered id was ACCEPTED: the checker got past `initialize` against a server that "
f"replied before it was asked (rc={rc}). At THIS stage an unguessable id is the only thing "
f"closing that race — the pending-registration cannot help, because the id is already in "
f"flight when the pre-answer arrives.\n{out}"
)
def test_a_PRE_ANSWERED_tools_list_is_refused_by_TWO_mechanisms_TOGETHER(harness_is_live, tmp_path) -> None:
"""The `tools/list` twin of the case above — and the measurement that explains why `id_tools` is
NOT this guard's declared clause.
`mcp_smoke.py` draws two unguessable ids, and only `id_init` is singly load-bearing. Measured on
an isolated copy: replacing `id_tools` alone leaves every case here green, because the reader
retains only the reply matching the id currently in flight (`if rid != pending ... continue`), so
a `tools/list` frame emitted while `initialize` is still pending is dropped whatever its id.
Disarming that retention clause alone is likewise green, because the real `id_tools` is still
unguessable. Disarm BOTH and this case goes red at rc 0 — the checker accepts a tool list at an
id it never requested.
So the tools stage is held by two mechanisms that mask each other (#685's shape, benign here but
worth naming): neither is detectable alone, which is exactly why the declared clause is the
`initialize` id, where the pre-answer IS singly exploitable. This case is behavioural coverage
for the pair, not a second mutation proof.
"""
rc, out, marker = _run(tmp_path, "preanswer_tools", "--expect-server", "stub-mcp", "--expect-tool", "alpha")
assert marker.exists(), "the stub never emitted its pre-answered tools/list, so this case posed nothing"
assert rc == 10 and "no 'tools/list' response" in out, (
f"a pre-answered tools/list was ACCEPTED: the checker took a tool list at an id it never "
f"requested (rc={rc}). That needs BOTH the pending-registration and an unguessable "
f"`id_tools`; this went green with one of them gone.\n{out}"
)