Files
ersatztv/scripts/tests/test_mcp_smoke.py
T
timothyandClaude Opus 5 cd9f24a67c fix(796): round 2 — the ruff red, and the twin that turned out not to be one
Cold re-review round 2. One finding was a genuine CI red I should have caught
locally; the rest are overclaims in prose. One of its findings dissolved under
measurement into something more useful.

**The red.** `ruff format --check` failed on `scripts/tests/test_mcp_smoke.py` —
the only file of the 47 tracked Python files that did. I never ran ruff on a file
I wrote, which is a standing rule and not a subtle one. Formatted; the whole
population is clean and `ruff check` passes.

**The twin that was not a twin.** The review flagged `id_tools` as an unproven
sibling clause: mutating it leaves every case green, while the inventory advertised
the guard as catching "an answer pre-answered before the request". Measured on
isolated copies, three ways:

  - `id_tools = 2` alone            -> all green
  - reader retention clause alone   -> all green
  - BOTH                            -> rc 0, the false green

So the tools stage is held by TWO mechanisms that mask each other — the reader keeps
only the reply matching the id in flight, so a `tools/list` frame emitted while
`initialize` is pending is dropped whatever its id. That is #685's shape, benign
here but worth naming. `id_init` is the one place a pre-answer is SINGLY
exploitable, which is why it stays the declared clause. The tools case is kept as
behavioural coverage with a docstring that states the measurement instead of the
mechanism I had assumed — the first version of it narrated a vulnerability that
does not exist, which is the defect this issue is about, one level in.

**The absence-assertion.** The review demonstrated that gutting the stub so it
emits NOTHING still satisfied the pre-answer test, because a silent server also
yields rc 9. The stub now drops a marker file once it has emitted, and the test
asserts it — so the case cannot pass without having posed the attack it narrates.

**Anti-vacuity is now a dependency, not a convention.** The positive control moved
into a module fixture the refusal tests depend on; a node-id selection or `-k` can
no longer skip it.

Prose corrections: coverage stated as six named refusal paths rather than "every
protocol branch" (the config/command/spawn stages are not driven); "the defect
history sits on the provable side" conceded — two of the five defects were in
`check-local-lsp.sh`, the file this record puts in the exempt branch; the wall-clock
timing dropped for the invariant; and the `mechanics` claim that one call site
carries all three traps narrowed to the two it actually carries. The `GUARD`
row citing itself is now recorded as a deliberate vocabulary stretch rather than
left to look routine.

Verification: 1104 passed, 2 skipped; ruff format+check clean over all 47 tracked
Python files; decisions-validate OK.

Refs: #796
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 20:32:17 +02:00

224 lines
11 KiB
Python

"""`scripts/mcp_smoke.py` answers its own question correctly (ersatztv#796).
WHY THIS FILE EXISTS AND WHY IT IS NOT A ROW FOR `mcp_smoke.py`. That script is the checker that
carried three of #793's five false greens: it accepted any reply carrying the right id, its
message/line caps re-parsed an over-long line's suffix as a fresh message, and it registered a
pending id before sending so a server could pre-answer a predictable one. Each was reproduced with a
control at the time and then fixed, and NONE of those controls survived as a test — so nothing
re-ran them and a later edit could silently undo what they proved. `testing.verification-code-needs-
its-own-proof` is the rule that says a checker owes an executed proof; this is that proof.
`mcp_smoke.py` itself gets no `docs/guard-inventory.md` row: the inventory's population is derived
from workflow and hook call sites and this script is reached only transitively, through
`scripts/check-local-lsp.sh` (scope limit item 6), so a row for it is rejected as a phantom. THIS
file joins the population as an ordinary `scripts/tests/test_*.py` and carries the row, and the
manifest entry names it as `guard` with `scripts/mcp_smoke.py` as `target` — the same shape
`test_mutation_harness.py` uses for `mutation_harness_lib.py`.
HERMETIC BY CONSTRUCTION, which is what makes the proof possible at all. `mcp_smoke.py` takes its
config path and server name as POSITIONAL ARGUMENTS, so nothing here touches the gitignored
`.mcp.json` or starts a language server: every case below drives it against a synthetic config in a
tmpdir pointing at the stub in `_STUB`. "It needs a real server" exempts the INTEGRATION —
`check-local-lsp.sh` driving the actual csharp-lsp — never this protocol logic.
WHAT IS COVERED, stated as a list rather than as "every branch", because the honest figure is a
subset: the success path, and the refusals for a pre-answered `initialize` (9), a pre-answered
`tools/list` (10), an `initialize` reply with no `result` (12), a wrong `serverInfo.name` (13) and a
missing expected tool (14). `mcp_smoke.py` has other `fail()` sites — the config, command-resolution
and spawn stages, and several further malformed shapes — and they are NOT driven here.
NOT TESTED, DELIBERATELY: the unbounded raw line buffer. `mcp_smoke.py` states that as an ACCEPTED
LIMIT, because the caps that would bound it were themselves defect 4. A test pushing toward
reintroducing them would drive the file back into the failure it already paid for.
"""
from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
SMOKE = REPO_ROOT / "scripts" / "mcp_smoke.py"
# The stub speaks just enough JSON-RPC to exercise each case. The two `preanswer` arms are the
# security cases: they emit replies for the LOW, GUESSABLE ids a predictable client would use,
# BEFORE the corresponding request arrives, and then go silent — so they can only be accepted by a
# client whose ids can be guessed. Each drops a MARKER file once it has emitted, because otherwise
# a stub that silently emitted NOTHING would satisfy the same assertion: "no answer" and "the
# pre-answer was refused" are the same observation from outside, and only the marker separates them.
_STUB = r"""
import json, sys, time
mode, marker = sys.argv[1], sys.argv[2]
def emit(obj):
sys.stdout.write(json.dumps(obj) + "\n")
sys.stdout.flush()
def mark():
with open(marker, "w") as fh:
fh.write("emitted\n")
def init_result(name="stub-mcp"):
return {"serverInfo": {"name": name, "version": "0.1"}, "capabilities": {}}
def tools_result(names):
return {"tools": [{"name": n} for n in names]}
if mode == "preanswer_init":
emit({"jsonrpc": "2.0", "id": 1, "result": init_result()})
mark()
time.sleep(30)
sys.exit(0)
for line in sys.stdin:
line = line.strip()
if not line:
continue
try:
msg = json.loads(line)
except ValueError:
continue
mid, method = msg.get("id"), msg.get("method")
if method == "initialize":
if mode == "impostor":
emit({"jsonrpc": "2.0", "id": mid, "result": init_result("impostor-mcp")})
elif mode == "no_result":
emit({"jsonrpc": "2.0", "id": mid})
else:
emit({"jsonrpc": "2.0", "id": mid, "result": init_result()})
if mode == "preanswer_tools":
# answer initialize honestly, then pre-answer the tools/list id we hope to guess
emit({"jsonrpc": "2.0", "id": 2, "result": tools_result(["alpha", "beta"])})
mark()
elif method == "tools/list":
if mode == "preanswer_tools":
continue # already "answered" it above, at a guessed id
names = ["alpha"] if mode == "missing_tool" else ["alpha", "beta"]
emit({"jsonrpc": "2.0", "id": mid, "result": tools_result(names)})
"""
def _run(tmp_path: Path, mode: str, *extra: str, timeout: int = 5):
"""Drive `mcp_smoke.py` against the stub in `mode`. Returns (rc, output, marker_path)."""
stub = tmp_path / "stub_server.py"
stub.write_text(_STUB)
marker = tmp_path / "emitted.marker"
cfg = tmp_path / "synthetic.mcp.json"
cfg.write_text(
json.dumps({"mcpServers": {"stub": {"command": sys.executable, "args": [str(stub), mode, str(marker)]}}})
)
proc = subprocess.run(
[sys.executable, str(SMOKE), str(cfg), "stub", str(timeout), *extra],
capture_output=True,
text=True,
timeout=timeout + 40,
)
return proc.returncode, proc.stdout + proc.stderr, marker
# ------------------------------------------------------------------------------------------------
# ANTI-VACUITY AS A DEPENDENCY, NOT A CONVENTION. Every case below asserts a REFUSAL, and a harness
# that could not make the checker SUCCEED would report all of them green while proving nothing. A
# sibling test placed first does not establish that: `-k`, a node-id selection, or the mutation
# harness running one test by id all skip it. A fixture the rejection tests DEPEND on cannot be
# skipped that way.
# ------------------------------------------------------------------------------------------------
@pytest.fixture(scope="module")
def harness_is_live(tmp_path_factory) -> None:
rc, out, _ = _run(
tmp_path_factory.mktemp("positive_control"),
"good",
"--expect-server",
"stub-mcp",
"--expect-tool",
"alpha",
)
assert rc == 0 and "OK:" in out and "stub-mcp" in out, (
f"the positive control did not pass, so every refusal asserted in this file proves nothing "
f"— the stub or the config shape is broken, not the checker (rc={rc}):\n{out}"
)
def test_the_harness_can_make_the_checker_PASS(harness_is_live) -> None:
"""The positive control, surfaced as a test so it is visible in the report as well as depended on."""
def test_an_IMPOSTOR_server_is_rejected_on_identity(harness_is_live, tmp_path) -> None:
"""#793 defect 2: 'it answered' is not 'the configured server answered'."""
rc, out, _ = _run(tmp_path, "impostor", "--expect-server", "stub-mcp")
assert rc == 13, f"an impostor server was accepted (rc={rc}): {out}"
assert "wrong server" in out, out
def test_a_reply_carrying_only_an_ID_is_rejected_as_malformed(harness_is_live, tmp_path) -> None:
"""#793 defect 2, the other half: a bare `{"id": N}` is a reply, not an answer."""
rc, out, _ = _run(tmp_path, "no_result")
assert rc == 12, f"a body with no 'result' was accepted (rc={rc}): {out}"
assert "no 'result' object" in out, out
def test_a_MISSING_expected_tool_is_reported(harness_is_live, tmp_path) -> None:
rc, out, _ = _run(tmp_path, "missing_tool", "--expect-tool", "alpha", "--expect-tool", "beta")
assert rc == 14, f"a server missing an expected tool was accepted (rc={rc}): {out}"
assert "missing expected tool" in out, out
def test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUESSABLE(
harness_is_live, tmp_path
) -> None:
"""#793 defect 5, and the declared clause mutation for this guard.
The stub answers id 1 before reading anything, then goes silent. With random request ids the
pre-answer cannot match, so `initialize` times out and the run is refused at THAT stage. Replace
`id_init = secrets.randbelow(...)` with `id_init = 1` and the pre-answer is accepted: the run
gets past identity and dies one stage later, at `tools/list`. Asserting the STAGE rather than
merely 'it failed' is what makes the mutation visible — the mutant still fails, just not here.
The marker assertion is not decoration. Without it this test is satisfied by a stub that emits
NOTHING AT ALL, since a silent server also produces rc 9 — an absence standing in for the
refusal the test claims to observe.
"""
rc, out, marker = _run(tmp_path, "preanswer_init", "--expect-server", "stub-mcp")
assert marker.exists(), (
"the stub never emitted its pre-answer, so this case did not pose the attack it narrates: "
"rc 9 below would only mean the server said nothing."
)
assert rc == 9 and "no 'initialize' response" in out, (
f"a pre-answered id was ACCEPTED: the checker got past `initialize` against a server that "
f"replied before it was asked (rc={rc}). Unguessable request ids are the only thing closing "
f"that race — a lock cannot reject a frame emitted before the request.\n{out}"
)
def test_a_PRE_ANSWERED_tools_list_is_refused_by_TWO_mechanisms_TOGETHER(harness_is_live, tmp_path) -> None:
"""The `tools/list` twin of the case above — and the measurement that explains why `id_tools` is
NOT this guard's declared clause.
`mcp_smoke.py` draws two unguessable ids, and only `id_init` is singly load-bearing. Measured on
an isolated copy: replacing `id_tools` alone leaves every case here green, because the reader
retains only the reply matching the id currently in flight (`if rid != pending ... continue`), so
a `tools/list` frame emitted while `initialize` is still pending is dropped whatever its id.
Disarming that retention clause alone is likewise green, because the real `id_tools` is still
unguessable. Disarm BOTH and this case goes red at rc 0 — the checker accepts a tool list at an
id it never requested.
So the tools stage is held by two mechanisms that mask each other (#685's shape, benign here but
worth naming): neither is detectable alone, which is exactly why the declared clause is the
`initialize` id, where the pre-answer IS singly exploitable. This case is behavioural coverage
for the pair, not a second mutation proof.
"""
rc, out, marker = _run(tmp_path, "preanswer_tools", "--expect-server", "stub-mcp", "--expect-tool", "alpha")
assert marker.exists(), "the stub never emitted its pre-answered tools/list, so this case posed nothing"
assert rc == 10 and "no 'tools/list' response" in out, (
f"a pre-answered tools/list was ACCEPTED: the checker took a tool list at an id it never "
f"requested (rc={rc}). That needs BOTH the pending-registration and an unguessable "
f"`id_tools`; this went green with one of them gone.\n{out}"
)