ersatztv/scripts/tests/mutation_harness_lib.py

"""Machinery for the clause-level mutation harness (ersatztv#790).

`docs/guard-inventory.md` grades each guard's proof `MUTATION`, `BEHAVIOUR-ONLY` or `NONE`, and
`MUTATION` means "a clause-level mutation was executed and this named test was witnessed red". A
witnessing performed once, by hand, decays the moment anyone edits the guard, and a grade nothing
re-checks can simply be wrong.

This module turns each such row from an assertion into a check: apply the guard's **declared**
clause mutation to an isolated copy of the repo and require the row's own named test to go RED.

WHAT IS DELIBERATELY NOT DONE. The mutation is declared per guard in `mutation_manifest.py`, never
inferred. A harness that guessed which clause of a 90-line hook is *the* guard would manufacture
exactly the confident-but-empty coverage this exists to prevent — the reason
`testing.guard-ships-with-mutation-proof` rejects a generic runner. Guessing is also unnecessary:
most of the proof tests already name their clause in source (the BOM test's `= "efbbbf" ]; then`,
`UNSET_CLAUSE`, `prove-fix.sh`'s `if [ "$RC" -eq 0 ]; then`), and the manifest reuses that same
string rather than inventing a second one.

THE SANDBOX IS A REAL GIT REPOSITORY, not a directory of copied files. Several guards derive their
population from `git ls-files` and one drives `git worktree add`, so a plain copy would send them
down their degraded paths and every mutation would "redden" for a reason having nothing to do with
the clause. Its contents are the TRACKED files with WORKING-TREE content — `git ls-files -s`, not a
filesystem walk (`testing.guard-derives-population-from-source`, and the reason #778's guard was red
on every developer checkout and green in CI: `.husky/_/` is generated by `npm ci` and untracked).

Two index entries are not regular files and are handled explicitly rather than by an exception:
the `.claude/skills/jellyfin` symlink is recreated as a symlink (it dangles outside `~/ersatztv`,
which is inherent to the cross-repo symlink pattern and not this harness's problem), and the
`ErsatzTV-macOS` gitlink is SKIPPED — no guard reads the submodule, and materialising one would
cost a fetch per run.
"""

from __future__ import annotations

import os
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parents[2]

# Long enough that a slow shared runner is not mistaken for a hang, short enough that a genuinely
# stuck inner pytest fails the job rather than burning the whole CI budget. The full set of proof
# tests runs in ~7s locally.
PYTEST_TIMEOUT = 300

# Every git call here is local and confined to the sandbox; anything slower is stuck, not slow.
GIT_TIMEOUT = 120

# The pristine commit of each sandbox, held OUT OF THE REPOSITORY the proof tests drive. A ref inside
# it would be one more thing a proof can move: `git branch -f` fails on a checked-out branch, a
# global `init.defaultBranch` can collide with the name, and any `git update-ref`/`git checkout -B` a
# proof runs could retarget it. An object id kept here cannot be reached from inside the sandbox at
# all, and `git reset --hard <oid>` needs no ref to exist.
_BASELINES: dict[str, str] = {}


@dataclass(frozen=True)
class Mutation:
    """One declared clause mutation and the test that must notice it.

    `guard` is the `Guard` column of `docs/guard-inventory.md` — the thing being graded.
    `target` is the file actually edited. They are usually the same; where they differ, `why` says
    why, and `test_a_cross_file_mutation_states_why` requires it.
    `clause` must occur EXACTLY ONCE in `target`: a mutation that lands on an unintended second site
    proves something about a clause nobody declared.
    `expect` is a substring the FAILING run's output must contain, and it is what stops exit code 1
    from being the whole verdict. Pytest reports an ordinary exception the same way it reports a
    failed assertion, so a mutation that merely CRASHES the proof test — an emptied population
    reaching an `IndexError`, a syntax error, an unrelated parametrisation — would otherwise be
    accepted as "the guard noticed". Naming the diagnostic the mutation is supposed to produce makes
    each row's evidence specific: a red for a different reason fails here and has to be re-declared.

    `granularity` is `CLAUSE` or `DETECTOR`, and it is the honest half of this harness. #790 opened
    on the observation that neutering `pin_population_faults` wholesale is "coarser than disarming
    one clause at a time — coarse enough that a single surviving clause would not be noticed". That
    is true, and it is also not always avoidable: a detector that accumulates faults from several
    independent arms answers on ANY of them, so disarming one arm leaves its proof test green and
    the only mutation that reddens is the whole detector. Recording which grade each guard actually
    admits turns that from an unstated weakness into a measured property.

    A `DETECTOR` entry does not merely SAY a finer mutation was tried; it carries that mutation in
    `survived_clause`/`survived_replacement`, and `test_every_SURVIVING_clause_mutation_still_does`
    re-runs it and requires the proof test to stay GREEN. WHERE SUCH AN ENTRY EXISTS, the
    justification for the coarse grade is therefore executed on every run, exactly like the grade it
    justifies — a prose claim would decay the same way the hand-run witnessing this harness replaces
    did. No entry is graded `DETECTOR` since ersatztv#891, so that runner currently executes nothing;
    `test_the_DETECTOR_survivor_set_is_empty_ON_PURPOSE` asserts the emptiness so it stays a recorded
    decision rather than an unread skip.
    """

    CLAUSE = "CLAUSE"
    DETECTOR = "DETECTOR"

    guard: str
    target: str
    clause: str
    replacement: str
    proof: str
    granularity: str
    expect: str
    why: str
    survived_clause: str = ""
    survived_replacement: str = ""

    @property
    def node_id(self) -> str:
        """The inventory records proof refs as `file.py::test`; pytest wants a path."""
        return f"scripts/tests/{self.proof}"


@dataclass(frozen=True)
class Verdict:
    ok: bool
    reason: str


def _clean_env(**extra: str) -> dict[str, str]:
    """The environment every subprocess here runs in, with git's ambient state REMOVED.

    Exported `GIT_*` variables override `-C` and `cwd`. `GIT_DIR`, `GIT_WORK_TREE`,
    `GIT_INDEX_FILE`, `GIT_COMMON_DIR` and `GIT_OBJECT_DIRECTORY` each redirect part of a repository;
    `GIT_CONFIG_COUNT`/`GIT_CONFIG_KEY_n`/`GIT_CONFIG_VALUE_n` inject arbitrary settings, `core.worktree`
    among them. Any of those reaching this module's `git init`/`add`/`commit`/`reset --hard` points
    them at the REAL repository, and the "sandbox" would then write through the tree it exists to
    stay out of. A git hook exports several of them, and this suite runs from one.

    So this is a DENY-BY-DEFAULT boundary rather than a list of the variables anyone has thought of:
    every `GIT_*` is dropped and only the identity this module sets itself is put back. An enumeration
    of the dangerous ones shipped here once, covering three of them — a list is what this replaces.
    """
    env = {k: v for k, v in os.environ.items() if not k.startswith("GIT_")}
    env.update(extra)
    return env


def _git(cwd: Path, *args: str) -> subprocess.CompletedProcess:
    # `-c` rather than the ambient configuration, because the sandbox must not inherit the
    # developer's machine: a global `core.hooksPath` would fire this repo's husky hooks against a
    # throwaway tree, and `commit.gpgsign` would block the commit on a signing key CI does not have —
    # indefinitely, at a pinentry prompt, which no pytest timeout is watching.
    return subprocess.run(
        # `core.worktree` is pinned along with the rest: a proof that plants one in the sandbox's own
        # config would otherwise redirect `reset --hard` and `clean -qffdx` at a tree outside it.
        [
            "git",
            "-c",
            "core.hooksPath=/dev/null",
            "-c",
            "commit.gpgsign=false",
            "-c",
            f"core.worktree={cwd}",
            *args,
        ],
        cwd=str(cwd),
        check=True,
        capture_output=True,
        timeout=GIT_TIMEOUT,
        env=_clean_env(
            GIT_AUTHOR_NAME="mutation-harness",
            GIT_AUTHOR_EMAIL="harness@example.invalid",
            GIT_COMMITTER_NAME="mutation-harness",
            GIT_COMMITTER_EMAIL="harness@example.invalid",
        ),
    )


def build_sandbox(dest: Path, root: Path = REPO_ROOT) -> Path:
    """Materialise `root`'s tracked files at `dest` and make it a git repository."""
    entries = subprocess.run(
        ["git", "-C", str(root), "ls-files", "-s", "-z"],
        capture_output=True,
        check=True,
        timeout=GIT_TIMEOUT,
        env=_clean_env(),
    ).stdout.decode()

    copied = 0
    for entry in entries.split("\0"):
        if not entry:
            continue
        meta, path = entry.split("\t", 1)
        mode = meta.split()[0]
        if mode == "160000":  # gitlink — see the module docstring
            continue
        src = root / path
        dst = dest / path
        dst.parent.mkdir(parents=True, exist_ok=True)
        if src.is_symlink():
            os.symlink(os.readlink(src), dst)
        else:
            shutil.copy2(src, dst)
        copied += 1

    if copied == 0:
        raise RuntimeError(
            "the sandbox population is EMPTY — `git ls-files` returned nothing, so every mutation "
            "below would run against an empty tree and report success. Anti-vacuity, not paranoia."
        )

    _git(dest, "init", "-q", ".")
    # `-f` because some tracked files are also gitignored; without it they would be dropped from the
    # sandbox's index and a guard deriving its population from `git ls-files` would see less than the
    # real repo does.
    _git(dest, "add", "-A", "-f", ".")
    _git(dest, "commit", "-qm", "mutation-harness sandbox")
    _BASELINES[str(dest.resolve())] = _git(dest, "rev-parse", "HEAD").stdout.decode().strip()
    return dest


def reset_sandbox(sandbox: Path) -> None:
    """Return the sandbox to its committed state between mutations.

    The proof tests write into `tmp_path`, but a guard driven through its real entry point can leave
    artifacts in the tree it is pointed at, and one mutation's residue reaching the next would make
    the second result a function of the first's.
    """
    # RESET TO THE RECORDED BASELINE COMMIT, never to bare HEAD. `git reset --hard` with no argument
    # resets to whatever HEAD currently is — so a proof test that COMMITS inside the sandbox moves
    # HEAD onto a commit containing the mutant, and every later "reset" would then faithfully restore
    # it. The `finally` in `verify_mutation` puts the file back, but nothing would put HEAD back, and
    # the contamination would surface as an unrelated red several mutations later.
    #
    # `-ff` rather than `-f` because a single `-f` refuses to delete a nested git repository, which is
    # precisely what a proof driving `git init` or `git worktree add` into the sandbox leaves behind.
    baseline = _BASELINES.get(str(sandbox.resolve()))
    if baseline is None:
        raise RuntimeError(
            f"no recorded baseline for {sandbox} — it was not built by build_sandbox, so there is "
            "nothing to reset TO and a reset here would pin whatever state the tree is in now"
        )
    _git(sandbox, "reset", "-q", "--hard", baseline)
    _git(sandbox, "clean", "-qffdx")


def run_pytest(sandbox: Path, node_ids: list[str]) -> subprocess.CompletedProcess:
    return subprocess.run(
        [
            sys.executable,
            "-m",
            "pytest",
            "-q",
            "--no-header",
            "--tb=short",  # the assertion MESSAGE, which `expect` is matched against
            "-p",
            "no:cacheprovider",  # keeps `git status` in the sandbox clean between mutations
            *node_ids,
        ],
        cwd=str(sandbox),
        capture_output=True,
        text=True,
        env=_clean_env(PYTHONPATH="."),
        timeout=PYTEST_TIMEOUT,
    )


# Only exit code 1 means "a test ran and failed", and it is the only status accepted here. Everything
# else is rejected, which matters most for the two ways a proof ref goes stale — measured, because
# they are easy to get the wrong way round: with an explicit `file.py::function` node id, a missing
# FILE and a missing FUNCTION both exit 4 ("ERROR: not found"), while 5 needs a successful collection
# that selected nothing — a deselection. Reading either as a guard going red is how a harness reports
# coverage it does not have.
_PYTEST_RED_MEANINGS = {
    0: "the named test still PASSED with the clause mutated, so the clause is not load-bearing for it",
    2: "the inner pytest was interrupted",
    3: "the inner pytest hit an internal error",
    4: "the inner pytest could not resolve the node id — the proof ref names a file or a test that does not exist",
    5: "the inner pytest collected successfully but selected NOTHING — the proof ref was deselected",
}


def verify_mutation(sandbox: Path, mutation: Mutation) -> Verdict:
    """Apply one declared mutation in `sandbox` and require its named test to go red.

    The sandbox is left as it was found; callers still `reset_sandbox` between mutations because a
    driven guard can dirty the tree in ways this function does not know about.
    """
    target = sandbox / mutation.target
    if not target.is_file():
        return Verdict(False, f"the mutation target {mutation.target} does not exist in the sandbox")

    original = target.read_text(encoding="utf-8")
    occurrences = original.count(mutation.clause)
    if occurrences != 1:
        return Verdict(
            False,
            f"the declared clause occurs {occurrences} times in {mutation.target}, not once. "
            "RETARGET it rather than loosening the match — a clause that has moved, or that now "
            "matches a second site, means the recorded proof no longer points at what it claims to.",
        )

    mutated = original.replace(mutation.clause, mutation.replacement, 1)
    if mutated == original:
        return Verdict(False, "the replacement is identical to the clause, so nothing was mutated")

    target.write_text(mutated, encoding="utf-8")
    try:
        result = run_pytest(sandbox, [mutation.node_id])
    finally:
        target.write_text(original, encoding="utf-8")

    output = result.stdout + result.stderr
    if result.returncode != 1:
        meaning = _PYTEST_RED_MEANINGS.get(result.returncode, f"unexpected pytest exit code {result.returncode}")
        return Verdict(False, f"{meaning}\n--- inner pytest output ---\n{output[-3000:]}")
    # MATCHED AGAINST THE EXCEPTION OUTPUT ALONE, not the whole run. `--tb=short` echoes the failing
    # SOURCE as well as the message, and every one of these assertions carries its message as a
    # string literal a line or two above — so matching the full output would let a red at assertion A
    # be certified by assertion B's text merely being on screen. Pytest prefixes exception lines with
    # `E `, and that is the only part that reports what actually failed.
    diagnostic = "\n".join(line[2:] for line in output.splitlines() if line.startswith("E "))
    # This couples the harness to pytest's traceback FORMAT, and pytest is deliberately unpinned in
    # `script-tests`. The coupling is fail-CLOSED: a release that stopped prefixing exception lines
    # with `E ` would empty `diagnostic` and every row would fail here naming its own expectation,
    # which is loud and instantly diagnosable. The alternative — matching the whole run — fails
    # silently in the direction that certifies rows on the wrong red. Note the join: a multi-line
    # assertion message arrives as several `E ` lines, so an expectation must not span a newline.
    if mutation.expect not in diagnostic:
        return Verdict(
            False,
            f"the named test went red, but NOT with the declared diagnostic {mutation.expect!r}. A red "
            "for a reason other than the one this row records is not evidence about the clause — a "
            "crash, a syntax error or an unrelated parametrisation all look like this. Re-declare "
            f"`expect` once you know what the mutation now produces.\n--- inner pytest output ---\n"
            f"{output[-3000:]}",
        )
    return Verdict(True, "the named test went red under the declared mutation, with the declared diagnostic")