Timothy asked why the kickoff handoff doc stores historical narrative when
it should be instructions. It shouldn't — its own lore section is chartered
as "STANDING workflow/orchestration rules only" with the why belonging in
docs/decisions.md. But an inventory of every bullet against the decision
corpus inverted the premise: only ~8 of ~38 were actually covered. 19 had
no record anywhere and 11 more were half-covered, so that single file was
the ONLY copy of the mandatory review rubric, the whole CI-triage
vocabulary, the build-concurrency policy, the H12 session-end audit, and
the plumbing-merge recipe. Pruning first would have destroyed them.
So the records come first. New topic file docs/decisions/workflow-process.md
carries 32 records (ci.*, process.*, testing.*) covering every NONE and
PARTIAL the inventory found, including the Gitea `?milestones=` no-op bug
whose only copy was the archived selector section this prune deletes.
Only then the prune: HARD CONSTRAINTS and the lore section become one- or
two-line rules, each citing the decision key that holds its evidence, and
the 40-line "Archived — do not follow" section is gone. 636 -> 353 lines,
with every cited key verified to resolve against the corpus.
The aggregate corpus budget is re-baselined 4800 -> 5600 with the reason in
the code: the corpus grew because knowledge MOVED into it, which is the
system working, not drift.
refs #542
A session was handed docs/handoffs/chicorytv-issue-queue.md pasted out of
/Users/timothy/ersatztv while that tree was 81 commits behind, so it still
described the queue protocol #520 retired the day before (read tracker
command was ever run against that tree, so every existing "never read its
HEAD" guard was irrelevant: a stale checkout serves stale FILES, and docs
are what a kickoff depends on. Nothing broke only because selection went
through scripts/select-queue.sh.
The lore bullet on that tree already prescribed the shape of the fix for
its earlier failure modes — "a design flaw, not a discipline failure; a
check does not stay true" — so this removes the stale condition instead of
adding another check.
scripts/refresh-shared-checkout.sh fast-forwards the tree to origin/main
and reinstalls web/node_modules when the lockfile moved. It is deliberately
timid: it refuses and changes nothing when the tree is not on main, is
dirty, is ahead, or is mid-rebase/merge, and it never switches branches,
stashes or discards. A NO-OP is a normal outcome.
Uses npm ci rather than npm install — the first version used install,
which rewrote package-lock.json and left the tree dirty, i.e. the exact
state the next run refuses on, so it would have disabled itself after one
use. Asserts the tree is clean at exit.
refs #541
- Exclude docs/decisions/retrieval-eval.md from active decision parsing
(_NON_DECISION_FILES); its `## N.` eval-question headings were being
miscounted as 7 legacy-unmigrated records.
- Add decisions_lib.metadata_line_count() + a decisions_validate guard
that fails a record with more than one `key:` metadata line, so a
stacked-metadata-block migration bug (which the parser silently
tolerated by reading only the first block) can't recur unnoticed.
TDD: test_duplicate_metadata_block_fails / test_single_metadata_block_passes.
- De-duplicate the 6 docs/decisions.md records left with two stacked
metadata blocks (scan.getoraddfolder-db-lookup #488,
scan.musicvideo-reconciliation #494, scan.jellyfin-mixed-content-library
#489, iptv.logo-drives-bug-preset #67, ffmpeg.qsv-decode-encode-split
#498, ci.small-lane-git-only server-management#639), merging the union
of Signals/paths/issues/Mechanics from both blocks and keeping the
richer Rule wording; rationale prose untouched.
- Fill in the deferred Q6b row in docs/decisions/retrieval-eval.md now
that startup.parallel-orientation is active in docs/decisions.md,
scoring it as a real active-vs-superseded question against the
archived docs.queue-state-gitea-tracker.
- Regenerate docs/decisions/README.md via build_decisions_catalog.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_anchor() mapped each space/hyphen to its own '-' without collapsing runs,
so the standard heading separator " — " (space, em-dash, space) produced a
double hyphen in every generated anchor. Since nearly every real decision
record heading uses that separator, this made the catalog emit a dead link
for essentially every row. Fix: after building the char list, collapse
consecutive '-' into one and strip leading/trailing '-' via re.sub, matching
how Goldmark/GitHub/Gitea sluggers behave.
Also fixed main() writing an extra trailing newline (want already ends in
"\n", then "+ \n" appended a second one) so docs/decisions/README.md now
ends with exactly one trailing newline; --check still compares via .strip().
Added test_anchor_collapses_em_dash_and_keeps_underscore to pin the anchor
behavior against the reported iptv.base_url case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/build_decisions_catalog.py, which renders docs/decisions/README.md
as a compact table of only 'active' decision records (sorted by key), and its
test scripts/tests/test_build_catalog.py. Supports --check for CI drift
detection. No decision records are migrated yet, so the generated catalog is
currently empty (banner + header only) — expected at this stage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/decisions_validate.py: lifecycle invariant checks (unique
active key, key format, reciprocal supersession, removed-without-archive,
rationale-rewrite-without-token, catalog staleness, corpus budget) plus
git-diff helpers for merge-base-based CI checks. Deviates from the task
brief in one spot: REQUIRED_META narrowed to (key, status) — the brief's
(key, status, since, supersedes, superseded_by) makes its own
test_clean_corpus_passes fail, since since/supersedes/superseded_by
default to None on bare Record() instances built without going through
the markdown parser.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/decisions_lib.py, the shared parser for ErsatzTV decision
records (docs/decisions.md + docs/decisions/*.md). Parses H2 sections
into Record dataclasses, distinguishing migrated records (visible
metadata block: key/status/since/supersedes/superseded-by + Rule/
Signals/Mechanics) from legacy-unmigrated ones with no metadata line.
scripts/ is now an importable package (scripts/__init__.py,
scripts/tests/__init__.py) so later tools can `import scripts.decisions_lib`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test-job memory instrument (#411) reported memory.peak — the high-water mark of
memory.current, which charges reclaimable page cache to the cgroup. A build does heavy
NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak, and page cache is reclaimed
under a tighter cap rather than OOM-killed. Sizing a per-job cap (server-management#604) off
memory.peak therefore inverts the decision. The OOM-forcing quantity is peak anon, which the
kernel exposes no counter for and which the end-of-job split misses (a job that peaks
mid-dotnet-test then frees reports a low anon).
New scripts/ci-peak-anon.sh: a `start` step (before Build/Test/Coverage) launches a detached
background sampler tracking the high-water mark of cgroup anon; a `report` step (last) stops it
and prints the sampled peak anon as the headline, keeping memory.peak + end-of-job split as a
cache-inflated ceiling and reference. Both continue-on-error + fail-open so they never redden a
build. Validated on bumblebee: survives step-boundary re-execs, catches a transient 2.5 GiB
anon spike the snapshot reports as 0, stops cleanly on kill, degrades gracefully.
Compiler-server A/B (swap-off, sampled peak-anon, n=2 interleaved): OFF (CI config) ~5.84 GiB
consistent; ON (defaults) 6.3-7.6 GiB, always higher, + a ~3 GiB resident VBCSCompiler.
Disabling the servers is worth it, but OFF sits right at 6 GiB for the build phase alone and the
test job adds test+coverage, so #406's "budget loosens well under 6 GiB" premise is not
supported. Size the cap off the live test-job sampler.
Docs: ci-cd.md instrument section rewritten (peak-anon headline + A/B table + premise verdict);
decisions.md entry added. No .cs touched.
fixes#412
Adds "Flow C" to scripts/e2e-functional.sh, the last deferred lock-contention flow from #363.
A playout build is enqueued onto the single-consumer WorkerService channel and the trigger
returns before BuildPlayoutHandler acquires the lock, so an accepted trigger does not prove the
lock is held. Flow C makes it deterministic: seed a Classic Flood schedule over a few short
ffmpeg episodes, crank PlayoutDaysToBuild=5 (~43k items ~= ~1s build), then POLL GET
/playouts/{id} until isLocked:true before firing. Asserts PUT /playouts/{id} -> 409, reset ->
409, and the list-projection isLocked:true while locked; then isLocked:false + PUT -> 200 after
the build (proving the 409 is lock-specific). Each racing assertion is guarded so a build that
finishes mid-flight degrades to an advisory skip, never a false red; the whole flow self-skips
without ffmpeg or if the build is never observed locked.
Sized by measurement on a fresh instance -- going wider is counter-productive (a 777k-item build
saturates the single worker with post-build gap/overlap jobs). Verified green across 6
fresh-instance runs; cold adversarial review MERGEABLE.
Docs: docs/e2e-local.md + docs/ci-cd.md updated to describe Flow C and drop it from the
"deferred" lists.
fixes#444
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kickoff dispatches a cheap model to rank the backlog, and the lore then made the
orchestrator re-derive its mechanical claims (deps, milestone tiering, priority order,
in-progress state) because a small model kept getting them wrong. This pays that tax down:
the mechanical checks are now CODE — GET /dependencies exclusion, LOCAL
.milestone.state/review/priority tiering (never the no-op ?milestones= filter),
(tier,priority,issue#) ordering, in-progress/parked/PR exclusion — leaving only two
JUDGMENT flags (CLAIM?, UMBRELLA?) for a human/model to resolve.
- scripts/select-queue.sh: fail-open on no-creds/unreachable; ~1.3s; shellcheck clean;
UMBRELLA? computed in the jq pass (no per-candidate body fetch).
- handoff kickoff: run the script FIRST; trust its deps/tiering/ordering, recheck only flags.
- handoff lore: the three 'cheap selector unreliable → re-derive by hand' bullets kept as the
EVIDENCE for why the script exists; the prescription is redirected to 'run the script'.
Operator-requested this session: 'rather than have the lore make us redo the selector's work,
improve the selector.'
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- check seed_library_path succeeded (print lastrowid) so a silent seeding
failure surfaces as a FAIL instead of degrading Flow A to an advisory skip
with no diagnostic (Medium)
- anchor the scan-status match to '"libraryId":2,' so it can't substring-match
"libraryId":20/23 if the suite ever creates more libraries (Low)
- drop the no-op ?deep=true (local scans always ForceScan; deep only affects
Plex/Jellyfin/Emby) + note why (Low)
- soften "guaranteed 409" for the scan flow to note the tiny residual TOCTOU
gap the multi-second scan covers; Flow B stays race-free by construction (Low)
- correct the "WAL tolerates a second writer" wording to the real reason (the
busy-timeout retry serializes the writer) in the script + both docs (Nit)
- use TEST-NET-1 192.0.2.1 (RFC 5737) instead of RFC1918 10.255.255.1 for the
non-routable Jellyfin address (Nit)
Re-verified: fresh-instance harness runs green (38/38), lock section
deterministic. Functional E2E CI job already green on the prior head.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends scripts/e2e-functional.sh with the two IEntityLocker 409 contracts the
first cut (ersatztv#299) deferred as "racy", made DETERMINISTIC by firing the
racing request only once the lock is provably held (no sleep-and-hope):
- library-scan "already scanning" 409: seed ~60 tiny ffmpeg clips into the
built-in Shows library so the scanner subprocess runs a few seconds, poll
GET /libraries/scan-status until the library is active (that window is a
strict subset of the scan lock's held window — StartScan after LockLibrary,
EndScan before UnlockLibrary), then a second POST .../scan is a guaranteed
409. Self-skips (advisory) when ffmpeg is absent.
- external-collections "already scanning" 409: seed a Jellyfin media-source row
pointing at a non-routable address so the background sync hangs and the
per-family lock stays held; the lock is taken synchronously before the 202,
so the 202 proves it held. collections-scan-status corroborates; unknown
source 404.
Seeding uses python3's stdlib sqlite3 (already a harness dep) to insert rows the
API can't create (LibraryPath, media-source); WAL mode tolerates the second
writer. No new CI step/dependency — ffmpeg ships in the toolchain image.
Verified: 4/4 fresh-instance runs green (38/38), lock section deterministic.
Still deferred to #363 follow-ups: the playout-build lock 409 + isLocked
projection (#215) and the UI-interactive Playwright flows.
Docs updated same PR: docs/e2e-local.md, docs/ci-cd.md, the functional-e2e
job comment in .gitea/workflows/docker-build.yml.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test/migrations jobs check out fetch-depth:1. A shallow clone has NO
origin/<base> tracking ref and no merge-base, so the three-dot
origin/main...HEAD errored -> empty diff -> docs_only=false -> EVERY docs-only PR
silently ran the full matrix (safe but the skip never fired). git fetch always
writes FETCH_HEAD, which resolves in a shallow clone; diff against it with a
two-dot tree diff (no merge-base). Confirmed in a real shallow file:// clone:
origin/main did NOT resolve and three-dot errored, while FETCH_HEAD two-dot
correctly returned the docs file. api-docs/format were unaffected only because
they use fetch-depth:0.
Refs #416
The changed-set git diff had rename detection on by default, so a code->docs
rename (Foo.cs -> docs/Foo.md) showed only the destination and was misclassified
as docs-only, skipping required tests on a code change. --no-renames surfaces the
source deletion -> full matrix. Empirically verified. This is the cold-review
MEDIUM; it was applied in the working tree but never committed before the first
push (index/worktree mismatch) — committing it now.
Refs #416
Docs-only changes (docs/** or *.md) ran the full docker-build matrix (~9 min).
Each heavy job (test, migrations, functional-e2e, build) now runs
scripts/ci-detect-docs-only.sh as its first post-checkout step and gates every
real step on docs_only!='true'. The jobs still RUN and report success in
seconds, so the two required contexts keep reporting — a docs-only PR stays
mergeable (never an if:-skipped required job; Gitea 1.25.4 reports if-skip as
'skipped', verified with a throwaway probe PR). build skips its image steps on a
docs-only push to main; tag builds force docs_only=false. Detection uses
--no-renames so a code->docs rename can never be misclassified as docs-only.
Refs #416
Re-review of the fix commits came back MERGEABLE with three findings worth acting on.
MEDIUM -- the PR documented a standing rule ("--memory without --memory-swap silently
grants 2x in swap") and then didn't apply it to the two sites this repo owns:
docker-build.yml's smoke container and scripts/migration-smoke.sh, both `--memory 2g` with
no --memory-swap. Pre-existing rather than a regression, but a rule you don't follow in
your own repo isn't a rule. The migration-smoke one matters most: it runs on the PROD host
in the release path, so a runaway migration should die against its cap rather than quietly
swap out the box serving media.
LOW -- and this is the important one: the docs leaned "peak 8305 MiB is probably mostly
reclaimable cache". An independent probe (full solution build, same CI image, shared
compilation off) measured peak 9457 MiB / anon 7134 MiB / file 421 MiB. ANON DOMINATED.
Having verified the *mechanism* (peak overstates because it counts page cache), I guessed
the *magnitude* in the direction I preferred -- the exact failure this entry criticises,
committed inside the entry criticising it. Corrected in ci-cd.md, decisions.md and on
server-management#604 (where the previous comment could have led to an unsafe 6g cap).
Consequences now recorded honestly: a 6g cap looks UNSAFE, #570's "6g proved too tight" is
the rule not an outlier, and #406's premise ("if this brings peak RSS well under 6 GiB the
whole budget loosens") is looking DEAD -- the 7134 MiB anon was measured with shared
compilation already off. The switches remain right; the looser budget they were meant to
buy does not follow.
NIT -- dropped the unverified claim that this also disables the Razor build server. The
UseRazorBuildServer -> UseSharedCompilation fallback is .NET 5-era; Razor has been an
in-process source generator since .NET 6, so there is likely no separate server to disable
on .NET 10. Unverified, zero impact, so it has no business in a doc arguing for
measurement over assumption.
[decisions-edit]: the touched docs/decisions.md lines were added by this PR's own earlier
commits, not settled entries on main -- net vs origin/main remains a pure insertion (0
deletions, verified). Also the sanctioned reason: the entry was factually wrong (see LOW).
Verified: both workflows parse; migration-smoke.sh passes bash -n; the parsed mysql option
string is `--memory=2g --memory-swap=2g --cpus=2`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codify the manual live-E2E curl flows sessions have been re-running by hand
into a CI regression net: a new `functional-e2e` job boots the app from source
(via scripts/e2e-local.sh, parameterized for Release) and drives scripts/
e2e-functional.sh against it.
First-cut contracts (all curl-only, deterministic, no seeded media/ffmpeg/browser):
- legacy->SPA redirect sweep + the /api,/artwork never-redirect exemption
- auth/CSRF/security-stamp flow (setup-claim, read-gate, CSRF, login, logout+revoke)
- library-scan status contract (404/202/scan-status)
- optimistic-concurrency If-Match/412 round-trip
Advisory by design (separate job, not a `build` dependency, not a required
check) so a functional-E2E flake can't block the unit-test gate; promote once
proven, mirroring the migrations-job rollout. SQLite default -> no DB service.
Deferred to #299 follow-ups (need scanner+seeded media or a browser to be
deterministic): the racy 409 re-trigger, playout-build lock 409, Playwright UI
flows.
Assertions verified 30/30 green against a real Release-built instance; caught
/artwork/* returning 400 (not the 404 a static read suggested).
Docs updated same PR: docs/ci-cd.md (new job), docs/e2e-local.md (harness),
docs/decisions.md (append-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cold review of PR #331:
- Move the ZAP-container reap from an inline post-scan line into cleanup() (the
EXIT trap) so a SIGINT/timeout kill mid-scan can't leave it running.
- semgrep absent now prints a stderr WARN instead of silently skipping the SAST
pass (no false confidence for a release gate).
refs #314
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/security-scan.sh: boots a THROWAWAY container from the image under test
(fresh config volume; never the deployed prod/test container — the active scan
attacks write endpoints), reads its machine key, and runs an authenticated OWASP
ZAP api-scan that imports /openapi/v1.json (all 160 /api/v1 ops) and injects
X-Api-Key on every request via a replacer rule so it reaches the
[RequiresAuthentication] + RequireKeyForReads surface — then a semgrep SAST
cross-check. Wrapped in `timeout` because zap-api-scan can hang in post-scan
cleanup after the report is already written.
docs/ci-cd.md: new 'Security scanning' section (out-of-ecosystem black-box gate,
run on the docker host per-release like migration-smoke, not a per-PR CI job) +
the Microsoft.OpenApi 2.7.5 pin note in dependency management.
refs #314#197
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new api-docs gate (#303 H4/H5) runs update-openapi.sh after only `dotnet
restore`. The script's `dotnet build -t:GenerateOpenApiDocuments` does not compile
the project (OpenApiGenerateDocumentsOnBuild=false), so in a clean tree getdocument
fails with 'ErsatzTV.deps.json does not exist' (exit 129). This PR is the first to
change the /api surface and thus the first to exercise the gate's regen path,
exposing the latent bug. Add a full `dotnet build` before the doc-gen target so the
assembly + deps.json exist. Verified: clean-tree regen now succeeds with zero diff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review (MERGEABLE-WITH-NITS) findings:
- Remove the broad FAIL_RE log-scan (matched benign ErsatzTV startup noise — library
scans against absent media mounts, EF connection retries — risking a false-FAIL that
blocks a good release). It was also redundant: a failed migration faults the
BackgroundService -> default StopHost -> container exit, which the early-exit check
already catches reliably (per the reviewer's own analysis). Migration failure is now
detected by early container exit + timeout + the post-boot serve probe.
- Validate --timeout is a positive integer (was: '--timeout abc' -> 0 -> instant false-FAIL).
- Log the resolved image id after (attempted) pull, so a pull-failure that rehearses a
stale local :latest is visible to the operator.
Re-validated live on bumblebee: :latest vs the 283MB prod-copy -> migrations clean, PASS,
image digest logged, no leftover temp dir/container. shellcheck + bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI's migrations job only proves a migration is well-formed against a fresh,
empty DB. It never exercises the migration — or ErsatzTV's startup data steps
(DatabaseMigratorService -> DbInitializer + PopulatePathHashes over the real
MediaFile table) — against the accumulated prod SQLite, so a migration green on
a fresh DB can still fail/corrupt on prod, found only mid-deploy.
scripts/migration-smoke.sh rehearses it on a THROWAWAY copy of the latest prod
backup: boots the new image against the copy, gates PASS on the "Done applying
database migrations" log line (the migrator is a BackgroundService running
concurrently with Kestrel, so HTTP readiness alone doesn't prove migrations
finished), FAILs on early container exit / migration exception / timeout / not
serving afterwards. Always operates on a copy, never the live DB; tears down its
container + temp dir (incl. the container's root-owned config files) on exit.
Validated live 2026-07-12: :latest vs a copy of the 283MB prod backup ->
migrations applied cleanly, app booted+served, temp dir removed.
Home split: this repo owns the script + docs; wiring it into the Komodo
pre-deploy step is server-management#589 (cross-repo). Docs: docs/ci-cd.md
(Migration integrity), docs/decisions.md (new entry, pure insertion).
fixes#315
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/issue-qualification-audit.sh: lists OPEN issues missing a `priority:` label
(the #237 ranking keys off priority:/gate labels, so an unlabeled issue is invisible
to it). "Fully qualified" = has a priority: label; gate-vs-backlog derives from the
review label / milestone, and a milestone is NOT required (backlog is unmilestoned).
Advisory (exit 1 if any unqualified); fail-open without Gitea creds. Wired into the
kickoff session-end protocol + a lore bullet. Tested live (flagged 2) + no-creds no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only run generate-endpoint-index.py if the GenerateOpenApiDocuments build
succeeded, so a failed build can't render docs/endpoint-index.md from a
stale/partial v1.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds docs/testing.md as the authoritative testing map (consolidated
from docs/contributing.md §8, now shrunk to a pointer), and a
generated docs/endpoint-index.md via scripts/generate-endpoint-index.py
(hooked into scripts/update-openapi.sh). Updates docs/README.md's
reading order and removes the "still to come" placeholder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* generate openapi definitions as separate build task
* first defns
* install etv-client module in docker
* include python entrypoint in docker
* update changelog
timothy
