Second pin-recovery cycle on this branch — main moved twice during review and each
rebase rewrites the sha of the commit that touched docker/ci. Records that #598 now
length-checks the pin at exactly 7 chars, which turns a locally-computed 8-char
`git rev-parse --short` into a loud gate failure instead of a confusing
manifest-unknown at image-pull time. That guard came from #594, filed by this
session after hitting exactly that ambiguity.
This commit also IS the recovery: it touches docker/ci, so ci-image.yml tags it and
the pin can be re-pointed in the follow-up commit.
Refs #445#594
Hit this for real on this branch. `ci-image-pin` went red after a rebase that
was otherwise clean, and the failure is confusing on three counts:
- the pin must equal the short sha of the commit that touched docker/ci, and a
rebase REWRITES that sha (e9fd26f6 -> 9130274c here);
- the pin still resolves to a real commit and the tagged image still exists in
the registry, so nothing looks broken;
- the force-push does NOT republish: ci-image.yml filters on
`paths: docker/ci/**`, and a rebase that leaves the Dockerfile's content
unchanged produces no diff for that path.
And it cannot be fixed by re-dispatching ci-image.yml, because that tags
`git rev-parse --short HEAD` — the branch HEAD, not the commit that touched
docker/ci. The two coincide only when the docker/ci commit IS HEAD, which is why
the original two-step worked and the post-rebase state does not.
Recorded in docs/ci-cd.md with the recovery, plus the cheaper lesson: land a
toolchain-image change on its OWN branch first, so the consuming branch never
carries the docker/ci commit through a rebase.
This commit is also the recovery itself — it touches docker/ci, so it becomes the
commit ci-image.yml tags, restoring the pin dance.
Refs #445
Adds `chromium-headless-shell` + its system deps to the shared CI toolchain
image so the `functional-e2e` job can run the UI-E2E Playwright flows without
installing a browser per run — the same "jobs install nothing at run time"
rule as the rest of this image.
Measured on this exact base (Ubuntu 24.04 ffmpeg base, amd64):
- headless shell: 267M vs full chromium: 656M
- `chromium.launch()` resolves to the shell anyway (the config never asks
for headed), so the only cost is that a HEADED run inside this image would
fail — CI-only, and deliberate.
- Chromium launches as root in a container with NO --no-sandbox /
chromiumSandbox:false opt-out, so the Playwright config needs no sandbox
workaround. Verified rather than assumed.
`ARG PLAYWRIGHT_VERSION` must stay equal to web/package.json's EXACT
`@playwright/test` pin: Playwright ties a browser revision to the package
version, so a mismatch leaves no usable browser.
The build-time smoke test LAUNCHES the browser (not just a file test), so a
missing system library fails the image build instead of a CI run.
First half of the deliberate two-step toolchain bump: this commit is what
ci-image.yml publishes as :<sha>; the follow-up commit pins it.
Refs #445
Review fixes from the cold adversarial pass on PR #411. Verdict was MERGEABLE, but one
HIGH is a genuine defect worth the round-trip.
[decisions-edit] rationale: the 12 "modified" lines in docs/decisions.md are lines THIS PR
added in its own previous commit (e33205e8), not a settled entry on main. The hook diffs
against HEAD; net against origin/main this file is still a pure insertion (65 added, 0
deleted -- verified, and CI's decisions-guard diffs against the PR base). It is also the
sanctioned reason regardless: the entry as first written contained a factual error (see
HIGH below).
HIGH -- `--memory=2g` alone silently grants 2g of SWAP as well. Docker defaults an unset
--memory-swap to 2x --memory. Verified live on bumblebee: `--memory=2g` alone gives
memory.max=2147483648 AND memory.swap.max=2147483648; adding `--memory-swap=2g` gives
memory.swap.max=0. Capping RAM while permitting swap is close to the worst outcome on the
one host whose swap thrash is the entire reason for the cap -- and a swapping mysqld
mid-DDL is exactly the pathology behind the known `Command Timeout expired` migrations
flake, so the naive cap could have made that flake worse rather than better. The comment
promising "if this ever OOM-kills the service" described a failure mode that would not
have happened first: it would have swapped, silently. Now --memory-swap=2g (swap off);
prefer a loud OOM over silent degradation.
The same 2x applies to the runners' 10g job slots (10 GiB RAM + 10 GiB swap each), which
is a plausible direct mechanism for the incident's 21 GiB swapped. That is #604's
boundary -- reporting it there.
MEDIUM -- `if: always()` does not make a step advisory. It controls whether the step RUNS,
not whether its failure fails the job, and defaults.run.shell: bash means -e is on, so a
failed cat/redirect in an informational step could redden a green test job. Use
continue-on-error: true, matching the neighbouring Coverage summary step.
MEDIUM -- the peak step must run LAST. memory.peak read at step N reports the peak only up
to N, so sitting before Coverage summary excluded reportgenerator's memory -- under-
reporting the very number #604 sizes caps on. Moved after it.
LOW -- dependency-scan.yml gets the env too; workflow env does not cross workflow files,
and that one runs unattended on a cron against the prod media host.
NIT -- Dockerfile ENV moved below the openapi-generator wget: an ENV invalidates every
layer under it, and that ~30MB download is deliberately ordered early to stay cached
(#190). Nothing between it and restore compiles, so this placement is free.
NIT -- docs no longer imply this shrinks `format`. dotnet format loads Roslyn in-process
via MSBuildWorkspace and never spawns csc, so its 3.95 GiB is untouched; sizing the small
lane on the assumption it shrank would be a trap.
Also record honestly that 543 MiB is init+idle rather than the 787-migration replay (so 2g
is a measured floor + headroom, not a measured ceiling -- the migrations job going green
is what validates it) and that --cpus=2 is judgement with no measurement behind it. The
entry criticises dressing estimates as measurements; it should not do that itself.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refs #406, server-management#604. Items 1 and 2 of #406 (item 3 landed in PR #407).
Root cause of the 2026-07-17 bumblebee incident (load 340, 21 GiB swapped, prod media
down until reboot) is CI-induced memory pressure, and the largest single consumer was
Roslyn's persistent VBCSCompiler at 7.8 GB RSS. It is a compiler *server*: it outlives
the build that started it to keep its heap warm for the next one. In CI there is no next
build -- each job container is torn down at the end of the run -- so it is pure cost, and
it is the actual reason each job needed a 10 GiB cap.
1. Disable the compiler servers via the workflow's top-level env (UseSharedCompilation,
DOTNET_CLI_USE_MSBUILD_SERVER, MSBUILDDISABLENODEREUSE). MSBuild surfaces env vars as
properties and only defaults UseSharedCompilation when empty, so the env var wins and
every dotnet call is covered without touching each call site.
Also set as ENV in the Dockerfile's SDK stage: the workflow env reaches runner-side
jobs only, and the `build` job compiles inside `docker build` where it does not
propagate -- that is the job server-management#570 measured pegging 5.999/6 GiB, so it
is the one that most needs this. Build stage only; the final image is FROM
runtime-base, so nothing lands in the shipped image.
Verified locally: a default build leaves 1 VBCSCompiler alive, the same build under
these vars leaves 0, and ErsatzTV.sln still builds clean (0 errors). Trade-off: csc
runs per project instead of via the server, costing some build time.
2. Cap the mysql service (--memory=2g --cpus=2). A runner's container.options applies to
the job container ONLY: a live migrations job reported HostConfig.Memory=10737418240
for the job and mem=0 nanocpus=0 for its mysql service, i.e. unbounded. 2g is sized on
measurement -- mysql:8.4 with this exact env peaked at 543 MiB during init, 481 MiB
idle (probed on bumblebee) -- not on inheritance.
Also add a `Report peak container memory` step to the test job (cgroup memory.peak,
always(), never fails a build). server-management#604 sizes both runners' caps on that
number and it was inherited rather than measured; now every run reports it.
Docs: ci-cd.md gains the compiler-server + services-cap sections and the lane table is
corrected (PR #407 reverted the #390 lane move without updating it). decisions.md records
all three, incl. the standing rule that a services: container never inherits a cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cold adversarial review findings:
- MEDIUM (Renovate generates pin drift): add a blocking ci-image-pin job. Renovate
manages docker/ci/Dockerfile's base pins but cannot bump an opaque :<sha> in
container.image, so a base bump would publish a new image, test the OLD one, and
merge with the Dockerfile disagreeing with the pin. The guard fails when the pin
isn't the last commit to touch docker/ci, or when the 5 jobs pin different tags —
making the documented two-step enforced rather than remembered.
- MEDIUM (cron was a no-op): the weekly rebuild updated nothing (jobs pin :<sha>) and
buildcache would have restored the apt layer verbatim, collecting none of the base
updates it existed for. Added no-cache on the schedule path and rewrote the comment
to state what it actually is: a build canary + a fresh :latest for the next bump.
- LOW: FFMPEG_TAG was referenced in the Dockerfile + docs but never existed (the FROM
is hardcoded); reworded both.
- LOW: paths: filtered the exact file while the docs claimed docker/ci/** — use **.
- NIT: docs oversold ENTRYPOINT reset as a gotcha; act overrides it anyway. Marked
defensive.
Refs #390
The build-time smoke test failed the image build even though the toolchain was
healthy: reportgenerator prints its version banner then exits 1 ("No report
files specified"), so it has no clean version probe. Probe the dotnet-tool shim
on PATH instead. dotnet --info confirmed the SDK + .NET 10.0.10 runtimes resolve
correctly on the ffmpeg base (ICU/glibc fine).
Refs #390
Layers the .NET 10 SDK + Node 22 onto our own ersatztv-ffmpeg base (the same
COPY --from=.../usr/share/dotnet pattern docker/Dockerfile uses for prod), and
bakes the dotnet-ef + reportgenerator global tools the jobs installed per run.
Builds/pushes 192.168.1.95:3000/timothy/ersatztv-ci:<sha> via a separate
workflow so a ~2GB toolchain image isn't rebuilt on every commit.
Refs #390
Guard analyzer package references for the inline-version MCP tool and copy the SDK suggestion baseline into the Docker build context before restore.
Refs #15
Co-Authored-By: Codex <codex@openai.com>
Three CI wins from ersatztv#190, all config-only:
- Cache ~/.nuget/packages via actions/cache@v4 in both the `test` and
`migrations` jobs, keyed on Directory.Packages.props + global.json, so
dotnet restore no longer starts from scratch every run.
- Shallow-checkout the `test` job (fetch-depth: 1) since it never runs
git describe/log; only `build` needs full history for versioning. The
`migrations` job's checkout was already effectively shallow (default
fetch-depth 1), documented explicitly.
- Reorder the Dockerfile so the openapi-generator jar wget happens before
the openapi spec COPY, decoupling the ~30MB download layer from spec
changes (e.g. v1.json) that previously busted it on every API PR.
Documented all three in docs/ci-cd.md alongside the existing job/Dockerfile
notes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
node:22-noble does not exist on Docker Hub (no noble variant of the
official node image; the -noble suffix was carried over from the MS
dotnet image tags). Broke the main image-build job (run 482) after
PR #120 landed, since PR runs skip the build job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move all 154 PackageReference versions (96 distinct packages) out of the 14
project files into a single central Directory.Packages.props with
ManagePackageVersionsCentrally=true. No version changes — every package was
already pinned identically across projects (no conflicts detected), so this is a
pure relocation: updates become one-line and cross-project version drift is
structurally impossible.
Also copy Directory.Packages.props into the Docker image build before restore:
with CPM the csproj carry no versions, so the image's `dotnet restore` fails
without the central manifest (verified: NU1015 across every project).
Restore + Release build verified locally, plus a simulation of the image's
restore layer under linux-x64 (0 errors; only the pre-existing
NCalcSync/SQLitePCLRaw advisories remain, demoted to warnings, tracked in #8).
Part of #14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add concurrency group so the single jazz runner can't run the
push-main-then-push-tag release flow in parallel (shared :buildcache
+ smoke container would collide).
- Add pull_request trigger running the test job only (PRs had no gate);
skip the build job on PRs.
- Only push images from main or a v* tag (workflow_dispatch from other
refs now builds without publishing, instead of clobbering :latest/:prod).
- Replace the log-grep smoke check with a real HTTP readiness probe
(docker exec python3 -> http://localhost:8409/), unique container name,
and trap-based cleanup to avoid leaks on cancel.
- dotnet test now runs -c Release --no-build (was rebuilding in Debug).
- Directory.Build.props: WarningsAsErrors=NU1904 so critical NuGet
advisories block in every project, not just ones with
TreatWarningsAsErrors.
- Dockerfile copies global.json + .editorconfig too, so the image build
matches CI's SDK pin and analyzer severities.
- Remove dead .github/dependabot.yml + FUNDING.yml (upstream-pointed).
- Rewrite docs/ci-cd.md to the implemented pipeline.
Refs #4, #3, #8.
.NET 10's NuGet audit promotes vulnerable transitive packages to NU1901-1904
warnings during restore; the app projects set TreatWarningsAsErrors=true, so
restore failed on NCalcSync 5.11.0 (moderate) and SQLitePCLRaw 2.1.10 (high).
Demote NU1901/NU1902/NU1903 to warnings in Directory.Build.props (NU1904
critical still errors). Also COPY Directory.Build.props/.targets in the
Dockerfile before restore so the image build honors the same props as
local/CI builds (it previously copied only the .sln). Underlying deps tracked
in #8.
Refs #4, #8.
Adds .gitea/workflows/docker-build.yml: a test-gated, amd64-only Docker
build that pushes the fork's own image to the Gitea container registry
(192.168.1.95:3000). The .NET test job gates the image build; the build
job stamps INFO_VERSION from git, uses buildx registry layer caching, and
smoke-tests that the built image boots before finishing.
Tagging:
- push to main -> :latest + :<short-sha> (test image; prod untouched)
- push tag v* -> :prod + :<version> + :<short-sha> (prod release)
Updates docker/Dockerfile to pull the FFmpeg base image from our Gitea
registry (192.168.1.95:3000/timothy/ersatztv-ffmpeg:7.1.1) instead of the
archived upstream ghcr.io image.
Removes the upstream .github/workflows/* — they target GHCR/DockerHub and
Azure/Apple signing secrets we don't have, call reusable workflows at dead
ersatztv/ersatztv@main paths, and were running as noise (incl. a daily
stale-issue cron) on the Gitea runner. Upstream is archived, so there are
no future merges to preserve them for.
Refs #4, #3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* generate openapi definitions as separate build task
* first defns
* install etv-client module in docker
* include python entrypoint in docker
* update changelog
timothy
