timothy/ersatztv
1
0
Fork 0
Code Issues 54 Pull Requests 6 Actions Packages Projects Releases 9 Wiki Activity

test_docs_only_detector_clone_depth.py flakes ~8% in the CI container (measured, pre-existing on main) #904

New Issue
Open
opened 2026-08-30 22:58:57 +02:00 by timothy · 0 comments
timothy commented 2026-08-30 22:58:57 +02:00
Owner
Copy Link
Copy Source

scripts/tests/test_docs_only_detector_clone_depth.py fails intermittently in the CI container.
Measured rate: ~7-10% per run of that file, and it is NOT attributable to any branch.

The measurement that settles it

Reproduced on the runner host (bumblebee) in the CI image under the CI job's real limits, running
ONLY that file:

docker run --rm --memory=1g --cpus=1 -v <repo>:/w -w /w \
  docker.gitea.com/runner-images:ubuntu-latest \
  bash -c 'pip install -q pytest pyyaml; PYTHONPATH=. python3 -m pytest \
           scripts/tests/test_docs_only_detector_clone_depth.py -q'

30 runs at each of two shas, same container, same image, git 2.54.0:

sha passed failed
b1b478f29 (PR #903 head) 28 2
dd0f75f1b (main) 27 3

Same rate on both. The failure is pre-existing on main and independent of the branch under test.

Symptom

Always the same two tests, always the module-scoped deep_origin_url fixture (211 commits), always
on the remote side of a file:// clone, with a DIFFERENT object sha each time:

remote: error: Could not read 1e5a33d383cdbf4a8471460ac939afa700a16319
remote: ... possible repository corruption on the remote side.
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Run 1 said fatal: unable to read <sha>, run 2 error: Could not read <sha> — different git code
paths, which is itself evidence of instability rather than a logic error. It fails in ~2s with the
file run alone, so it is not suite size, memory accumulation, or test ordering.

Corrections to my earlier reasoning, recorded so nobody repeats them

  • I first called it a transient flake and re-ran the job. It failed identically, which looked
    deterministic.
  • I then measured the file passing at main and failing at the PR head in the same container and
    concluded it WAS the branch. That was a single run per sha — n=1 on a ~8% flake proves nothing,
    and a 17-commit bisect then passed at every commit including the head.
  • Only the paired 30-run measurement gave an answer. A hypothesis I floated earlier (concurrent
    pytest-as-root runs sharing /tmp/pytest-of-root) is REFUTED: the job's container is created per
    job with AutoRemove:true, so /tmp starts empty and there is nothing to collide with.

Still unknown

Why the origin cannot read its own object. Candidates not yet tested: gc --auto racing the clone;
the 1 CPU / 1 GiB cap interacting with upload-pack on a 211-commit repo; a git 2.54 regression on
file:// transport. The fixture writes 211 commits via 211 sequential git commit processes.

Done-when

  • Mechanism established by measurement
  • Either a fix, or a recorded decision to tolerate it with its reason and a documented re-run policy
  • Adversarial review passed
`scripts/tests/test_docs_only_detector_clone_depth.py` fails intermittently in the CI container. **Measured rate: ~7-10% per run of that file**, and it is NOT attributable to any branch. ## The measurement that settles it Reproduced on the runner host (bumblebee) in the CI image under the CI job's real limits, running ONLY that file: ``` docker run --rm --memory=1g --cpus=1 -v <repo>:/w -w /w \ docker.gitea.com/runner-images:ubuntu-latest \ bash -c 'pip install -q pytest pyyaml; PYTHONPATH=. python3 -m pytest \ scripts/tests/test_docs_only_detector_clone_depth.py -q' ``` 30 runs at each of two shas, same container, same image, git 2.54.0: | sha | passed | failed | |---|---|---| | `b1b478f29` (PR #903 head) | 28 | **2** | | `dd0f75f1b` (`main`) | 27 | **3** | Same rate on both. The failure is pre-existing on `main` and independent of the branch under test. ## Symptom Always the same two tests, always the module-scoped `deep_origin_url` fixture (211 commits), always on the **remote** side of a `file://` clone, with a DIFFERENT object sha each time: ``` remote: error: Could not read 1e5a33d383cdbf4a8471460ac939afa700a16319 remote: ... possible repository corruption on the remote side. fatal: early EOF fatal: fetch-pack: invalid index-pack output ``` Run 1 said `fatal: unable to read <sha>`, run 2 `error: Could not read <sha>` — different git code paths, which is itself evidence of instability rather than a logic error. It fails in ~2s with the file run alone, so it is not suite size, memory accumulation, or test ordering. ## Corrections to my earlier reasoning, recorded so nobody repeats them - I first called it a transient flake and re-ran the job. It failed identically, which looked deterministic. - I then measured the file passing at `main` and failing at the PR head in the same container and concluded it WAS the branch. That was a single run per sha — **n=1 on a ~8% flake proves nothing**, and a 17-commit bisect then passed at every commit including the head. - Only the paired 30-run measurement gave an answer. A hypothesis I floated earlier (concurrent pytest-as-root runs sharing `/tmp/pytest-of-root`) is REFUTED: the job's container is created per job with `AutoRemove:true`, so `/tmp` starts empty and there is nothing to collide with. ## Still unknown Why the origin cannot read its own object. Candidates not yet tested: `gc --auto` racing the clone; the 1 CPU / 1 GiB cap interacting with `upload-pack` on a 211-commit repo; a git 2.54 regression on `file://` transport. The fixture writes 211 commits via 211 sequential `git commit` processes. ## Done-when - [ ] Mechanism established by measurement - [ ] Either a fix, or a recorded decision to tolerate it with its reason and a documented re-run policy - [ ] Adversarial review passed
timothy changed title from `test_docs_only_detector_clone_depth.py` failed once in CI with the origin unable to read its own object to `test_docs_only_detector_clone_depth.py` flakes ~8% in the CI container (measured, pre-existing on main) 2026-08-30 23:20:56 +02:00
timothy added the priority: low label 2026-09-02 20:48:01 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ad-hoc
One-off / ad-hoc work not tracked by a dedicated issue
 api
REST API / HTTP endpoints
 bug
Something isn't working
 ci-cd
Build, test, deploy pipeline
 content
Channel content / schedules / playlists
 dependencies
Dependency updates (Renovate)
 enhancement
New feature or improvement
 frontend
ChicoryTV React SPA frontend
 in-progress
Claimed by an active session — do not pick up
 jellyfin
Jellyfin tuner / IPTV integration
 parked
Excluded from automatic queue pickup; work only when explicitly selected
 priority: high
priority: low
priority: medium
review
Adversarial review finding
 security
Security / vulnerability fix
No labels priority: low
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate timothy (Timothy)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#904