ersatztv/docs/superpowers/specs/2026-07-17-smartcollection-rule-builder-design.md

SmartCollection visual rule builder — design (sub-project #1 of #176)

Issue: ersatztv#176 — "Rethink channel/playout/schedule creation: PseudoTV-style channel-first flow with a Kodi-smart-playlist-like query builder."

Date: 2026-07-17 · Status: design approved, pre-implementation.

---

1. Context & why the scope shrank

#176 as originally filed described three legs of a PseudoTV-style vision. Recon (2026-07-17) shows two of them already shipped since the issue was written:

• Autotune (channel-first bulk create) — shipped as #69: /app/auto-tune, a 3-step Configure → Preview → Create wizard, each channel backed by an auto-generated SmartCollection query (AutoTuneScreen.tsx, CreateAutoTunedChannelsHandler.cs, AutoTuneAxisMap.cs).
• Composite Create-Channel wizard — shipped as #63: web/src/builder/ChannelBuilder.tsx at /app/new-channel, composing collection-lineup + template + playout + advanced overrides in one call (POST /api/v1/channels/from-lineup).

The one genuinely missing leg is the visual WYSIWYG rule builder. Authoring a SmartCollection query today is a single free-text box (CollectionsScreen.tsx:271) over the Lucene grammar — no field/operator/AND-OR UI. This design covers sub-project #1: a visual rule builder landing in the SmartCollection create/edit dialog. Inline adoption by ChannelBuilder and Auto-Tune is deliberately deferred (the builder is built as a reusable component so that later adoption is a trivial plug-in).

2. Approved architectural decisions

Three pillars, settled during brainstorming:

1. Scope = the rule builder in the SmartCollection editor (highest-traffic surface, standalone value, no dependency on the wizard).
2. Compile-only, closed subset, no stored AST, no schema change. The builder emits a canonical closed subset of the Lucene grammar and parses exactly that subset back. Because compile and parse are exact inverses over a closed subset, anything the builder produces round-trips losslessly with nothing new persisted — the SmartCollection keeps storing a plain Lucene query string, exactly as today. Queries outside the subset (hand-written fuzzy/boost/wildcard-heavy Lucene) open in a raw-text fallback. This was chosen over persisting a rule AST because even an authoritative AST would still need a Lucene→rules parser for the many pre-existing free-text queries (every Auto-Tune query included), so the migration cost of storing an AST buys almost nothing.
3. Field catalog on the backend, compile/parse on the client. A new read-only GET /api/v1/search/fields is the single source of truth for the field vocabulary (no client/index drift; MCP can introspect it). The compile/parse/round-trip logic lives in the SPA.

3. The rule model (Kodi model, one level of nesting)

Rule  = { field: string, operator: Operator, value: string | [string, string] }
Group = { match: "all" | "any", children: Array<Rule | Group> }

• Top level is a Group. Its children may be Rules and/or Groups.
• Sub-groups are one level deep only — a sub-Group's children are a flat list of Rules (no Group inside a Group inside a Group).
• Why one level: pure-flat (match all/any only) can't express the most common real query, type:movie AND (genre:Horror OR genre:Thriller); one level of nesting expresses it exactly (all: [type:movie, any:[Horror, Thriller]]) and matches what Kodi ships. Arbitrary nesting is YAGNI for channel queries and bloats the compiler/parser/property-test surface.

The closed Lucene subset

• rule → clause:
  • text is → field:"value" · is not → NOT field:"value" · contains → field:*value* · starts with → field:value*
  • fulltext (plot) contains → field:value · not contains → NOT field:value
  • number = → field:N · > → field:{N TO *} · < → field:{* TO N} · between → field:[a TO b]
  • date before → field:{* TO d} · after → field:{d TO *} · between → field:[a TO b]
  • enum is/is not → field:"value" / NOT field:"value" (value from a fixed allowed set)
• group → clauses joined by AND (match=all) or OR (match=any), wrapped in parentheses when the group is nested.
• Quoting: values containing whitespace or reserved characters are quoted; the compiler is the sole author of quoting so the parser can rely on it.

Any string not matching this shape is unparseable → the editor stays in raw-text mode.

4. The field catalog endpoint

GET /api/v1/search/fields — read-only, returns a curated, typed, labeled catalog derived from LuceneSearchIndex:

[{ "name": "genre", "label": "Genre", "type": "text", "group": "General" },
 { "name": "minutes", "label": "Duration (min)", "type": "number", "group": "Technical" },
 { "name": "release_date", "label": "Release date", "type": "date", "group": "Dates" },
 { "name": "type", "label": "Item type", "type": "enum", "group": "General",
   "values": ["movie", "episode", "music_video", "other_video", "song", "image"] }]

• Curation happens server-side. Internal/index-plumbing fields are excluded: id, library_id, library_folder_id, tag_full, jump_letter, title_and_year_search, language_tag, sub_language_tag, metadata_kind (kept only if it proves user-meaningful). The rest get friendly labels + a group (General / TV / Movie / Music / Technical / Dates).
• Type taxonomy: text (keyword-exact metadata), fulltext (tokenized — plot), number (minutes, seconds, height, width, season_number, episode_number, video_bit_depth, chapters), date (added_date, release_date), enum (small closed sets — type, video_dynamic_range, state; each carries a values array).
• The endpoint is the single source of truth: the SPA renders exactly what it returns, and MCP can introspect the same list.

Follow the api-conventions checklist, then regenerate v1.json + endpoint-index.md via ./scripts/update-openapi.sh and npm run generate:api.

5. UI integration in the SmartCollection dialog

The create/edit dialog (CollectionsScreen.tsx:200-283) gains a Builder | Advanced mode toggle:

• Builder mode — the rule-group UI: a Match: All | Any toggle, a list of rule rows (field ▾ → operator ▾ → value input), + Add rule / + Add group (one nested level), remove buttons. The value input is chosen by field type: text box (text/fulltext), number input(s) (number, single or a-to-b pair), date picker(s) (date), dropdown (enum). Every edit debounced-compiles → live preview (item count + sample) via the existing getLibraryBrowseItems({ query }).
• Advanced mode — today's raw text box (also live-previewed).
• Mode switching: Builder→Advanced fills the text with the compiled query. Advanced→Builder best-effort parses; if the text is outside the closed subset, it stays in Advanced with an inline note ("This query is too advanced to show in the builder").
• Opening an existing SmartCollection: parse the stored query — fits the subset → open in Builder with rules populated; otherwise → open in Advanced. (Every Auto-Tune query like type:episode AND genre:"Horror" fits, so those open cleanly in the builder.)
• Save is unchanged — posts the compiled (or raw) Lucene string to the existing SmartCollection write path. No backend write-path change; the only new backend surface is the read-only GET /api/v1/search/fields.

Component structure (isolated & reusable)

web/src/builder/rules/            (co-located with ChannelBuilder; final path validated vs spa-conventions.md)
  types.ts         Rule | Group | Match | Operator | FieldType
  fieldCatalog.ts  useSearchFields() hook over GET /api/v1/search/fields
  compile.ts       rules → lucene subset (sole author of quoting)
  parse.ts         lucene subset → rules (exact inverse; returns null on out-of-subset)
  RuleBuilder.tsx  the group/rule UI (consumed by CollectionsScreen; later by ChannelBuilder/Auto-Tune)

Each unit has one purpose and a narrow interface: compile(Group) → string, parse(string) → Group | null, RuleBuilder({ value: Group, onChange }). RuleBuilder can be understood and tested without reading compile/parse internals, and those can change without touching the UI.

6. Testing

• Round-trip property test (keystone): a deterministic generator produces rule trees within the closed subset → compile → parse → assert deep-equal. This is what makes "compile-only, no stored AST" safe. Hand-rolled generator over the finite field/operator space (deterministic; introduce fast-check only if it is already a web/ dependency).
• Compile/parse unit tests: each operator → expected Lucene; each Lucene shape → expected rules; representative out-of-subset strings → parse returns null (drives the raw-text fallback).
• RuleBuilder component tests (vitest + testing-library): add/remove rule & group, match toggle, mode-switch fills text, editing an existing query populates the builder. Give heavy-render tests explicit vitest timeouts (CI-VM guidance).
• Backend (NUnit + Shouldly): GET /api/v1/search/fields returns the curated list, excludes internal fields, and reports correct types.
• Live-E2E: not gated by the write-path rule — the only new backend surface is a read-only GET and the SmartCollection write path is unchanged. Still run a light Playwright smoke (build a rule → preview count → save → reopen in builder) and a cold-context adversarial review over the diff (the parser is subtle enough to warrant it even though the skip rubric would permit skipping).

7. Docs to update in the same PR

• New endpoint → docs/api-conventions.md checklist, then regenerate v1.json + endpoint-index.md (./scripts/update-openapi.sh + npm run generate:api).
• docs/decisions.md (append-only) — record the compile-only / closed-subset / no-stored-AST decision and the one-level-nesting choice.
• docs/spa-conventions.md — document the reusable rule-builder component pattern if it establishes one.
• Not touched: blazor-route-parity.md / domain-model.md — no route change, no new entity.

8. Scope boundaries — deferred as tracked follow-up issues

These are explicitly out of sub-project #1 and will be filed as separate issues:

1. Facet-value typeahead (v1b) — a GET /api/v1/search/fields/{name}/values?q= distinct-values endpoint turning text values into autocomplete comboboxes. Purely additive: swap the text input for a combobox, nothing else changes. (Keyword-exact matching means typos silently return zero; live-preview count is the v1 safety net.)
2. Relative dates ("in the last N days") — can't be stored as relative in a compile-only Lucene string; needs a query-time macro.
3. Nested groups beyond one level.
4. Inline adoption in ChannelBuilder & Auto-Tune — reuse RuleBuilder for inline query authoring in the channel wizard and to generalize Auto-Tune's fixed 3-axis picker.

9. Risks

• Round-trip correctness is the central risk; the property test is the mitigation, and the compiler being the sole author of quoting keeps the parser's job bounded.
• Field/label curation drift — mitigated by deriving the catalog from LuceneSearchIndex server-side (single source of truth). Adding an index field simply won't appear until curated — additive, low harm.
• Live-preview chatter — debounce compile→preview; reuse the existing preview call so no new load pattern is introduced.