Structural invariant checks over a large generated-bundle corpus

#6 · open · 1 comments

View on GitHub ↗

dazuma

**Placeholder, deliberately not a spec.** Raised during the 2026-09-15 grilling session that produced #4, and explicitly scoped *out* of it as a larger project in its own right. Recorded so it is not forgotten. ## The idea The template's tests check generated output against hand-authored fixtures in `examples/geometry/doc`. That gates correctness for the features the example exercises, but says nothing about properties that must hold across *all* real gems — and consumers like the `agentdocs lookup` Reader depend on exactly those properties. Candidate invariants: - `## Member Summary` is a prefix of every class/module concept — no member section precedes it. - Every `### ` heading is unique within its file. - FQN to file path is total: every name listed in `index.md` resolves to a file. - Every intra-bundle link resolves. ## Why it is worth doing Checking the first invariant ad hoc over a local 117-bundle corpus (7,473 concept files) during the #4 design immediately found the edge case that would have broken a naive implementation: 5 files where an H2 appears *before* `## Member Summary`, because the gem's own docstring prose contains `##` headings — `fileutils`, `bundler/TSort`, `psych`, `did_you_mean`. A further 316 files have no H2 at all. Neither shape exists in `examples/geometry/doc`, so no in-repo test could have surfaced them. They were found by measurement, and only because someone thought to measure. ## Why it is not simply added to CI A corpus of this size is not in the repo and is not reproducible in CI — building it takes hours and depends on whatever is installed locally. So this would be an opt-in diagnostic, deliberately outside `toys ci`, which raises its own questions about how results are recorded and acted on. ## Open Corpus selection and reproducibility, which invariants are real requirements versus incidental regularities, how a violation is triaged (template bug versus legitimate gem-side variation), and the relationship to `docs/dev/Dogfood.md`, which already records per-gem findings by hand.

Comments

dazuma

Corpus selection — one of the open items above — now has an answer. `toys corpus manifest` (repo-internal, `.toys/corpus/`) generates a durable manifest of gem names and exact versions: - `.toys/.data/corpus-manifest.yml` — the generated corpus, currently 250 gems, committed. - `.toys/.data/corpus-config.yml` — hand-written pins, exclusions and family caps, merged in on every run. Gems are ranked by two strategies from the public ClickHouse dataset behind ClickGems (trailing-window downloads, and distinct runtime dependents) and interleaved by rank; versions are resolved against rubygems.org. Regeneration takes about 25 seconds. The manifest is committed rather than derived at check time, which is what makes a check reproducible: rankings move daily and the service behind them has no availability guarantee. The four gems named in the issue body — `fileutils`, `psych`, `did_you_mean`, `tsort` — are pinned with their reasons recorded, so the H2-before-`## Member Summary` edge case stays in the corpus regardless of how the rankings move. Still open here: which invariants are real requirements versus incidental regularities, and how a violation is triaged (template bug versus legitimate gem-side variation).