I've been running the benchmark against the pinned repos and went through some of the misses by hand. Four gold labels don't hold up once you open the files they point at:
**model2vec — "how vocabulary is pruned during distillation"**
The primary gold is `model2vec/distill/utils.py`, but that file contains a single function, `select_optimal_device()` — it picks cuda/mps/cpu and has nothing to do with vocabulary or pruning. The actual pruning happens via `prune_added_tokens()`, called from `model2vec/tokenizer/tokenizer.py:71` and `model2vec/distill/distillation.py:81`. The latter is already the secondary gold here, so the fix might just be dropping `distill/utils.py` (or swapping in `tokenizer/tokenizer.py`).
**model2vec — "tokenizer construction and vocabulary building"**
Same file again as secondary gold. Still just device selection, no tokenizer or vocabulary content.
**pydantic — "custom field and model validators"**
The secondary gold `pydantic/class_validators.py` is a 5-line V1 migration shim (`__getattr__ = getattr_migration(__name__)`). If the intent was to credit the deprecated V1-style validators, that file is `pydantic/deprecated/class_validators.py`.
**aiohttp — "WebSocket client implementation"**
The secondary gold `aiohttp/_websocket/reader.py` is a 31-line import switch that picks the C or Python reader at import time. The implementation it re-exports lives in `aiohttp/_websocket/reader_py.py`.
All four checked at the SHAs pinned in repos.json and still present in the annotations on main. Happy to put up a PR with whichever corrections you'd prefer — dropping the entries vs. redirecting them changes scores a little either way, so I didn't want to guess at intent.
Went through the two TypeScript repos the same way. In zod, 8 of the 20 queries point at `packages/zod/src/v4/core/api.ts`, so I checked how much of that file's code actually runs. Of its 114 exported functions, 49 have zero call sites anywhere in the repo (grepped `core.<fn>(` across all of `packages/`, tests included): `classic/schemas.ts` and `mini/schemas.ts` each carry their own independent, identically-behaved builders that construct the schema classes directly instead of calling into `api.ts`. Six of the eight queries land squarely in that unused set; the other two (the general "core public API functions for building schemas", and ".refine/.superRefine") hold up fine and aren't included below.
**zod — "union and discriminated union schema types"**
`api.ts` exports `_union` (line 1167) and `_discriminatedUnion` (line 1204), but neither has a call site — `classic/schemas.ts:1340` and `:1398` (mirrored in `mini/schemas.ts:1012`/`:1064`) define their own `union()`/`discriminatedUnion()` that build `new ZodUnion(...)`/`new ZodDiscriminatedUnion(...)` directly. The actual union-matching logic (`handleUnionResults`, `core/schemas.ts:2096`; the `$ZodUnion` class, `:2121`) is in the current *secondary* gold, `core/schemas.ts`. Suggest swapping primary/secondary — `core/schemas.ts` primary, `classic/schemas.ts` secondary for the builder.
**zod — "optional and nullable type wrappers"**
Same shape: `api.ts`'s `_optional` (1426) / `_nullable` (1439) are never called. The live builders are `classic/schemas.ts:1843`/`:1893` (`mini/schemas.ts:1376`/`:1419`), backed by `$ZodOptional`/`$ZodNullable` in `core/schemas.ts:3343`/`:3437`.
**zod — "z.transform and z.pipe for chaining Zod output transformations"**
`_transform` (api.ts:1413) and `_pipe` (api.ts:1512) are unused. `z.transform`/`z.pipe` actually resolve to `classic/schemas.ts:1816`/`:2080` (`mini/schemas.ts:1352`/`:1584`), backed by `$ZodTransform`/`$ZodPipe` in `core/schemas.ts:3282`/`:3869`.
**zod — "z.record and z.map schema builders for key-value types"**
`_record` (api.ts:1277) and `_map` (api.ts:1294) are unused. The live builders are `classic/schemas.ts:1506`/`:1572` (`mini/schemas.ts:1160`/`:1213`), backed by `$ZodRecord`/`$ZodMap` in `core/schemas.ts:2754`/`:2908`.
**zod — "ZodDefault and ZodCatch schema wrappers that supply fallback values on parse failure"**
Worth calling out separately — there's a naming trap here. `classic/schemas.ts` and `mini/schemas.ts` each define their *own* function literally named `_default` (`classic:1923`, `mini:1445`) and `_catch` (`classic:2035`, `mini:1542`) — same names as `api.ts`'s exports, but independent bodies that construct `new ZodDefault(...)`/`new ZodCatch(...)` directly, never calling `api.ts`'s `_default` (1452) / `_catch` (1497). It's easy to see the matching name and assume the call goes through `api.ts`; it doesn't. `$ZodDefault`/`$ZodCatch` (`core/schemas.ts:3489`/`:3753`) hold the runtime behavior.
**zod — "enum schema types for literal value sets"**
Same naming trap: `classic/schemas.ts` (`_enum` at 1682, exported as `enum`; `nativeEnum` at 1700; `literal` at 1740) and `mini/schemas.ts` (`_enum` at 1265; `nativeEnum` at 1284; `literal` at 1314) each reimplement these independently of `api.ts`'s `_enum` (1338), `_nativeEnum` (1367), and `_literal` (1392), none of which have a call site. `$ZodEnum`/`$ZodLiteral` (`core/schemas.ts:3081`/`:3135`) hold the runtime behavior.
**vitest — "test reporter interface for listening to test events and results"**
The gold `packages/vitest/src/public/reporters.ts` is a 30-line re-export barrel, and since [vitest-dev/vitest#9347](https://github.com/vitest-dev/vitest/pull/9347) (merged 2026-01-12, Vitest 4.1) it warns at import time:
```ts
console.warn('Importing from "vitest/reporters" is deprecated since Vitest 4.1. Please use "vitest/node" instead.')
```
The whole file is just that warning plus `export { AgentReporter, DefaultReporter, ... } from '../node/reporters'` / `export type { Reporter, ... } from '../node/reporters'` — it defines nothing of its own. The actual `Reporter` interface (`onInit`, `onTestRunStart`, `onTestModuleStart`, `onTestCaseResult`, `onHookStart`, `onCoverage`, etc. — exactly "the test reporter interface for listening to test events and results") lives in `packages/vitest/src/node/types/reporter.ts`. The built-in-reporters barrel it's assembled through is `packages/vitest/src/node/reporters/index.ts`. And the entry point the warning itself tells you to use instead, `packages/vitest/src/public/node.ts` (the `vitest/node` subpath export), re-exports the identical reporter surface with no warning at all. A system that resolves this query to the interface's real definition, or to the actively-recommended `vitest/node` entry, rather than to the deprecated shim, scores zero against the current gold.
For context: the neighboring "snapshot testing serialization and comparison" gold in the same file, `packages/vitest/src/integrations/snapshot/chai.ts`, does point at real implementation (the `SnapshotPlugin` chai plugin and its `SnapshotClient`/comparator wiring — no barrel, no deprecation notice), so this isn't a case where the annotations consistently favor the public entry point over the implementation; the reporters one looks like a one-off.
Happy to fold any of these into a PR along with the four from the original report — same caveat as before, dropping vs. redirecting the entries moves scores a little differently per query, so I didn't want to guess at intent.
Hey @Hartistic!
Thanks for taking the time to go through the benchmark. Feel free to submit PRs with fixes and, if you have time/compute, rerun the benchmarks. The main issue will be that while running semble itself is relatively easy, the other models do cost a lot of time to run.
Hey @Hartistic, thanks for checking this and fixing the annotations! I merged your PR, we will rerun the benchmarks soon to reflect the changes. Feel free to reopen this, or contribute more changes, if you encounter any other wrong annotations.