Portable emit targets (ts/go/rust) have no incremental `edit` — full re-parse only

#57 · closed · 1 comments

View on GitHub ↗

johnsoncodehk

The target-agnostic emitter (#6, #56) emits portable parsers in TypeScript, Go, and Rust, but they are **full-parse only** — they expose `tokenize(src)` + `parse(tokens)` and have no incremental re-parse. Incremental `edit` exists only in the JS paths: | Path | Incremental | | --- | --- | | `createParser()` interpreter | ✅ `.edit(...)` | | `jsTarget` (optimized JS emit) | ✅ exported `parseEdited` (PR #38) | | portable `tsTarget` / `goTarget` / `rustTarget` | ❌ full `parse(tokenize(src))` only | ### Why The incremental machinery — watermarked memo carry, subtree reuse (`adoptPath`), and the maxPos watermark — lives in the optimized path (`src/emit-parser.ts` / `src/gen-parser.ts`), shipped as `parseEdited` in PR #38 and gated `≡ fresh` (1.4–5.6×). The portable emitter (`src/emit-portable.ts`'s `buildIR`) renders a different, simpler parser: a clean backtracking recursive-descent + Pratt core with **no memo / no incremental infrastructure**. So nothing in the portable IR or the `target-*.ts` renderers can carry a prior parse forward. ### What it would take - Lift the memo-carry + watermark + subtree-reuse model into the language-agnostic IR layer so each `Target` can render it. - Per-target incremental state: ts via module globals; go/rust need incremental **arena reclamation** (reusing surviving subtrees without leaking the flat `nodes`/`kids` slices). - Extend `test/portable-targets.ts` to assert `edit ≡ fresh` for the portable targets, mirroring the existing JS incremental gate. Substantial — not a single-construct change. Tracking it as a known boundary of the portable emitter; the JS interpreter / `jsTarget` remain the path for incremental use until then.

Comments

johnsoncodehk

Closing: the portable targets (ts/go/rust) now have a full incremental `edit` path, shipped incrementally across #61–#70. Mapping back to the three items in the issue body: **"Lift the memo-carry + watermark + subtree-reuse model into the language-agnostic IR layer so each Target can render it"** — done, with one deliberate design change: instead of porting jsTarget's memo-carry machinery, the portable targets use a simpler model that each renderer emits symmetrically: - token alignment window after each edit batch (#63) - windowed relexing for all four lexer state classes — stateless, newline-mode, regex-context, template/combined (#64, #65, #66, #67) - top-level subtree reuse with a per-subtree lookahead watermark (`ext`), exact boundary adoption, and full-parse fallback on any miss (#69) - segment reuse for newline-interleaved entries (`[opt(Stmt)] many(Newline, opt(Stmt))` — the env-spec family) (#70) **"Per-target incremental state: ts via module globals; go/rust need incremental arena reclamation"** — done in #69: Go appends to the persistent package-level arena on the reuse path (fresh parses still reset), Rust's `Doc` owns long-lived `nodes`/`kids` vecs (no `unsafe`), and both bound garbage with a jsTarget-style policy (`len(nodes) > 4 × max(baseline, live)` → fresh rebuild; measured: 500 edits stay at 607 rows vs unbounded 3151 with reclamation disabled). **"Extend test/portable-targets.ts to assert edit ≡ fresh"** — done: every edit scenario runs per target in validated and fast modes; the final CST is byte-compared against a fresh parse of the final text; validated mode additionally self-checks `streamEq` (relexed token stream ≡ fresh tokenize) and `treeEq` (reused tree ≡ fresh parse) per batch. Alignment fields are cross-checked against the interpreter's lexer, and reuse counters (`reused`) are pinned per scenario. API surface: `createDoc(src, { validate? })` / `Doc.edit(edits)` in TS, `NewDoc`/`Doc.Edit` in Go, `Doc::new`/`Doc::edit` in Rust; the self-checks are opt-in (off by default) per #68. Known boundaries (tracked deliberately, not silently): reuse granularity is top-level statements/segments (no nested block surgery à la jsTarget's `trySurgery`); reject inputs always take the full-parse fallback (portable targets have no error trees / recovery, and accept/reject parity with the interpreter is the gate contract); entry shapes beyond `star(rule|alt)` and the newline-interleaved form fall back to full reparse with `reused: 0`.