johnsoncodehk/tsslint-dify-bench

Subprocess-isolated bench: @tsslint/compat-eslint vs ESLint Linter on Dify web/

★ 0Forks 0JavaScriptGitHub ↗Compare

README

tsslint-dify-bench

Benchmark for @tsslint/compat-eslint (TSSLint's ESLint compatibility layer) on a real-world Next.js codebase: Dify web/. Compares three pipelines on the rule Dify's CI actually enables — react-x/no-leaked-conditional-rendering:

  1. TSSLint 3.0.4 — published @tsslint/compat-eslint from npm
  2. TSSLint local — local checkout of @tsslint/compat-eslint (main)
  3. ESLint Linter — vanilla ESLint with @typescript-eslint/parser

5860 source files, single shared ts.Program, noInlineConfig for fairness (none of the three implements eslint-disable directives).

Methodology

Three things this bench gets right that an obvious-looking single-process bench gets wrong:

1. Subprocess-isolated, OS-level peak memory

Single-process process.memoryUsage().heapUsed deltas before/after a lint pass swing 20× run-to-run on a workload like this — V8 GC scheduling dominates. Even worse, V8 heap totals miss native overhead from the TS compiler, parser, and ESTree converter.

This bench forks a separate subprocess per pipeline and reads process.resourceUsage().maxRSS (the OS's own peak resident-set tracker — monotonic over the subprocess lifetime, no sampling needed). Each pipeline gets clean V8 state, no fragmentation carry-over, no JIT cache pollution.

2. Force-bind before lint

ts.createProgram doesn't populate ts.Node.parent pointers eagerly. Most ESLint pipelines need them — typescript-estree.canContainDirective crashes on undefined.kind if you call it on a program that hasn't been bound. The tsslint CLI dodges this by going through ts.createLanguageService (which binds during construction); a hand-rolled bench has to do it explicitly:

program.getTypeChecker();
program.getSemanticDiagnostics();

Skipping this turns the bench into a measurement of how fast each pipeline silently throws.

3. Three parity gates against silent-zero results

A pipeline that returns 0 reports in 0.3 s looks like a great result. It's also exactly what you get when the rule never actually runs. This bench fails loudly on any of:

  • Canary: a synthetic <div>{count && <Span />}</div> fixture (count: number) must produce ≥1 report. If a pipeline scores 0 on this, the bench aborts before timing.

  • Visit counter: each rule.create() call is counted. Must equal the file count. Catches glob/path mismatches that cause ESLint flat config to silently skip files. (We hit this exact failure during development due to macOS /tmp → /private/tmp symlink resolution.)

  • Throw counter: any exception inside the rule run is counted. Must be 0. Catches the parent-pointer issue described above and any other per-file crash that would otherwise be swallowed.

Running it

Node 18+ and pnpm required.

  1. Clone Dify and pin to the commit this bench was last validated against:

    mkdir ~/dify && cd ~/dify
    git init
    git remote add origin https://github.com/langgenius/dify.git
    git fetch --depth 1 origin 38eb04dc98c8b21684cb10167bfe567c5b6f77e9
    git checkout FETCH_HEAD
    cd web && pnpm install --ignore-scripts

    Pinning matters: rule output and file count are sensitive to upstream refactors, and a moving target makes results noncomparable across runs.

  2. Clone TSSLint and build @tsslint/compat-eslint:

    git clone https://github.com/johnsoncodehk/tsslint.git ~/tsslint
    cd ~/tsslint && pnpm install && pnpm -r exec tsc --build
  3. Run the bench:

    DIFY_WEB=~/dify/web \
    TSSLINT_LOCAL=~/tsslint/packages/compat-eslint/index.js \
      node --max-old-space-size=8192 bench.mjs

The coordinator forks one subprocess per pipeline; total runtime is around 3–4 minutes on a developer Mac.

Sample output

──── Summary ────
Wall time (min of 3 timed iterations after warmup):
  TSSLint 3.0.4   22917 ms  (0 reports, 0 throws)
  TSSLint local    1509 ms  (0 reports, 0 throws)
  ESLint          25020 ms  (0 reports, 0 throws)

  local vs 3.0.4:  15.19× faster
  local vs ESLint: 16.58× faster

Peak memory (OS-level maxRSS, separate subprocess per pipeline):
  TSSLint 3.0.4    7040 MB
  TSSLint local    3750 MB
  ESLint           7017 MB

  local vs 3.0.4:  1.88× lower
  local vs ESLint: 1.87× lower

The 3-4 GB baseline in maxRSS is the ts.Program itself — Dify pulls a lot of node_modules types into the binder. The ~3.3 GB difference between local and the other two reflects the cost of typescript-estree's eager ESTree conversion plus ESLint's SourceCode / scope-manager state, which TSSLint local lazy-materializes per listener hit instead.

Why this rule

react-x/no-leaked-conditional-rendering is what Dify CI actually enables (web/tsslint.config.ts). It's type-aware (needs parserServices and ts.Checker), JSX-heavy, and triggers on a narrow set of nodes — exercising both the parser/type pipeline and the selector dispatch hot path.

Run-to-run variance is high for the slow pipelines (3.0.4 / ESLint) — they allocate enough that GC pauses dominate. TSSLint local is an order of magnitude more stable for the same reason.

License

MIT

Contributors

johnsoncodehk

Issues