SkinnnyJay/token-optimizer.utils

[](https://www.npmjs.com/package/@simpill/token-optimizer.utils) [](https://github.com/SkinnnyJay/simpill-utils/tree/main/utils/@simpill-token-optimizer.utils)

npm

npm install @simpill/token-optimizer.utils

GitHub (from monorepo)

git clone https://github.com/SkinnnyJay/simpill-utils.git && cd simpill-utils/utils/@simpill-token-optimizer.utils && npm install && npm run build

Then in your project: npm install /path/to/simpill-utils/utils/@simpill-token-optimizer.utils or npm link from that directory.

import { cleanPrompt } from "@simpill/token-optimizer.utils";

const result = cleanPrompt({
  text: "  Hello   world  \n\n  tokens  ",
  options: { trim: true, collapseWhitespace: true, normalizeNewlines: true },
});
// result.output, result.appliedTransforms

import { ... } from "@simpill/token-optimizer.utils";         // Everything
import { ... } from "@simpill/token-optimizer.utils/server";  // Server (telemetry, factory)
import { ... } from "@simpill/token-optimizer.utils/client";  // Client (re-exports shared)
import { ... } from "@simpill/token-optimizer.utils/shared";  // Shared (cleaner, strategies, types)

• cleanPrompt({ text, options }) → CleanResult
• CompressionStrategy, PassthroughStrategy, JsonCompressionStrategy, etc.
• TokenOptimizer — optimizePrompt, dependencies
• createTokenOptimizer (server), createTelemetryStorage (server)
• createDefaultValidatorRegistry, createDefaultTokenizerAdapter
• Types: CompressionType, CleanOptions, TokenOptimizationResult, etc.

• LLM tokenizer implementation — Token counts and validation use a tokenizer adapter; you provide or register an implementation (e.g. tiktoken, custom). We do not bundle a specific tokenizer.
• Built-in caching — No cache layer for optimization results; cache at the call site if needed.
• Additional compression formats — Strategies cover JSON, Markdown, XML, YAML, CSV, Toon, Tonl, Passthrough; for others implement CompressionStrategy or use a separate library.

npx ts-node examples/01-basic-usage.ts

Benchmark data (CSV, TXT, JSON, MD, XML, JSONL) at configurable sizes and a no-cache performance runner:

Override iterations with BENCHMARK_ITERATIONS (e.g. BENCHMARK_ITERATIONS=10 npm run benchmark). Reports include avg ms, p95 ms, tokens/bytes saved, and compression ratio per file and compression type.

• CONTRIBUTING — Monorepo package guide.

ISC