Skip to content

Token-estimation calibration

Divergence of the dependency-free heuristic (HeuristicEstimator, heuristic/v2) and the tiktoken default from real tokenizers, across fixed corpus shapes. Generated by benchmarks/token_calibration.py (issue #493).

  • Generated: 2026-06-11
  • tiktoken model: cl100k_base
  • ⚠️ tiktoken encoding unavailable when generated (offline / no warmed TIKTOKEN_CACHE_DIR). The tiktoken and relative-error columns read n/a; regenerate in an environment with a warmed cache to populate them.
  • Provider tokenizers (Anthropic / Gemini): not run (opt-in via CW_TOKEN_CALIBRATION_PROVIDERS + a registered counter; never run in CI).
Shape Chars Heuristic tiktoken Heuristic vs tiktoken
prose_en 132 32 n/a n/a
prose_cjk 96 88 n/a n/a
json 144 36 n/a n/a
code 123 30 n/a n/a
logs 147 36 n/a n/a

Reading the table: a large negative Heuristic vs tiktoken error on prose_cjk is the failure mode issue #525 fixes — the script-aware heuristic keeps CJK within roughly ±30% of tiktoken instead of under-counting ~4×. For Latin prose, JSON, code, and logs the heuristic tracks len // 4, which is adequate for budget headroom decisions.