research area Speech and Natural Language Processingconference ACL
content type paperpublished June 2025

Beyond Text Compression: Evaluating Tokenizers Across Scales

AuthorsJonas F. Lotz†‡, António Vilarinho Lopes, Stephan Peitz, Hendra Setiawan, Leonardo Emili

View publication

Tokenizer design significantly impacts language model performance, yet evaluating tokenizer quality remains challenging. While text compression has emerged as a common intrinsic metric, recent work questions its reliability as a quality indicator. We investigate whether evaluating tokenizers on smaller models (350M parameters) reliably predicts their impact at larger scales (2.7B parameters). Through experiments with established tokenizers from widely-adopted language models, we find that tokenizer choice minimally affects English tasks but yields significant, scale-consistent differences in machine translation performance. Based on these findings, we propose additional intrinsic metrics that correlate more strongly with downstream performance than text compression. We combine these metrics into an evaluation framework that enables more reliable intrinsic tokenizer comparisons.

  • † Work done while at Apple
  • ‡ University of Copenhagen & ROCKWOOL Foundation Research Unit

Related readings and updates.

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length

February 19, 2025research area Computer Vision

This work was done in collaboration with Swiss Federal Institute of Technology Lausanne (EPFL).

Image tokenization has enabled major advances in autoregressive image generation by providing compressed, discrete representations that are more efficient to process than raw pixels. While traditional approaches use 2D grid tokenization, recent methods like TiTok have shown that 1D tokenization can achieve high generation quality by eliminating grid...

Read more

Training a Tokenizer for Free with Private Federated Learning

May 23, 2022research area Privacy, research area Speech and Natural Language Processingconference ACL

Federated learning with differential privacy, i.e. private federated learning (PFL), makes it possible to train models on private data distributed across users’ devices without harming privacy. PFL is efficient for models, such as neural networks, that have a fixed number of parameters, and thus a fixed-dimensional gradient vector. Such models include neural-net language models, but not tokenizers, the topic of this work. Training a tokenizer...

Read more