SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding

2026-01-14

AI Summary

This work addresses the poor performance of large language models (LLMs) on character-level tasks—such as letter counting—attributed primarily to their subword tokenization schemes and the lack of evaluation benchmarks grounded in real-world applications. To bridge this gap, the authors introduce a practical benchmark for subword comprehension, comprising ten decoupled complex reasoning tasks across four domains, which uniquely integrates subword processing capabilities with realistic scenarios. Through multi-domain task design, test-time scaling analyses, and probing of character-level information in hidden states, the study systematically evaluates nine prominent LLMs, quantifying their subword understanding deficiencies. The findings reveal that test-time scaling yields limited performance gains, underscoring the critical impact of tokenization mechanisms on model efficacy in authentic tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.

Problem

• sub-token understanding
• tokenization
• character-level tasks
• large language models
• practical benchmark

Innovation

• SubTokenTest
• sub-token understanding
• tokenization
• practical benchmark
• character-level reasoning