Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers

2026-07-04

AI Summary

This work challenges the prevailing reliance of Chinese language models on discrete token embeddings by investigating whether effective language modeling can be achieved using only glyph images. To this end, the authors construct a dual-branch controlled framework: one branch rasterizes character sequences into images processed by a visual encoder composed of ResNet and a shallow Vision Transformer (ViT), while the other employs conventional index-based embeddings as a baseline; both branches share an identical decoder to ensure strict variable control. Experiments demonstrate for the first time that pure glyph-based input is not only viable but consistently outperforms the baseline across all decoders—achieving up to 0.429 accuracy (a 21% relative improvement), converging nearly twice as fast, and showing advantages with only 21% of the training data. The approach also exhibits greater robustness to character perturbations, revealing both the modality-agnostic capacity of Transformers and the information-rich structural properties inherent in Chinese characters.

Abstract

Modern language models generally represent text as sequences of discrete token embeddings, an assumption deeply rooted in current practice but rarely questioned. We challenge this representation, especially for Chinese, by replacing index-based token embeddings entirely with a single rasterized image of the character sequence, processed by a vision encoder composed of a shared ResNet and a shallow Vision Transformer. To isolate the role of input representation, we construct a dual-branch controlled framework in which both a Vision-based model and an index-based baseline share an identical decoder backbone, training objective, optimizer, and data curriculum. Any performance difference is therefore attributable to the input modality only. Across all tested decoder backbones, the Vision-based model consistently outperforms the baseline, reaching a peak accuracy of 0.429 versus 0.355 for the index-based baseline,that is, a 21% relative improvement, while converging in about half the number of training epochs. The advantage emerges especially within the first five epochs (under 21% of total data) and persists under moderate character corruption: the corrupted Vision model matches the clean index-based baseline. Ablation studies reveal that the advantage requires both spatially coherent input and a ViT encoder with 2D positional encodings. A cross-script comparison on English shows the advantage does not transfer directly to alphabetic writing systems, suggesting that the uniform visual density and radical structure of Chinese characters are enabling conditions. These findings suggest that transformers are more modality-agnostic than commonly assumed, and that discrete tokenization is not a fundamental requirement for Chinese language modeling.

Problem

• Chinese language modeling
• token embeddings
• glyph images
• input representation
• modality-agnostic

Innovation

• glyph image
• vision-based language modeling
• controlled study
• Chinese character representation
• modality-agnostic transformer