MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

2026-04-13

AI Summary

This work addresses the challenge of perceptual diversity in pixel-level language modeling across multiple languages and writing systems by introducing MIXAR, the first large-scale autoregressive pixel-level language model trained on eight distinct languages and scripts. By directly modeling raw pixels without relying on explicit tokenization, MIXAR circumvents the limitations and complexities inherent in traditional tokenizers. Experimental results demonstrate that MIXAR significantly outperforms existing pixel-level and tokenizer-based baselines on both generative and discriminative tasks. It exhibits superior performance and robustness on benchmarks such as LAMBADA and under orthographic adversarial attacks. Furthermore, scaling the model to 0.5 billion parameters enhances its cross-lingual generalization capabilities, underscoring the effectiveness of pixel-level modeling in multilingual settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges. However, the inherent perceptual diversity across languages poses a significant hurdle for multilingual generalization in pixel space. This paper introduces MIXAR, the first generative pixel-based language model trained on eight different languages utilizing a range of different scripts. We empirically evaluate MIXAR against previous pixel-based models as well as comparable tokenizer-based models, demonstrating substantial performance improvement on discriminative and generative multilingual tasks. Additionally, we show how MIXAR is robust to languages never seen during the training. These results are further strengthened when scaling the model to 0.5B parameters which not only improves its capabilities in generative tasks like LAMBADA but also its robustness when challenged with input perturbations such as orthographic attacks.

Problem

• pixel-based language models
• multilingual generalization
• script diversity
• tokenization challenges
• cross-lingual robustness

Innovation

• pixel-based language model
• multilingual generalization
• tokenization-free
• orthographic robustness
• generative modeling