Structure over Pixels: Learning Variable-Length Visual Programs

2026-05-26

AI Summary

Existing visual tokenizers struggle to adaptively learn structured representations and variable-length visual programs, often relying excessively on pixel-level reconstruction while neglecting high-level semantics. This work proposes the STROP architecture, which jointly optimizes structured representations and visual program length through a four-stage curriculum learning scheme under frozen DINOv2 feature supervision. A dedicated length prediction head enables effective prefix prediction in a single forward pass. STROP achieves, for the first time, continuous and image-adaptive visual program length learning, optimizing a discrete codebook solely based on high-level semantic objectives without any pixel reconstruction. The resulting program lengths adaptively scale with scene complexity, yielding strong performance on dense prediction tasks, while the learned codebook exhibits pronounced compositional structure.

Abstract

Discrete visual tokenizers translate images into ordered sequences of codes, providing a natural representation for structural description of scenes. Yet existing adaptive tokenizers either require post-hoc search or select among a discrete set of pre-trained rates, rather than learning a continuous per-image sequence length coupled to the model and scene, and they typically train against pixel reconstruction, emphasizing texture rather than structure. We propose STROP, a discrete visual tokenizer architecture that forms structural scene representations and simultaneously learns how long an image's visual program should be. Using a four-phase curriculum supervised by local rate--distortion probes against frozen DINOv3 features, STROP optimizes a dedicated length head that estimates the active prefix length in a single forward pass. By bypassing pixel-level reconstruction gradients, the codebook is shaped entirely by the quality of higher-level latent representations. Program length grows with scene complexity, and signs of compositional structure emerge both in downstream dense-prediction transfer and in direct inspection of the learned code vocabulary.

Problem

• visual tokenization
• variable-length representation
• scene structure
• discrete tokenizer
• structural representation

Innovation

• visual tokenization
• variable-length programs
• structure-aware representation
• rate-distortion curriculum
• compositional codebook