Semantic Allocation in Ordered Bottlenecks: Predictive Residual Inference for Visual Representation Learning

2026-06-23

AI Summary

Existing ordered bottleneck methods, such as MBOP, suffer from diminishing utility of later tokens, the absence of an explicit optimization objective, and fragile performance under discrete or quantized representations. This work proposes PRIOR (Predictive Residual Inference for Ordered Representations), a framework that decouples explained and unexplained information through a log₂-scaled hierarchical structure equipped with level-wise predictors, enabling each level to focus on modeling residual errors. Departing from conventional activation-rate control, PRIOR introduces a hierarchical residual prediction mechanism that substantially enhances the robustness and performance of ordered representations across continuous, discrete, and quantized settings. Experiments demonstrate that PRIOR learns well-ordered representations—delivering coarse descriptions under low budgets and progressively refining details as budget increases—matching or exceeding baseline performance at full budget, and notably closing the gap between discrete/quantized and continuous models.

Abstract

Ordered bottlenecks aim to provide utility at flexible budgets by assigning coarse information to early tokens and task-relevant detail to later ones. Prior work, including tail dropping (TD), typically enforces ordering by means of a masking-based ordering pressure (MBOP): Late tokens are masked more frequently than early tokens and are therefore encouraged to store less essential fine details. We introduce predictive residual inference for ordered representations (PRIOR), a framework designed to address inherent weaknesses of MBOP. MBOP is prone to weak late-token utility because it lacks an explicit refinement objective and uses gradient exposure as a proxy for importance. Furthermore, representations may become particularly brittle in optimization-sensitive settings, such as when using discrete or quantized token representations. PRIOR replaces activation-rate control with log2-scaled levels and level-wise predictors. These predictors separate already explained from unexplained information, focusing each level on residual error. We compare PRIOR against MBOP-TD and independent tail-biased dropout (MBOP-ITD) in contrastive learning and image reconstruction tasks. Unlike the baselines, PRIOR learns well-ordered representations across experiments: low budgets provide coarse descriptors, while high budgets add refinements. Simultaneously, full-budget performance with PRIOR is higher in all but one experimental setting, where performance remains comparable. MBOP baselines are severely limited in discrete and quantized settings, while PRIOR approaches the performance of continuous counterparts. Taken together, these findings establish PRIOR as an effective framework for ordered representation learning.

Problem

• ordered bottlenecks
• masking-based ordering pressure
• representation learning
• discrete representations
• residual inference

Innovation

• predictive residual inference
• ordered representation learning
• bottleneck allocation
• contrastive learning
• quantized representations