🤖 AI Summary
Under hardware resource constraints, compact deep ensembles suffer significant degradation in accuracy, calibration, uncertainty estimation, and out-of-distribution (OOD) detection. To address this, we propose Packed-Ensembles (PE): a memory-aligned, lightweight structured ensemble method that enables backbone sharing and single-pass forward propagation within the memory budget of a single model, achieved via spatial-dimension modulation encoding and grouped convolutions. PE is the first approach to compress deep ensembles into a compact, end-to-end trainable structured packing—preserving ensemble diversity and statistical performance while drastically reducing computational and memory overhead. Experiments demonstrate that PE matches standard deep ensembles in accuracy, calibration, OOD detection, and robustness to distributional shift, while accelerating both training and inference. Crucially, its memory footprint remains strictly bounded by that of a single base model.
📝 Abstract
Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties. We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds. PE is designed to operate within the memory limits of a standard neural network. Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity, and performs equally well in terms of accuracy, calibration, out-of-distribution detection, and robustness to distribution shift. We make our code available at https://github.com/ENSTA-U2IS/torch-uncertainty.
