On the Sparsity-Storage-Accuracy Tradeoff in Parsimoniously Activated Dictionary Learning

2026-06-21

AI Summary

This work addresses the challenges in dictionary learning under global sparsity constraints—namely, the absence of an interpretable generative model and the difficulty in balancing sparsity, storage cost, and reconstruction accuracy. To resolve these issues, the authors propose a structured probabilistic generative model that reformulates Pruned Activation Dictionary Learning (PADL) as a maximum a posteriori estimation problem by introducing auxiliary latent variables. This approach establishes, for the first time, a probabilistic generative perspective for PADL, analytically characterizing the trade-offs among sparsity, memory efficiency, and reconstruction fidelity, while also providing generalization error bounds and enabling automatic hyperparameter estimation. Experimental results demonstrate that, at equivalent sparsity levels, the proposed method achieves superior reconstruction performance on visual benchmarks and significantly accelerates inference in vision-language models.

Abstract

Dictionary learning has long been studied from both optimization and probabilistic perspectives. While formulations with element-wise sparsity regularization (e.g., L1-based sparse coding) admit well-established probabilistic interpretations, many structured variants that impose global constraints lack a clear and tractable generative view. In this paper, we revisit a class of practically effective yet theoretically under-explored dictionary learning methods that impose a simple global regularization on the number of activated dictionary atoms, which we term parsimoniously activated dictionary learning (PADL). We show that PADL admits an equivalent formulation as maximum a posteriori estimation under a structured generative model, with auxiliary latent variables that govern global activation patterns. This formulation allows us to derive generalization guarantees that are difficult to obtain under the original formulation. More importantly, it yields an analytical characterization of the tradeoff between sparsity, storage cost, and reconstruction accuracy, enabling data-driven estimation of optimal hyperparameters. Based on this connection, we develop an efficient and interpretable PADL algorithm that eliminates manual hyperparameter tuning, achieving improved reconstruction performance under comparable sparsity levels on visual benchmarks. We further demonstrate its practical utility in accelerating inference for vision-language models.

Problem

• dictionary learning
• sparsity
• generative model
• hyperparameter estimation
• storage-accuracy tradeoff

Innovation

• parsimoniously activated dictionary learning
• structured generative model
• sparsity-storage-accuracy tradeoff
• maximum a posteriori estimation
• hyperparameter-free optimization