The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

2026-06-14

AI Summary

This work investigates whether joint multi-task modeling under orthogonality constraints offers a parameter complexity advantage over independent task modeling. By constructing an orthogonal function class composed of shared hard features and task-specific readout heads—leveraging tools such as Rademacher–Haar wavelets, Sawtooth–Walsh readout functions, and Heaviside-activated networks—the study provides the first information-theoretic proof that, under orthogonality, the description length of a joint representation is significantly shorter than that of separate representations. The theoretical analysis establishes tight upper and lower bounds for both approximation regimes, revealing the critical role of shared latent features in reducing the number of encoding bits and thereby demonstrating the expressive efficiency of compositional multi-output architectures.

Abstract

Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models. Empirically, exploiting similarity across different problems, instead of solving them individually, can significantly improve overall performance. While the generalization and sample complexity properties of multitask learning have been widely studied, the parametric complexity of joint approximation in comparison to separate approximation remains less well understood. The question is particularly relevant in modern deep learning, where models are increasingly required to satisfy structural constraints such as equivariance, conservation laws, or orthogonality. We prove lower and upper bounds on the description-length for separate and joint approximation classes, respectively, in uniform norm. We build a class of orthogonal functions by composing a shared hard feature, realized by a Rademacher-Haar wavelet series, with Sawtooth-Walsh readouts to enforce orthogonality of output coordinates. The dyadic tree structure of the Rademacher-Haar wavelet concentrates the approximation hardness in the common feature component, while the readouts act as task-specific heads. Using an information-theoretic framework, we obtain a sharp gap between the optimal approximation rates achievable by joint and separate coding. Finally, we realize this separation in a neural network model using Heaviside activations via reduction to triangle-wave approximation. Our results show that even under an orthogonality constraint joint approximation requires strictly fewer bits in compositional architectures, provided the tasks share a latent hard feature. This provides theoretical insight into the description-length-efficiency of compositional multi-output architectures and clarifies how neural networks can retain expressivity under geometric constraints.

Problem

• multitask learning
• orthogonality constraints
• description length
• shared representations
• approximation complexity

Innovation

• shared representations
• orthogonality constraints
• information-theoretic bounds
• Rademacher-Haar wavelets
• compositional architectures