Fourier Preconditioning for Neural Feature Learning

2026-07-02

AI Summary

This work addresses the limitations of mutual information–based feature learning in data-scarce settings, where noisy distribution estimation and the sensitivity of the H-Score to input basis rotations hinder performance. To mitigate these issues, the authors propose using the fast Fourier transform (FFT) as a low-overhead, data-agnostic unitary preconditioner that concentrates predictive dependencies into a few dominant spectral modes, thereby reducing truncation error in finite-width networks. They further introduce two training-free metrics—spectral entropy and cumulative dependency energy—to predict the efficacy of preprocessing and guide basis selection. Experiments across eight multivariate datasets demonstrate that FFT preconditioning can reduce normalized mean squared error (NMSE) by up to 50% under resource-constrained conditions, and the proposed metrics reliably anticipate both performance gains and failure cases.

Abstract

Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental.

Problem

• mutual information
• H-Score
• feature learning
• basis rotation
• finite-width approximation

Innovation

• Fourier preconditioning
• H-Score
• spectral concentration
• unitary transformation
• training-free metrics