Compact Circulant Layers with Spectral Priors

2026-02-25

AI Summary

This work addresses the need for memory-efficient, uncertainty-aware compact neural networks on edge devices in safety-critical domains such as healthcare and robotics. The authors propose a frequency-domain parameterized compact circulant convolutional layer (BCCB), which, for the first time, discretizes the spectral representation of stationary kernels by directly modeling weights in the real FFT half-spectrum. They introduce a Hermitian-symmetric complex Gaussian process prior over filters and design a low-rank plus diagonal structured variational posterior. This formulation enables exact spectral norm computation and global Lipschitz constant estimation. Experiments demonstrate that the model matches or exceeds strong baselines on tasks including MNIST→Fashion-MNIST transfer, CIFAR-10 feature head prediction, and ViT projection, while using significantly fewer parameters and yielding tighter robustness certificates.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Critical applications in areas such as medicine, robotics and autonomous systems require compact (i.e., memory efficient), uncertainty-aware neural networks suitable for edge and other resource-constrained deployments. We study compact spectral circulant and block-circulant-with-circulant-blocks (BCCB) layers: FFT-diagonalizable circular convolutions whose weights live directly in the real FFT (RFFT) half (1D) or half-plane (2D). Parameterizing filters in the frequency domain lets us impose simple spectral structure, perform structured variational inference in a low-dimensional weight space, and calculate exact layer spectral norms, enabling inexpensive global Lipschitz bounds and margin-based robustness diagnostics. By placing independent complex Gaussians on the Hermitian support we obtain a discrete instance of the spectral representation of stationary kernels, inducing an exact stationary Gaussian-process prior over filters on the discrete circle/torus. We exploit this to define a practical spectral prior and a Hermitian-aware low-rank-plus-diagonal variational posterior in real coordinates. Empirically, spectral circulant/BCCB layers are effective compact building blocks in both (variational) Bayesian and point estimate regimes: compact Bayesian neural networks on MNIST->Fashion-MNIST, variational heads on frozen CIFAR-10 features, and deterministic ViT projections on CIFAR-10/Tiny ImageNet; spectral layers match strong baselines while using substantially fewer parameters and with tighter Lipschitz certificates.

Problem

• compact neural networks
• uncertainty-aware
• edge deployment
• resource-constrained
• Bayesian neural networks

Innovation

• spectral circulant layers
• BCCB
• frequency-domain parameterization
• structured variational inference
• Lipschitz certification