Hi-fi functional priors by learning activations

2025-08-12

AI Summary

Bayesian neural networks (BNNs) face a fundamental limitation in effectively encoding expressive function-space priors, as conventional approaches restrict priors to weight space. Method: We propose a high-fidelity function-space prior modeling framework based on learnable activation functions. By parameterizing flexible activation structures—such as Padé approximants and piecewise-linear functions—we explicitly embed complex functional priors into the network architecture, while systematically addressing identifiability, symmetry, and loss-function design challenges. Contribution/Results: Our method transcends the weight-space prior paradigm, enabling the first direct and accurate matching of target function distributions. Empirically, even a single wide hidden layer suffices to substantially enhance prior expressivity. The approach consistently outperforms baselines in regularization strength, uncertainty calibration, and risk-sensitive decision-making across diverse benchmarks.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Function-space priors in Bayesian Neural Networks (BNNs) provide a more intuitive approach to embedding beliefs directly into the model's output, thereby enhancing regularization, uncertainty quantification, and risk-aware decision-making. However, imposing function-space priors on BNNs is challenging. We address this task through optimization techniques that explore how trainable activations can accommodate higher-complexity priors and match intricate target function distributions. We investigate flexible activation models, including Pade functions and piecewise linear functions, and discuss the learning challenges related to identifiability, loss construction, and symmetries. Our empirical findings indicate that even BNNs with a single wide hidden layer when equipped with flexible trainable activation, can effectively achieve desired function-space priors.

Problem

• Learning trainable activations to impose function-space priors
• Accommodating higher-complexity priors in Bayesian Neural Networks
• Matching intricate target function distributions through optimization

Innovation

• Learning trainable activations for Bayesian Neural Networks
• Using flexible activation models like Pade functions
• Achieving function-space priors with single wide hidden layer