Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression

2024-10-08

arXiv.org

AI Summary

This work addresses the poor training stability and high sensitivity to hyperparameters in Adaptive Random Fourier Features (ARFF) for shallow neural networks. We propose a novel ARFF training algorithm based on particle filter resampling, which replaces the conventional Metropolis-Hastings acceptance test with a resampling mechanism to adaptively update feature frequencies—eliminating one critical hyperparameter and significantly improving convergence robustness. To our knowledge, this is the first work to extend ARFF to image coordinate regression tasks, enabling fully automated frequency optimization of the RFF layer within coordinate-based MLPs. Experiments demonstrate that the proposed method enhances RFF representation capability on both function and image regression benchmarks; when used as pretraining, it accelerates subsequent gradient-based optimization; and per-iteration computational cost is lower than standard ARFF, yielding more stable empirical performance.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

This paper presents an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks, building upon the work introduced in"Adaptive Random Fourier Features with Metropolis Sampling", Kammonen et al., emph{Foundations of Data Science}, 2(3):309--332, 2020. This improved method uses a particle filter-type resampling technique to stabilize the training process and reduce the sensitivity to parameter choices. The Metropolis test can also be omitted when resampling is used, reducing the number of hyperparameters by one and reducing the computational cost per iteration compared to the ARFF method. We present comprehensive numerical experiments demonstrating the efficacy of the proposed algorithm in function regression tasks as a stand-alone method and as a pretraining step before gradient-based optimization, using the Adam optimizer. Furthermore, we apply the proposed algorithm to a simple image regression problem, illustrating its utility in sampling frequencies for the random Fourier features (RFF) layer of coordinate-based multilayer perceptrons. In this context, we use the proposed algorithm to sample the parameters of the RFF layer in an automated manner.

Problem

• Stabilize ARFF training with resampling for neural networks
• Reduce hyperparameters and computational cost in ARFF method
• Apply ARFF to image regression via automated RFF sampling

Innovation

• Particle filter resampling stabilizes ARFF training
• Omits Metropolis test to reduce hyperparameters
• Automates RFF layer parameter sampling