Stochastic Forward-Forward Learning through Representational Dimensionality Compression

2025-05-22

AI Summary

Existing energy-based learning (EBL) “goodness” functions rely solely on the sum of squared postsynaptic activations, neglecting inter-neuronal correlations and limiting Forward-Forward (FF) algorithm performance. To address this, we propose Stochastic Forward-Forward (SFF): the first FF framework to employ **effective dimension (ED)** as a layer-wise goodness metric, explicitly modeling the second-order statistical structure of neural responses and treating noise as an explicit regularizer for generalization. SFF eliminates backpropagation and negative sampling entirely, enabling noise-driven representation learning. Our method integrates ED estimation, stochastic dynamical modeling, and layerwise goodness optimization, augmented by mean energy-term prediction for enhanced stability. On standard benchmarks, SFF achieves state-of-the-art performance among non-backpropagation methods, empirically demonstrating that injected noise significantly improves generalization and inference robustness. The code is open-sourced and designed for neuromorphic hardware deployment.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

The Forward-Forward (FF) algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise"goodness"function to guide learning. Existing goodness functions, inspired by energy-based learning (EBL), are typically defined as the sum of squared post-synaptic activations, neglecting the correlations between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for clamped inputs when noise is considered while maximizing it across the sample distribution, promoting structured representations without the need to prepare negative samples. We demonstrate that this formulation achieves competitive performance compared to other non-BP methods. Moreover, we show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean of squared outputs, which is equivalent to making predictions based on the energy term. Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is a computational resource rather than a nuisance. The code is available at https://github.com/ZhichaoZhu/StochasticForwardForward

Problem

• Improves Forward-Forward algorithm with dimensionality compression
• Enhances neural representation using second-order statistical structure
• Demonstrates noise benefits for generalization and inference

Innovation

• Dimensionality compression goodness function for learning
• Effective dimensionality minimizes noise, maximizes sample structure
• Noise enhances generalization via mean squared outputs