Sign-Symmetry Learning Rules are Robust Fine-Tuners

2025-02-09

AI Summary

This work addresses the longstanding challenge in deep neural networks of designing biologically plausible learning rules—specifically, alternatives to backpropagation (BP)—that simultaneously achieve high accuracy and strong robustness. We propose a fine-tuning paradigm based on sign-symmetric local updates: starting from a BP-pretrained model, we apply lightweight, biologically inspired sign-symmetric weight adjustments without global error propagation. To our knowledge, this is the first empirical demonstration that such updates match the classification accuracy of full BP-based fine-tuning across diverse benchmarks—including ImageNet-style image classification, adversarial robustness (e.g., under PGD attacks), out-of-distribution generalization, and noise resilience—while substantially improving robustness to input perturbations. Our approach overcomes the long-standing performance bottleneck of sign-symmetric learning rules, bridging the gap between neuroscientific plausibility and practical deep learning performance. This establishes a new pathway toward integrating biologically interpretable learning mechanisms with state-of-the-art deep learning systems.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Backpropagation (BP) has long been the predominant method for training neural networks due to its effectiveness. However, numerous alternative approaches, broadly categorized under feedback alignment, have been proposed, many of which are motivated by the search for biologically plausible learning mechanisms. Despite their theoretical appeal, these methods have consistently underperformed compared to BP, leading to a decline in research interest. In this work, we revisit the role of such methods and explore how they can be integrated into standard neural network training pipelines. Specifically, we propose fine-tuning BP-pre-trained models using Sign-Symmetry learning rules and demonstrate that this approach not only maintains performance parity with BP but also enhances robustness. Through extensive experiments across multiple tasks and benchmarks, we establish the validity of our approach. Our findings introduce a novel perspective on neural network training and open new research directions for leveraging biologically inspired learning rules in deep learning.

Problem

• Enhancing neural network robustness
• Integrating biologically plausible learning rules
• Fine-tuning pre-trained models effectively

Innovation

• Sign-Symmetry learning rules
• Fine-tuning BP-pre-trained models
• Enhancing robustness in training