Training Across Reservoirs: Using Numerical Differentiation To Couple Trainable Networks With Black-Box Reservoirs

2025-10-28

AI Summary

This paper addresses the challenge of enhancing model performance without increasing trainable parameters and under the constraint that the computational graph of black-box modules—such as fixed or untrained networks—is inaccessible. To this end, we propose Bounded Numerical Differentiation (BOND), a method grounded in bounded numerical differentiation and perturbation-based gradient estimation. BOND enables end-to-end coupling between trainable networks and black-box modules without requiring architectural modification or explicit gradient modeling of the black-box component, thereby unlocking its representational capacity without optimization. Crucially, it circumvents costly gradient approximation schemes, substantially reducing computational overhead. Experiments demonstrate that BOND significantly outperforms existing black-box integration methods on tasks including image classification; notably, it achieves accuracy gains even when integrating untrained networks. The approach delivers high performance, parameter efficiency, and strong scalability—establishing a novel paradigm for analog-digital hybrid modeling and lightweight neural architecture design.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce Bounded Numerical Differentiation (BOND), a perturbative method for estimating partial derivatives across network structures with inaccessible computational graphs. BOND demonstrates improved accuracy and scalability from existing perturbative methods, enabling new explorations of trainable architectures that integrate black-box functions. We observe that these black-box functions, realized in our experiments as fixed, untrained networks, can enhance model performance without increasing the number of trainable parameters. This improvement is achieved without extensive optimization of the architecture or properties of the black-box function itself. Our findings highlight the potential of leveraging fixed, non-trainable modules to expand model capacity, suggesting a path toward combining analogue and digital devices as a mechanism for scaling networks.

Problem

• Estimating derivatives across networks with inaccessible computational graphs
• Integrating black-box functions into trainable architectures effectively
• Enhancing model performance without increasing trainable parameters count

Innovation

• BOND method estimates derivatives via numerical differentiation
• Couples trainable networks with black-box reservoir components
• Uses fixed untrained networks to enhance model performance