Online Learning of Neural Networks

2025-05-14

AI Summary

This paper studies online classification with sign-activated feedforward neural networks under unit-sphere inputs and finite label sets, aiming to overcome the strong dependence of classical mistake bounds on input dimension $d$. We introduce two novel theoretical frameworks: the *full-layer margin assumption* and the *multi-index model*. For the first time, we precisely characterize geometric learnability in online learning via the TS-packing number. Our analysis yields tight, dimension-free mistake bounds: $Theta(mathrm{TS}(d,gamma))$ in the general case; $ ilde{O}((1.5/gamma)^{k+2})$ under the multi-index model; and $ ilde{O}((log Y)/gamma^{O(L)})$ under an extended margin assumption. All bounds are accompanied by matching exponential lower bounds, substantially mitigating the curse of dimensionality. This work provides the first systematic theoretical foundation for online generalization of sign-activated networks.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

We study online learning of feedforward neural networks with the sign activation function that implement functions from the unit ball in $mathbb{R}^d$ to a finite label set ${1, ldots, Y}$. First, we characterize a margin condition that is sufficient and in some cases necessary for online learnability of a neural network: Every neuron in the first hidden layer classifies all instances with some margin $gamma$ bounded away from zero. Quantitatively, we prove that for any net, the optimal mistake bound is at most approximately $mathtt{TS}(d,gamma)$, which is the $(d,gamma)$-totally-separable-packing number, a more restricted variation of the standard $(d,gamma)$-packing number. We complement this result by constructing a net on which any learner makes $mathtt{TS}(d,gamma)$ many mistakes. We also give a quantitative lower bound of approximately $mathtt{TS}(d,gamma) geq max{1/(gamma sqrt{d})^d, d}$ when $gamma geq 1/2$, implying that for some nets and input sequences every learner will err for $exp(d)$ many times, and that a dimension-free mistake bound is almost always impossible. To remedy this inevitable dependence on $d$, it is natural to seek additional natural restrictions to be placed on the network, so that the dependence on $d$ is removed. We study two such restrictions. The first is the multi-index model, in which the function computed by the net depends only on $k ll d$ orthonormal directions. We prove a mistake bound of approximately $(1.5/gamma)^{k + 2}$ in this model. The second is the extended margin assumption. In this setting, we assume that all neurons (in all layers) in the network classify every ingoing input from previous layer with margin $gamma$ bounded away from zero. In this model, we prove a mistake bound of approximately $(log Y)/ gamma^{O(L)}$, where L is the depth of the network.

Problem

• Online learnability of feedforward neural networks with sign activation.
• Characterizing margin conditions for optimal mistake bounds in neural networks.
• Reducing dimension dependence via multi-index and extended margin models.

Innovation

• Online learning with sign activation function
• Margin condition ensures learnability
• Multi-index model reduces dimension dependence