Singular Curvature in ReLU Training:Differentiation and the Gradient-Flow Limit Need Not Commute

2026-08-31

AI Summary

研究解决了ReLU训练中梯度下降与梯度流极限不一致的问题，通过分析状态准确性和导数差异来解释这一现象。

Abstract

Gradient descent (GD) is explicit Euler for gradient flow, but a state-accurate continuous-time surrogate need not remain accurate after differentiation. At every fixed nonresonant step size, ordinary automatic differentiation exactly differentiates the executed hard-ReLU GD program. We prove that, over a fixed finite horizon, the GD states converge and these exact discrete derivatives approach an event-free regional propagator, whereas the derivative of the limiting flow also contains speed-normalized activation-event transfers. A prepoint Stieltjes representation separates the absolutely continuous regional Hessian from atomic interface curvature; one nonzero gradient jump produces an exactly rank-one endpoint discrepancy, and global convexity prevents complete multi-event cancellation whenever an event is strict. Nevertheless, a standard family of globally 1-strongly convex residual-ReLU squared-loss risks realizes arbitrarily large reciprocal sensitivity ratios on open initialization sets, with a uniform transversality margin. The same discrete-versus-flow decomposition extends to parameters and reverse-mode adjoints; resolved smoothing in the scalar or autonomous-normal regime and consistent event localization recover the flow sensitivity. The results concern deterministic full-batch, finite-horizon dynamics with a stable finite itinerary of separated same-direction transverse events; they are consistency theorems, not prevalence claims for large-scale training.

Problem

• ReLU
• Gradient Descent
• Gradient Flow
• Automatic Differentiation
• Convexity

Innovation

• Gradient Descent
• ReLU Training
• Gradient-Flow Limit
• Stieltjes Representation
• Activation-Event Transfers