Lyapunov-Guided Training for Hardware-Safe Neural Networks Under Fixed-Point Arithmetic

2026-07-05

AI Summary

Fixed-point two’s complement overflow can severely distort the sign and magnitude of neural network hidden activations, leading to numerical instability and abrupt accuracy degradation. This work introduces Lyapunov stability theory into low-precision training for the first time, proposing a hardware-aware monotonic projection mechanism that enforces bounded and non-increasing evolution of hidden states across network depth by monitoring layer-wise Lyapunov functions as proxies for state energy. Integrated with fixed-point quantization, quantization-aware training, and a lightweight block-wise Transformer architecture, the method achieves 86.55% accuracy at 12-bit precision with an activation overflow rate below 0.012%, substantially outperforming conventional approaches—whose performance remains near random levels across 6–16 bits.

Abstract

Low-precision neural networks are attractive for resource-constrained hardware, but fixed-point arithmetic introduces failure modes that are often hidden by idealised quantisation models. In particular, two's-complement overflow wrapping can corrupt hidden activations by changing both their magnitude and sign, leading to unstable numerical error propagation and severe accuracy degradation. This paper proposes a Lyapunov-stabilised quantisation framework for low-precision neural networks operating under hardware-style wrapping arithmetic. The hidden-state energy is monitored through a layerwise Lyapunov function, and a monotone projection is applied to enforce bounded and non-increasing state evolution across depth. The method is evaluated on MNIST using a compact patch-based transformer under post-training quantisation and quantisation-aware training with fixed-point bit-widths from 4 to 16 bits. Monte Carlo results show that unconstrained wrapped quantisation-aware training collapses to near-chance accuracy across 6-16 bits, with activation overflow rates exceeding 11%. In contrast, the proposed monotone Lyapunov projection suppresses activation overflow to below 0.012% and restores stable low-precision learning, achieving 86.55% accuracy at 12 bits. These results demonstrate that Lyapunov-based state control can act as a hardware-aware stabilisation mechanism for reliable fixed-point neural inference and training.

Problem

• fixed-point arithmetic
• two's-complement overflow
• activation corruption
• numerical instability
• low-precision neural networks

Innovation

• Lyapunov stability
• fixed-point arithmetic
• quantization-aware training
• activation overflow
• hardware-safe neural networks