Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

2026-07-05

AI Summary

This work addresses the significant state redundancy present in deep neural networks, a phenomenon inadequately tackled by existing compression methods that fail to explicitly model the dynamic role of hidden states. The authors introduce, for the first time, the concept of controllability-observability balancing from control theory into network compression, treating a trained network as a nonlinear dynamical system. By estimating data-driven reachability and observability Gramian matrices, they compute a joint layer-wise rank and directly determine compressed layer widths accordingly. The proposed approach is both diagnostic and constructive, achieving 72.95% state and 73.48% parameter compression on MNIST with only a 1.15% accuracy drop, and 70.94% state and 83.09% parameter compression on CIFAR-10 with negligible accuracy loss, while reducing CUDA inference latency by approximately threefold.

Abstract

Deep neural networks often contain substantial hidden-state redundancy, but most compression methods operate directly on weights, neurons, or quantised representations without explicitly characterising the dynamical role of internal states. This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks. By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. The resulting A/B/C tests estimate layer-wise reachable, observable, and jointly reachable--observable ranks. These ranks are then used not only as diagnostic measures of hidden-state redundancy, but also as actual compressed layer widths for realised reduced networks. Experiments on MNIST and CIFAR-10 compare the proposed balanced realisation against projection-based reduction, unstructured pruning, structured pruning, low-rank SVD, dynamic INT8 quantisation, and linear baselines. On MNIST, a four-layer SiLU DNN is reduced from state order 1024 to 277, giving 72.95% state compression and 73.48% parameter compression, while maintaining 95.45% accuracy compared with 96.60% for the full model. On CIFAR-10, a larger SiLU DNN is reduced from state order 4608 to 1339, giving 70.94% state compression and 83.09% parameter compression, while preserving accuracy from 54.45% to 54.44% and reducing CUDA inference latency by approximately 3X. The results show that balanced reachable-observable ranks provide a principled empirical minimal-realisation criterion for designing compact neural architectures with little or no loss in accuracy.

Problem

• hidden-state redundancy
• neural network compression
• minimal realisation
• controllability-observability
• state-order reduction

Innovation

• controllability-observability
• empirical minimal realisation
• balanced Gramians
• state-order reduction
• data-driven compression