SymTorch: A Framework for Symbolic Distillation of Deep Neural Networks

2026-02-24

AI Summary

Although symbolic distillation can transform neural network components into interpretable closed-form mathematical expressions, its engineering complexity has hindered widespread adoption in deep learning. This work proposes SymTorch, a library that enables general-purpose symbolic distillation for mainstream architectures—including GNNs, PINNs, and Transformers—for the first time. By capturing input-output behaviors of network components and leveraging PySR for symbolic regression, SymTorch automatically generates human-readable expressions while supporting seamless switching between neural and symbolic forward propagation. Key technical contributions include optimized GPU-CPU data transfer, caching mechanisms, and model serialization. Experiments demonstrate that replacing MLP layers in large language models with their symbolic surrogates improves inference throughput by 8.3% with negligible performance degradation.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Symbolic distillation replaces neural networks, or components thereof, with interpretable, closed-form mathematical expressions. This approach has shown promise in discovering physical laws and mathematical relationships directly from trained deep learning models, yet adoption remains limited due to the engineering barrier of integrating symbolic regression into deep learning workflows. We introduce SymTorch, a library that automates this distillation by wrapping neural network components, collecting their input-output behavior, and approximating them with human-readable equations via PySR. SymTorch handles the engineering challenges that have hindered adoption: GPU-CPU data transfer, input-output caching, model serialization, and seamless switching between neural and symbolic forward passes. We demonstrate SymTorch across diverse architectures including GNNs, PINNs and transformer models. Finally, we present a proof-of-concept for accelerating LLM inference by replacing MLP layers with symbolic surrogates, achieving an 8.3\% throughput improvement with moderate performance degradation.

Problem

• symbolic distillation
• symbolic regression
• deep neural networks
• interpretable models
• engineering barrier

Innovation

• symbolic distillation
• interpretable AI
• PySR
• neural-symbolic integration
• LLM acceleration