Neural Tangent Knowledge Distillation for Optical Convolutional Networks

2025-08-11

AI Summary

Hybrid optical neural networks (ONNs) suffer from insufficient training accuracy and a significant simulation-to-hardware performance gap, while existing end-to-end optimization methods lack generalizability. To address these challenges, we propose a task- and hardware-agnostic deployment framework centered on Neural Tangent Kernel-based Knowledge Distillation (NTKD). NTKD leverages an electronic neural network as a teacher model to enable pre-training accuracy estimation, in-training knowledge alignment, and post-fabrication error compensation—without requiring architecture- or task-specific redesign. The framework supports cross-platform deployment across diverse tasks, including image classification and segmentation. Evaluated on MNIST, CIFAR-10, and Carvana datasets under multiple optical hardware configurations, NTKD consistently narrows the simulation–hardware accuracy gap, achieving an average measured accuracy improvement of over 8.2%. This work provides a scalable, system-level solution toward practical ONN deployment.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Hybrid Optical Neural Networks (ONNs, typically consisting of an optical frontend and a digital backend) offer an energy-efficient alternative to fully digital deep networks for real-time, power-constrained systems. However, their adoption is limited by two main challenges: the accuracy gap compared to large-scale networks during training, and discrepancies between simulated and fabricated systems that further degrade accuracy. While previous work has proposed end-to-end optimizations for specific datasets (e.g., MNIST) and optical systems, these approaches typically lack generalization across tasks and hardware designs. To address these limitations, we propose a task-agnostic and hardware-agnostic pipeline that supports image classification and segmentation across diverse optical systems. To assist optical system design before training, we estimate achievable model accuracy based on user-specified constraints such as physical size and the dataset. For training, we introduce Neural Tangent Knowledge Distillation (NTKD), which aligns optical models with electronic teacher networks, thereby narrowing the accuracy gap. After fabrication, NTKD also guides fine-tuning of the digital backend to compensate for implementation errors. Experiments on multiple datasets (e.g., MNIST, CIFAR, Carvana Masking) and hardware configurations show that our pipeline consistently improves ONN performance and enables practical deployment in both pre-fabrication simulations and physical implementations.

Problem

• Bridging accuracy gap between optical and digital neural networks
• Addressing simulation-fabrication discrepancies in optical systems
• Developing task-agnostic optimization for diverse optical hardware

Innovation

• Neural Tangent Knowledge Distillation for model alignment
• Task-agnostic pipeline supports diverse optical systems
• Pre-fabrication accuracy estimation with user constraints