Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimization

2026-05-19

AI Summary

This work investigates how to effectively transfer the performance of large models to compact ones in combinatorial optimization tasks through knowledge distillation. To this end, the authors propose a novel distillation framework based on algorithmic alignment, which explicitly aligns the architecture of graph neural networks with dynamic programming algorithms. For the first time, they establish theoretical guarantees for this distillation process: under the assumption that the teacher model admits a linear representation, and by leveraging decision tree complexity analysis, they prove that distillation can be performed efficiently with respect to the decision tree complexity of the dynamic programming transition function. By integrating graph neural networks, dynamic programming, and learning theory, this study introduces a new paradigm and provides rigorous theoretical foundations for model compression in combinatorial optimization.

Abstract

Distillation transfers knowledge from a large model trained on broad data to a smaller, more efficient model suitable for deployment. In structured prediction settings, prior knowledge about the task can guide the choice of a target architecture that is algorithmically aligned with the underlying problem. Building on recent learning-theoretic analyses of decision-tree (DT) distillation (Boix-Adsera, 2024), we study when distillation succeeds for combinatorial optimization tasks. We focus on the case where the target model is a graph neural network whose architecture is aligned with a dynamic programming (DP) algorithm for the task. Assuming that the source model is sufficiently rich, formalized through the linear representation hypothesis (LRH) (Elhage et al., 2022; Park et al., 2024), we show that the distillation problem can be solved efficiently in the complexity parameters of the DP transition function, represented as a DT. Our results provide a rigorous sufficient condition for successful distillation in the flavour of algorithmic alignment.

Problem

• distillation
• combinatorial optimization
• algorithmic alignment
• graph neural network
• dynamic programming

Innovation

• algorithmic alignment
• knowledge distillation
• combinatorial optimization
• graph neural networks
• dynamic programming