A Comprehensive Evaluation of Vertex Elimination Algorithms for Algorithmic Differentiation

2026-05-22

AI Summary

This study addresses the optimal vertex elimination problem—a core NP-complete challenge in algorithmic differentiation that has long lacked scalable exact solvers for evaluating heuristic performance. The work models the problem as a vertex elimination sequence on directed acyclic graphs and introduces a novel integer programming formulation capable of solving instances one to two orders of magnitude larger than previous approaches. It establishes the first approximate lower bounds for both minimum fill-in and minimum operation count variants and designs a parameterized approximation algorithm based on minimum s-t cuts. Additionally, the paper provides tight theoretical analyses for both forward and reverse modes of automatic differentiation. A new medium-scale graph benchmark with known optimal solutions is introduced, enabling empirical validation that demonstrates the strong practical performance of state-of-the-art heuristics and significantly advances both the tractable scale and theoretical understanding of the problem.

Abstract

The algorithmic differentiation (AD) of mathematical functions can be interpreted as a sequence of vertex eliminations in an underlying directed acyclic graph. The problem of determining a minimum-cost elimination ordering, which we call Optimal Vertex Elimination, is NP-complete. Consequently, much effort has been devoted to the design of heuristics. Many of these heuristics are widely believed to perform well in practice, but this hypothesis has so far been difficult to test due to the lack of scalable exact methods. We design and engineer new integer programming formulations for Optimal Vertex Eliminatioin and for a related objective we call Minimum Edge Count. Our implementations scale to graphs one-to-two orders of magnitude larger than existing techniques, enabling the assembly of a corpus of medium-sized graphs for which optimal solutions are known. This corpus facilitates a study of existing heuristics, confirming that on real data popular methods achieve high quality solutions. We also make several theoretical contributions. We give a tight analysis of the forward and reverse modes of AD, and extend our techniques to provide a simple algorithm for Optimal Vertex Elimination with approximation ratio parameterized by the size of a minimum source-sink separator. On the complexity side, we give the first approximation lower bounds for both problems.

Problem

• Optimal Vertex Elimination
• Algorithmic Differentiation
• Minimum Edge Count
• Vertex Elimination Ordering
• Directed Acyclic Graph

Innovation

• Optimal Vertex Elimination
• Algorithmic Differentiation
• Integer Programming
• Approximation Algorithm
• Minimum Edge Count