Higher Order Automatic Differentiation of Higher Order Functions

2021-01-17

Log. Methods Comput. Sci.

AI Summary

This paper addresses the lack of semantic foundations for automatic differentiation (AD) of algebraic data types (e.g., lists, trees) and arbitrary-order derivatives in higher-order functional languages. Methodologically, it introduces the first higher-order differentiable programming framework grounded in diffeological spaces, integrating categorical semantics and logical relations to formally define the semantics of forward-mode AD for higher-order functions and rigorously prove its structural preservation and Taylor-approximation completeness. Key contributions are: (1) the first complete correctness proof of AD semantics for languages with algebraic data types; (2) the first generalization of AD to arbitrary-order derivatives, unifying the behavior of derivative selection across primitive operations; and (3) the establishment of a unique macro-characterization of higher-order AD, providing a rigorous mathematical foundation for differentiable programming.

Abstract

We present semantic correctness proofs of automatic differentiation (AD). We consider a forward-mode AD method on a higher order language with algebraic data types, and we characterise it as the unique structure preserving macro given a choice of derivatives for basic operations. We describe a rich semantics for differentiable programming, based on diffeological spaces. We show that it interprets our language, and we phrase what it means for the AD method to be correct with respect to this semantics. We show that our characterisation of AD gives rise to an elegant semantic proof of its correctness based on a gluing construction on diffeological spaces. We explain how this is, in essence, a logical relations argument. Throughout, we show how the analysis extends to AD methods for computing higher order derivatives using a Taylor approximation.

Problem

• Automatic Differentiation
• Complex Data Types
• Differential Programming

Innovation

• Automatic Differentiation
• DiffManifold Framework
• Higher-Order Derivatives