Compilation of Generalized Matrix Chains with Symbolic Sizes

2025-11-25

AI Summary

This paper addresses the core challenge of efficiently generating optimized code for Generalized Matrix Chain (GMC) multiplication under *symbolic dimensions*—i.e., when tensor sizes are unknown at compile time. Conventional approaches assume fixed, concrete dimensions and thus fail to guarantee computational optimality across arbitrary dimension combinations. To resolve this, we propose a *multi-variant code generation framework*, the first to integrate symbolic dimension modeling into GMC optimization. Our approach combines symbolic reasoning, BLAS/LAPACK kernel scheduling, and runtime dimension-aware decision-making: it generates multiple candidate execution paths at compile time and selects, at runtime, the variant minimizing floating-point operations (FLOPs). We provide theoretical guarantees that the selected variant is near-globally optimal for all valid dimension assignments. Experimental evaluation shows that, for 95% of test cases, both FLOP count and execution time increase by less than 15% compared to the optimal static schedule—substantially outperforming state-of-the-art methods.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

Generalized Matrix Chains (GMCs) are products of matrices where each matrix carries features (e.g., general, symmetric, triangular, positive-definite) and is optionally transposed and/or inverted. GMCs are commonly evaluated via sequences of calls to BLAS and LAPACK kernels. When matrix sizes are known, one can craft a sequence of kernel calls to evaluate a GMC that minimizes some cost, e.g., the number of floating-point operations (FLOPs). Even in these circumstances, high-level languages and libraries, upon which users usually rely, typically perform a suboptimal mapping of the input GMC onto a sequence of kernels. In this work, we go one step beyond and consider matrix sizes to be symbolic (unknown); this changes the nature of the problem since no single sequence of kernel calls is optimal for all possible combinations of matrix sizes. We design and evaluate a code generator for GMCs with symbolic sizes that relies on multi-versioning. At compile-time, when the GMC is known but the sizes are not, code is generated for a few carefully selected sequences of kernel calls. At run-time, when sizes become known, the best generated variant for the matrix sizes at hand is selected and executed. The code generator uses new theoretical results that guarantee that the cost is within a constant factor from optimal for all matrix sizes and an empirical tuning component that further tightens the gap to optimality in practice. In experiments, we found that the increase above optimal in both FLOPs and execution time of the generated code was less than 15% for 95% of the tested chains.

Problem

• Optimizing matrix chain computations with symbolic unknown sizes
• Generating multi-version code for generalized matrix operations
• Selecting optimal kernel sequences at runtime for matrix chains

Innovation

• Generates multi-version code for symbolic matrix chains
• Uses compile-time selection of kernel call sequences
• Guarantees near-optimal performance via empirical tuning