Protean Compiler: An Agile Framework to Drive Fine-grain Phase Ordering

2026-02-05

AI Summary

This work addresses the phase-ordering problem in compiler optimization, which remains challenging due to its vast and unbounded search space and the limited generalizability of traditional hand-tuned heuristics. The authors propose a fine-grained phase-ordering framework embedded within LLVM that, for the first time, enables dynamic selection and sequencing of optimizations at the code segment level. The framework incorporates over 140 static features and supports plug-and-play integration of third-party machine learning models, including large language models, thereby overcoming the limitations of conventional global or coarse-grained approaches. Experimental results demonstrate an average speedup of 4.1% (up to 15.7%) over LLVM -O3 on the Cbench benchmark. When augmented with external models, further improvements of 10.1% and 8.5% are achieved on Susan and Jpeg, respectively, with only a few seconds of additional compilation overhead.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The phase ordering problem has been a long-standing challenge since the late 1970s, yet it remains an open problem due to having a vast optimization space and an unbounded nature, making it an open-ended problem without a finite solution, one can limit the scope by reducing the number and the length of optimizations. Traditionally, such locally optimized decisions are made by hand-coded algorithms tuned for a small number of benchmarks, often requiring significant effort to be retuned when the benchmark suite changes. In the past 20 years, Machine Learning has been employed to construct performance models to improve the selection and ordering of compiler optimizations, however, the approaches are not baked into the compiler seamlessly and never materialized to be leveraged at a fine-grained scope of code segments. This paper presents Protean Compiler: An agile framework to enable LLVM with built-in phase-ordering capabilities at a fine-grained scope. The framework also comprises a complete library of more than 140 handcrafted static feature collection methods at varying scopes, and the experimental results showcase speedup gains of up to 4.1% on average and up to 15.7% on select Cbench applications wrt LLVM's O3 by just incurring a few extra seconds of build time on Cbench. Additionally, Protean compiler allows for an easy integration with third-party ML frameworks and other Large Language Models, and this two-step optimization shows a gain of 10.1% and 8.5% speedup wrt O3 on Cbench's Susan and Jpeg applications. Protean compiler is seamlessly integrated into LLVM and can be used as a new, enhanced, full-fledged compiler. We plan to release the project to the open-source community in the near future.

Problem

• phase ordering
• compiler optimization
• fine-grain optimization
• performance modeling
• LLVM

Innovation

• phase ordering
• fine-grained optimization
• LLVM integration
• machine learning for compilers
• static feature extraction