A Deep Dive into Function Inlining and its Security Implications for ML-based Binary Analysis

2025-12-15

AI Summary

This work exposes a security threat to machine learning–driven binary analysis posed by function inlining—a compiler optimization that enhances performance but severely perturbs static features (e.g., instruction sequences and control-flow graphs), degrading model robustness. We introduce the novel concept of “extreme inlining” and establish a causal analysis framework linking inlining decisions to model robustness, demonstrating that adversarial compiler configurations can generate evasion variants. Leveraging LLVM’s cost model to dissect inlining mechanics, we evaluate 20 ML models across five security tasks—including disassembly and function boundary identification—and show that minor compiler flag adjustments induce drastic inlining rate fluctuations, violating the training-deployment consistency assumption and causing significant accuracy degradation. This is the first systematic study to reveal the adversarial misuse potential of compilation optimizations in binary security analysis.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Adversarial Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

A function inlining optimization is a widely used transformation in modern compilers, which replaces a call site with the callee's body in need. While this transformation improves performance, it significantly impacts static features such as machine instructions and control flow graphs, which are crucial to binary analysis. Yet, despite its broad impact, the security impact of function inlining remains underexplored to date. In this paper, we present the first comprehensive study of function inlining through the lens of machine learning-based binary analysis. To this end, we dissect the inlining decision pipeline within the LLVM's cost model and explore the combinations of the compiler options that aggressively promote the function inlining ratio beyond standard optimization levels, which we term extreme inlining. We focus on five ML-assisted binary analysis tasks for security, using 20 unique models to systematically evaluate their robustness under extreme inlining scenarios. Our extensive experiments reveal several significant findings: i) function inlining, though a benign transformation in intent, can (in)directly affect ML model behaviors, being potentially exploited by evading discriminative or generative ML models; ii) ML models relying on static features can be highly sensitive to inlining; iii) subtle compiler settings can be leveraged to deliberately craft evasive binary variants; and iv) inlining ratios vary substantially across applications and build configurations, undermining assumptions of consistency in training and evaluation of ML models.

Problem

• Investigates security impacts of function inlining on ML-based binary analysis
• Evaluates robustness of ML models under extreme inlining scenarios
• Explores how compiler settings can create evasive binary variants

Innovation

• Investigates extreme inlining via LLVM cost model modifications
• Evaluates ML model robustness across five security analysis tasks
• Reveals inlining can evade ML models by altering static features