RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis

2026-02-12

AI Summary

This work addresses the challenge of uniformly evaluating inference efficiency of small language models (SLMs) on resource-constrained edge devices, where objective cross-hardware benchmarks remain scarce. To this end, we propose a systematic evaluation framework grounded in the Roofline model, introducing a novel metric—relative inference potential—that leverages operational intensity (OI) to jointly characterize hardware constraints and model architecture, thereby defining distinct inference potential regions. Empirical analysis reveals how sequence length and model depth influence performance and OI, uncovers efficiency pitfalls arising from hardware heterogeneity, and demonstrates that architectural optimizations such as Multi-Head Latent Attention (MLA) can effectively unlock hardware potential. This study provides both theoretical foundations and practical guidance for hardware-software co-design in edge-side intelligence.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The transition toward localized intelligence through Small Language Models (SLMs) has intensified the need for rigorous performance characterization on resource-constrained edge hardware. However, objectively measuring the theoretical performance ceilings of diverse architectures across heterogeneous platforms remains a formidable challenge. In this work, we propose a systematic framework based on the Roofline model that unifies architectural primitives and hardware constraints through the lens of operational intensity (OI). By defining an inference-potential region, we introduce the Relative Inference Potential as a novel metric to compare efficiency differences between Large Language Models (LLMs) on the same hardware substrate. Extensive empirical analysis across diverse compute tiers reveals that variations in performance and OI are significantly influenced by sequence length. We further identify a critical regression in OI as model depth increases. Additionally, our findings highlight an efficiency trap induced by hardware heterogeneity and demonstrate how structural refinements, such as Multi-head Latent Attention (M LA), can effectively unlock latent inference potential across various hardware substrates. These insights provide actionable directions for hardware-software co-design to align neural structures with physical constraints in on-device intelligence. The released code is available in the Appendix C.

Problem

• on-device LLMs
• Roofline analysis
• performance characterization
• hardware heterogeneity
• operational intensity

Innovation

• Roofline model
• on-device LLMs
• operational intensity
• Relative Inference Potential
• hardware-software co-design