Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs

2026-02-10

AI Summary

This work addresses the challenge of deploying large language models (LLMs) on edge devices, where balancing accuracy, latency, and hardware efficiency remains difficult due to the lack of systematic hardware-software co-design. We propose the first hardware-aware scaling law tailored for edge LLMs, jointly modeling training loss and the Roofline performance model to establish a mapping between architectural parameters and the accuracy–latency trade-off. By training 170 models—each on 10 billion tokens—across 1,942 diverse architectures, we fit this scaling law to enable efficient architecture search, reducing design cycles from months to days and revealing the Pareto frontier of accuracy versus latency. At parity with the latency of Qwen2.5-0.5B, our selected architecture achieves a 19.42% lower perplexity on WikiText-2, significantly enhancing both performance and efficiency of edge-deployed LLMs.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language-Action Models (VLAs) have emerged as a key paradigm of Physical AI and are increasingly deployed in autonomous vehicles, robots, and smart spaces. In these resource-constrained on-device settings, selecting an appropriate large language model (LLM) backbone is a critical challenge: models must balance accuracy with strict inference latency and hardware efficiency constraints. This makes hardware-software co-design a game-changing requirement for on-device LLM deployment, where each hardware platform demands a tailored architectural solution. We propose a hardware co-design law that jointly captures model accuracy and inference performance. Specifically, we model training loss as an explicit function of architectural hyperparameters and characterise inference latency via roofline modelling. We empirically evaluate 1,942 candidate architectures on NVIDIA Jetson Orin, training 170 selected models for 10B tokens each to fit a scaling law relating architecture to training loss. By coupling this scaling law with latency modelling, we establish a direct accuracy-latency correspondence and identify the Pareto frontier for hardware co-designed LLMs. We further formulate architecture search as a joint optimisation over precision and performance, deriving feasible design regions under industrial hardware and application budgets. Our approach reduces architecture selection from months to days. At the same latency as Qwen2.5-0.5B on the target hardware, our co-designed architecture achieves 19.42% lower perplexity on WikiText-2. To our knowledge, this is the first principled and operational framework for hardware co-design scaling laws in on-device LLM deployment. We will make the code and related checkpoints publicly available.

Problem

• hardware-software co-design
• on-device LLMs
• inference latency
• model accuracy
• resource-constrained deployment

Innovation

• hardware-software co-design
• scaling laws
• Roofline modeling
• on-device LLMs
• Pareto frontier