mobile deployment profiling

Existing large language model (LLM) evaluation benchmarks lack coverage of mobile application development scenarios, failing to capture platform-specific constraints, framework lifecycles, and complex API interactions inherent to such environments. To address this gap, this work introduces the first multilingual repair benchmark targeting three major mobile platforms—Android Native, React Native, and Flutter—comprising 384 real-world production issues. Each task is accompanied by an executable test patch enabling automated validation of cross-file and cross-artifact modifications. Experimental results reveal that state-of-the-art code LLMs achieve end-to-end repair success rates of only 3.39%–5.21% on this benchmark, substantially lower than their performance on existing datasets, thereby exposing critical limitations in multi-file fault localization and coordinated repair capabilities.

To address computational constraints, poor energy efficiency, and insufficient real-time performance in deploying vision-language models (VLMs) on mobile devices, this work conducts a systematic end-to-end performance evaluation of three leading inference frameworks—llama.cpp, MLC-LLM, and mllm—on state-of-the-art VLMs (LLaVA, MobileVLM, and Imp) using the OnePlus 13R as the hardware platform. We develop a full-stack benchmarking toolkit measuring CPU/GPU/NPU utilization, power consumption, thermal behavior, and end-to-end latency. Our analysis uncovers a previously unreported cross-stage hardware-resource mismatch: GPU saturation during image encoding, severe CPU bottlenecks during text generation, and low, highly volatile NPU utilization. The core contribution is a hardware-aware, VLM-specific analytical methodology for mobile deployment, complemented by an open-source lightweight monitoring tool. This work provides empirically grounded insights and concrete optimization directions for efficient on-device VLM inference.

To address the need for lightweight and efficient inference of large language models (LLMs) on mobile devices, this paper introduces the first automated benchmarking framework that jointly evaluates resource efficiency and safety risks. Methodologically, it integrates weight and activation quantization configurations, supports cross-platform performance profiling across ARM CPU/GPU/NPU backends, and incorporates real-time power consumption monitoring alongside automated hallucination and toxicity detection. Contributions include: (1) a mobile-specific, multi-dimensional evaluation paradigm that unifies assessment of generation quality, latency, throughput, memory footprint, power draw, and harmful output; (2) empirical insights into nonlinear trade-offs among mobile chip energy efficiency, quantization strategies, and their impacts on latency and memory; and (3) quantitative evidence of systematic accuracy and safety degradation under model compression—establishing a reproducible, scalable evaluation standard for on-device LLM deployment.

Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory. We benchmark Qwen 2.5 1.5B (4-bit quantised) across four platforms: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, we measure throughput, latency, power, and thermal behaviour. For mobile platforms, thermal management supersedes peak compute as the primary constraint: the iPhone 16 Pro loses nearly half its throughput within two iterations, and the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely. On dedicated hardware, distinct constraints dominate: the RTX 4050 is bounded by its battery power ceiling, while the Hailo-10H is limited by on-module memory bandwidth. The RTX 4050 sustains 131.7 tok/s at 34.1 W; the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance, matching the RTX 4050 in energy proportionality at 19x lower throughput. Results should be interpreted as platform-level deployment characterisations for a single model and prompt type, reflecting hardware and software combined, rather than general claims about hardware capability alone.

Neuromorphic computing lacks a unified evaluation standard, hindering objective technology assessment, cross-method comparison, and research directionality. To address this, we propose the first open, collaborative neuromorphic computing benchmarking framework, introducing a novel dual-track evaluation paradigm—“inclusive, iterative, and community-driven”—that concurrently supports algorithm-level (hardware-agnostic) and system-level (hardware-dependent) evaluation. The framework comprises a modular toolchain, a multi-domain task suite (covering speech, vision, and time-series prediction), standardized performance measurement protocols, and an open-source evaluation interface. We release the first version of multi-task baseline results, benchmarking state-of-the-art neuromorphic models alongside conventional AI approaches across accuracy, energy efficiency, and latency. All evaluations are reproducible, cross-platform, and cross-architecture, establishing a foundational infrastructure for standardized neuromorphic computing evaluation.

This work addresses the significant performance degradation of mobile agents in real-world environments caused by interface perturbations. The authors propose AndroidReality, a framework that introduces the first systematic three-axis taxonomy of mobile interface perturbations—encompassing state, transition, and action—and establishes a robustness evaluation benchmark with controllable perturbations built upon AndroidWorld. Their key contributions include uncovering substantial robustness gaps in existing agents under four common perturbation types and introducing a training-free, test-time introspective recovery mechanism (TTIR). Experimental results demonstrate that TTIR effectively mitigates task failures in both clean and perturbed settings, thereby validating the proposed benchmark’s utility for stress testing and vulnerability identification.

This study addresses a critical gap in current AI evaluation methodologies, which often overlook the impact of low-resource deployment conditions—such as noisy inputs, limited hardware capabilities, and unstable network connectivity—on system usability. The work proposes a novel evaluation framework that treats the deployed system as the unit of assessment, integrating task performance with real-world deployment contexts across multiple dimensions. Departing from conventional leaderboard-based approaches, the framework tailors evaluation criteria to specific application categories and introduces a standardized reporting system comprising benchmark cards, deployment profiles, and failure-handling mechanisms. By balancing comparability with contextual sensitivity, this approach provides policymakers and practitioners with clear, actionable insights for informed AI deployment decisions.

This work addresses the challenge of evaluating mobile GUI agents in real-world, closed-source applications, where existing benchmarks relying on simulated or open-source environments fall short in enabling automatic and verifiable assessment. To bridge this gap, the authors introduce AndroidDaily, a large-scale benchmark comprising 94 high-frequency Android applications and 350 everyday tasks, along with GRADE, a novel evaluation framework. GRADE enables process-aware, automatic diagnosis of long-horizon interaction trajectories in closed-source apps without requiring access to internal states, by leveraging three external, observable criteria: action fidelity, output quality, and negative constraints. Experiments demonstrate that GRADE achieves 87.37% agreement with human evaluators, while the strongest existing agent attains only a 62.0% task success rate on AndroidDaily, revealing substantial limitations of current approaches in realistic settings.

This work addresses a critical gap in large language model (LLM)-driven neural architecture search for mobile deployment, where existing approaches prioritize GPU accuracy while neglecting essential constraints such as INT8 quantization, delegate selection, and on-device latency, often yielding impractical models. To overcome this, the authors propose the first “device-first” framework that integrates real-device performance feedback into an LLM-based architecture generation loop, establishing a fully automated end-to-end pipeline. This pipeline encompasses QLoRA fine-tuning for architecture proposal, GPU evaluation, INT8 TensorFlow Lite export, automatic delegate selection, on-device benchmarking, and a gating mechanism for dynamic training corpus expansion with support for interruption recovery. Experiments demonstrate that the method achieves a 25.6× improvement in mobile deployment score on CIFAR-10 within the first round (reaching 46.9% quantized accuracy), further revealing a non-monotonic relationship between GPU accuracy and on-device performance.

This work addresses the challenge that state-dependent behaviors in modern computing environments—such as those introduced by adaptive system mechanisms—induce time-dependent biases in traditional software benchmarking, undermining reliable performance comparisons. The paper reframes benchmarking as a decision problem aimed at identifying the fastest program and introduces an experimental paradigm centered on pairwise performance comparisons, thereby avoiding strong assumptions about modeling system dynamics. By leveraging contrastive estimators, consistent statistical inference, and test strategies under finite evaluation budgets, the approach eliminates program-specific biases without relying on absolute performance metrics. The method provides asymptotic guarantees for correct decisions in stateful environments, offering a robust and reliable benchmarking framework for performance-sensitive software development.