Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

2026-08-16

AI Summary

This study addresses the mismatch between theoretical compression efficacy and empirical performance in deploying large language models on edge devices, alongside the absence of practical deployment guidelines. Through extensive multi-hardware benchmarking integrating quantization, pruning, LoRA, and latency decomposition, we systematically evaluate diverse compression strategies. Our analysis reveals complex trade-offs between latency and accuracy, demonstrating that structured pruning can paradoxically increase latency by 3.4× while inducing spurious capabilities. Consequently, we distill task-specific, actionable guidelines for efficient edge deployment and release our codebase as open source. This work provides critical theoretical insights and practical references to facilitate the real-world implementation of edge AI, bridging the gap between model compression research and hardware-aware deployment optimization.

Abstract

Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation. What compresses well, however, need not deploy well. We survey dozens of recent works that report compression results on real hardware and extract practical deployment guidelines from them. Following these guidelines, we deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation. No single technique wins across tasks. For question answering, Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 at a 1% ratio. For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU. Pruning can even inflate the deployed artifact by 21-49% by breaking k-quant super-block alignment; combined with longer, less format-compliant outputs, this raises Raspberry Pi latency up to 3.4x. Compression can also manufacture the appearance of competence rather than destroy it visibly: one LoRA-recovered variant stays fully parseable and holds 71% strict BoolQ accuracy while sending 97 of 100 predictions to a single class, at 52.6% balanced accuracy. We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition, and condense them into task-specific deployment research directions. The right technique depends on the task, the model, and the hardware. Our experiment code and artifacts are open-sourced at https://github.com/Arnavvvkumar/deployment

Problem

• Edge AI Deployment
• Model Compression
• Resource-constrained Devices
• Empirical Analysis
• Large Models

Innovation

• Edge AI Deployment
• Model Compression
• Empirical Analysis
• Neural-flow Graph Analysis
• Latency Decomposition