embedded systems and on-device ai hardware

The rapid deployment of machine learning across platforms from milliwatt-class TinyML devices to large language models has made energy efficiency a primary constraint for sustainable AI. Across these scales, performance and energy are increasingly limited by data movement and memory-system behavior rather than by arithmetic throughput alone. This work reviews energy efficient software hardware codesign methods spanning edge inference and training to datacenter-scale LLM serving, covering accelerator architectures (e.g., ASIC/FPGA dataflows, processing-/compute-in-memory designs) and system-level techniques (e.g., partitioning, quantization, scheduling, and runtime adaptation). We distill common design levers and trade-offs, and highlight recurring gaps including limited cross-platform generalization, large and costly co-design search spaces, and inconsistent benchmarking across workloads and deployment settings. Finally, we outline a hierarchical decomposition perspective that maps optimization strategies to computational roles and supports incremental adaptation, offering practical guidance for building energy and carbon aware ML systems.

To address inefficient N:M sparse DNN inference on resource-constrained microcontrollers (MCUs), this paper proposes a hardware-software co-design acceleration framework. It introduces a lightweight RISC-V instruction set extension supporting indirect loading and index decompression, develops high-efficiency sparse GEMM and convolution kernels, and integrates end-to-end sparse compilation into TVM. The key contribution is the first complete N:M sparse computing stack tailored for RISC-V MCUs, balancing hardware extensibility with software deployability. Experimental results show that the sparse kernels achieve 2.1×–3.4× speedup over dense baselines; the custom instructions further improve performance by 1.9×. End-to-end inference acceleration reaches 3.21× for ResNet-18 and 1.81× for ViT, with negligible accuracy degradation (<1.5% top-1 error).

To address the low energy efficiency and poor adaptability of deploying deep learning models on resource-constrained FPGAs in IoT edge scenarios, this paper proposes a co-optimized, automated accelerator generation methodology. Our approach systematically embeds application-specific knowledge into the RTL synthesis flow for the first time, jointly optimizing RTL-level customization, load-aware scheduling, and semantics-aware mapping. Guided by design space exploration (DSE), the automated framework achieves joint optimization of energy efficiency and real-time performance under strict hardware resource constraints. Evaluated on Xilinx UltraScale+ FPGAs, the generated accelerators achieve an average 2.3× improvement in energy efficiency over baseline implementations, meet real-time inference latency requirements, and maintain FPGA resource utilization at or below 92%.

This work addresses the challenges of deploying Transformer models on resource-constrained edge devices, where computational complexity, memory footprint, and power consumption pose significant bottlenecks. The study systematically evaluates lightweight Transformer architectures alongside optimization strategies—including compression, quantization, pruning, and knowledge distillation—and integrates sparse attention mechanisms, mixed-precision quantization (INT8/FP16), and hardware-aware neural architecture search to enable efficient deployment within frameworks such as TensorFlow Lite and CoreML. A proposed six-step deployment pipeline achieves 4–10× model compression and 3–9× latency reduction at a power budget of 2–5 W, with accuracy degradation below 2% (retaining 75–96% of original accuracy). The analysis further uncovers a consistent memory bandwidth bottleneck, revealing that models with 15–40 million parameters attain 60–75% hardware utilization on mainstream edge platforms.

This study systematically investigates the impact of deep learning runtime configurations—specifically engine–execution provider pairings (e.g., PyTorch vs. ONNX Runtime, CUDA vs. CPU)—on energy efficiency, latency, and resource utilization during inference for small language models (SLMs) specialized for code. Leveraging a standardized benchmarking framework, we evaluate 12 open-source code SLMs using RAPL-based power measurement, system-level performance counters, and a unified inference API. To our knowledge, this is the first empirical, cross-engine, cross-provider energy-efficiency comparison for code SLMs. Results show that PyTorch with CUDA delivers the best overall trade-off: it reduces energy consumption by 37.99%–89.16% versus all other configurations while achieving lower latency and higher GPU utilization. For CPU-only deployment, ONNX Runtime with CPU execution improves energy efficiency by 8.98%–72.04% over comparable CPU-based alternatives. The study provides reproducible, evidence-based guidelines for energy-aware SLM deployment in production environments.

This work addresses the high energy consumption and hardware dependency of large models on resource-constrained microcontrollers by proposing an integrated compression and deployment methodology that efficiently adapts FastGRNN to 8/16-bit MCUs lacking hardware multipliers. The approach features low-rank weight decomposition, iterative hard-thresholding sparsification, Q15 post-training quantization, activation calibration, and a novel lookup-table-based acceleration mechanism for sigmoid and tanh functions tailored to multiplier-less architectures, enabling bit-accurate deterministic inference across platforms. The resulting model occupies only 566 bytes and achieves a macro F1 score of 0.918 on the HAPT dataset. It enables real-time 50 Hz inference in 9.21 ms on Arduino and 13 ms on MSP430, with the lookup-table method yielding a 30.5× speedup and reducing energy consumption by 96.7%.

This study addresses the challenges of deploying large language models with over 70 billion parameters on consumer-grade hardware, where performance, memory capacity, and energy efficiency are constrained by stark architectural differences between NVIDIA and Apple Silicon platforms. The work presents a systematic evaluation of local inference performance across both ecosystems, uncovering for the first time the “backend dichotomy” and “memory wall” phenomena in TensorRT-LLM. It further quantifies the scalability and energy efficiency advantages of Apple’s Unified Memory Architecture (UMA). Empirical analyses employing NVFP4/BF16 quantization, CPU offloading, and 4-bit inference reveal that NVIDIA’s NVFP4 format boosts throughput by 1.6× at the cost of increased cold-start latency, whereas Apple’s UMA enables near-linear scaling up to 80B-parameter models and achieves 23× higher energy efficiency than NVIDIA counterparts.

This work addresses the limitations of existing deep learning frameworks, which rely on static sequential execution models that hinder efficient exploitation of intra-device parallelism, resulting in suboptimal hardware utilization and requiring extensive model-specific code to integrate parallelization strategies. To overcome this, the paper introduces a novel paradigm that decouples logical model definition from physical execution scheduling. By leveraging frontend annotations and a programmable scheduling interface, developers can flexibly specify parallelism strategies without altering the model architecture. The system integrates graph partitioning, asynchronous control flow, and zero-copy memory management, while remaining compatible with optimizations such as TorchInductor and CUDA Graphs. Evaluated across six mainstream models, the approach enables integration of common parallelization schemes with minimal code changes and achieves up to a 1.29× throughput improvement. The implementation is publicly released.

This work addresses the challenge of deploying spiking neural networks (SNNs) on low-power FPGAs while preserving consistency between PyTorch-style development and hardware execution. The authors propose a semantics-preserving hardware-software co-design framework that leverages unified metadata artifacts to bridge software specification and board-level implementation. For the first time, this approach enables end-to-end reproducible deployment of PyTorch-defined SNNs onto cost-effective FPGAs with full behavioral equivalence between software and hardware. Implemented with time-to-first-spike (TTFS) encoding, an event-driven architecture, and an 80 MHz routing design, the system achieves 87.40% accuracy on MNIST, with a per-image latency of only 0.1375 µs and dynamic energy consumption of 31.6 nJ. Crucially, it produces results identical to the software reference across all 10,000 test images.

Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory. We benchmark Qwen 2.5 1.5B (4-bit quantised) across four platforms: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, we measure throughput, latency, power, and thermal behaviour. For mobile platforms, thermal management supersedes peak compute as the primary constraint: the iPhone 16 Pro loses nearly half its throughput within two iterations, and the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely. On dedicated hardware, distinct constraints dominate: the RTX 4050 is bounded by its battery power ceiling, while the Hailo-10H is limited by on-module memory bandwidth. The RTX 4050 sustains 131.7 tok/s at 34.1 W; the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance, matching the RTX 4050 in energy proportionality at 19x lower throughput. Results should be interpreted as platform-level deployment characterisations for a single model and prompt type, reflecting hardware and software combined, rather than general claims about hardware capability alone.