tensorflow lite

This work addresses the challenges of deploying Transformer models on resource-constrained edge devices, where computational complexity, memory footprint, and power consumption pose significant bottlenecks. The study systematically evaluates lightweight Transformer architectures alongside optimization strategies—including compression, quantization, pruning, and knowledge distillation—and integrates sparse attention mechanisms, mixed-precision quantization (INT8/FP16), and hardware-aware neural architecture search to enable efficient deployment within frameworks such as TensorFlow Lite and CoreML. A proposed six-step deployment pipeline achieves 4–10× model compression and 3–9× latency reduction at a power budget of 2–5 W, with accuracy degradation below 2% (retaining 75–96% of original accuracy). The analysis further uncovers a consistent memory bandwidth bottleneck, revealing that models with 15–40 million parameters attain 60–75% hardware utilization on mainstream edge platforms.

To address the high latency and energy consumption of Tensor Train Decomposition (TTD) on edge devices—caused by repeated SVD computations and matrix multiplications—this paper proposes TT-Edge, a hardware-software co-design framework. Methodologically, it decouples SVD into two stages—bidiagonalization and diagonalization—enabling reuse of the GEMM accelerator and integrating a shared floating-point unit to minimize data movement overhead. A dedicated TTD engine is designed atop the RISC-V architecture and tightly coupled with the GEMM module, validated on both FPGA prototypes and 45 nm backend synthesis. Experimental results on compressing ResNet-32 demonstrate a 1.7× speedup over a pure GEMM-based baseline, a 40.2% improvement in system-level energy efficiency, and only a 4% increase in hardware area. TT-Edge thus achieves efficient, low-power, and area-efficient TTD acceleration for edge deployment.

Edge TPUs suffer from limited on-chip memory, causing frequent host memory accesses and severely constraining end-to-end throughput for multi-model inference. To address this bottleneck, we propose a fine-grained model partitioning and cross-TPU pipelined inference method guided by performance profiling. Our approach tightly integrates layer-wise performance profiling, graph-structure-aware model splitting, multi-device coordinated pipelined scheduling, and Edge TPU compiler optimizations—enabling dynamic, layer-specific computational load distribution and joint optimization of memory and bandwidth resources. Evaluated on four Edge TPUs, our method achieves 46× and 6× end-to-end inference speedup for fully connected and convolutional networks, respectively. It significantly enhances real-time inference throughput for large models at the edge and, for the first time, establishes an efficient, scalable multi-TPU pipelined inference architecture on resource-constrained edge devices.

In edge vision inference, GPU resources—particularly Streaming Multiprocessors (SMs) and Tensor Cores on Jetson devices—exhibit low utilization (15%–30%), while CPU scheduling emerges as the dominant performance bottleneck. Method: We propose a multi-granularity collaborative profiling framework that integrates NVIDIA Nsight, Linux Perf, and custom event-tracing tools to simultaneously capture GPU hardware metrics (SM/TC utilization, memory bandwidth) and CPU system-level events (context switches, thread scheduling latency). Contribution/Results: Our analysis reveals that high aggregate GPU utilization masks underutilization of individual compute units and confirms CPU-side overhead as the primary contributor to end-to-end latency. Based on these insights, we derive hardware-aware optimization principles tailored for edge AI. Experimental validation demonstrates a 2.1× improvement in concurrent inference throughput. This work establishes a reproducible empirical foundation and systematic optimization pathway for heterogeneous resource orchestration at the edge.

Deploying large language models (LLMs) on edge devices—such as smartphones—is hindered by high memory footprint and slow inference latency. To address these bottlenecks, we propose an on-device efficient inference framework featuring a novel DRAM-Flash hybrid memory architecture and a mobile CPU/GPU-coordinated dynamic weight-input reordering strategy. Our method tightly integrates multiple optimization techniques: post-training quantization, mixed-precision floating-point computation, multi-core load balancing, geometric computation optimization, and instruction-set-aware weight layout customization. These synergistic optimizations significantly improve hardware utilization and computational efficiency. Experimental results demonstrate up to 8.6× speedup over state-of-the-art LLM inference frameworks, alongside substantial memory reduction. The framework enables real-time execution of mainstream open-weight models—including LLaMA and Phi—on both Android and iOS platforms.

Although NF4 quantization effectively reduces memory consumption in large language models, it incurs a significant performance bottleneck during inference on NVIDIA GPUs due to the costly dequantization back to FP16. This work proposes a lightweight shared memory optimization that utilizes only 64 bytes of shared memory per thread block, simplifies indexing logic, and efficiently leverages the GPU memory hierarchy to accelerate NF4 dequantization. The approach is fully compatible with the HuggingFace ecosystem, requires no modifications to existing frameworks, and offers plug-and-play deployment. Evaluated on Gemma-27B, Qwen3-32B, and Llama3.3-70B models, the proposed kernel achieves 2.0–2.2× speedup over BitsAndBytes, with end-to-end inference acceleration of up to 1.54×.

Existing frameworks for real-time dynamic DNN inference on edge devices suffer from CPU underutilization, high latency, and memory spikes when unsupported operators fall back to the CPU. This work addresses these issues by proposing a branch-aware memory management scheme and an adaptive heterogeneous scheduling mechanism—enabling fine-grained, subgraph-level CPU-accelerator co-execution without model restructuring. Our approach leverages computation DAG partitioning, dedicated memory pool allocation, buffer reuse, and branch-aware scheduling to build a lightweight heterogeneous subgraph execution engine. Evaluated across three mobile devices and five dynamic DNN models, our method reduces end-to-end inference latency by up to 46%, incurs only 26.5% average memory overhead, and improves energy efficiency by up to 30%. To the best of our knowledge, this is the first framework achieving such efficient, transparent, and fine-grained heterogeneous execution for dynamic DNNs on resource-constrained edge platforms.

This work addresses the high latency, substantial memory footprint, and numerical instability—such as NaN outputs under FP16 precision—that hinder real-time inference of Transformer models. Building upon NVIDIA TensorRT, the authors develop a modular, containerized GPU-accelerated inference pipeline featuring a structure-aware mixed-precision strategy: FP16 is employed in linear layers while Softmax and LayerNorm operations retain FP32 to prevent overflow and preserve output fidelity, achieving cosine similarity ≥0.9998. A reproducible evaluation framework spanning over 360 configurations reveals that random inputs severely underestimate FP16 instability. Experiments demonstrate up to a 64.4× speedup over CPU baselines (single-sample latency <10 ms), a 63% reduction in memory usage, no degradation in downstream task accuracy, and stable FP16 acceleration ratios between 1.84× and 2.00×.

This work addresses the high communication overhead and inefficient state caching that hinder the multi-GPU deployment of selective state space models (SSMs). It presents the first efficient tensor parallelism scheme tailored for selective SSMs, significantly reducing synchronization costs through optimized parameter sharding, improved locality of state caching, and the introduction of quantized AllReduce. The proposed method achieves 1.6–4.0× higher batch throughput for models such as Mamba on 2–4 GPUs, with even more pronounced gains in long-context scenarios. Further throughput improvements of 10–18% are obtained by incorporating quantized AllReduce, demonstrating the effectiveness of the approach in scaling selective SSMs across multiple devices.

This work addresses the performance bottleneck caused by CPU–GPU data staging during distributed Transformer inference on embedded edge devices, which can result in worse performance than single-device execution. The study is the first to identify this bottleneck in integrated GPU architectures and proposes a dynamic execution decision mechanism grounded in real hardware performance profiling. By combining lightweight offline profiling with Segment Means tensor compression and the GLOO communication backend, the system enables runtime-adaptive scheduling on the NVIDIA Jetson Orin Nano. Compared to static full-tensor exchange strategies, the proposed approach reduces end-to-end latency by 65%–77% and cuts energy consumption by 34%–52%.