ABO: Abandon Bayer Filter for Adaptive Edge Offloading in Responsive Augmented Reality

2025-04-29

The Web Conference

AI Summary

To address bandwidth and latency bottlenecks in DNN-based edge offloading for augmented reality (AR), this paper identifies that conventional Bayer demosaicing preprocessing neither aids offloaded inference nor reduces transmission overhead—instead, it increases it. We propose a novel paradigm: directly offloading RAW frames to the edge and jointly parallelizing demosaicing with DNN inference. Our key contributions are: (1) the first multi-granularity, configurable block-wise RAW neural codec; (2) a dynamic encoding control mechanism jointly driven by content-aware analysis and real-time bandwidth estimation; and (3) a computation-communication co-designed low-latency pipelined scheduler. Experiments demonstrate that, compared to state-of-the-art methods, our approach achieves a 40% increase in frame throughput, a 30% reduction in end-to-end latency, up to 15% higher DNN inference accuracy, and significantly improved robustness under low-light conditions and motion blur.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Bayer-patterned color filter array (CFA) has been the go-to solution for color image sensors. In augmented reality (AR), although color interpolation (i.e., demosaicing) of pre-demosaic RAW images facilitates a user-friendly rendering, it creates no benefits in offloaded DNN analytics but increases the image channels by 3 times inducing higher transmission overheads. The potential optimization in frame preprocessing of DNN offloading is yet to be investigated. To that end, we propose ABO, an adaptive RAW frame offloading framework that parallelizes demosaicing with DNN computation. Its contributions are three-fold: First, we design a configurable tile-wise RAW image neural codec to compress frame sizes while sustaining downstream DNN accuracy under bandwidth constraints. Second, based on content-aware tiles-in-frame selection and runtime bandwidth estimation, a dynamic transmission controller adaptively calibrates codec configurations to maximize the DNN accuracy. Third, we further optimize the system pipelining to achieve lower end-to-end frame processing latency and higher throughput. Through extensive evaluations on a prototype platform, ABO consistently achieves 40% more frame processing throughput and 30% less end-to-end latency while improving the DNN accuracy by up to 15% than SOTA baselines. It also exhibits improved robustness against dim lighting and motion blur situations.

Problem

• Optimize DNN offloading by avoiding Bayer filter demosaicing overhead
• Develop adaptive RAW frame compression for bandwidth-efficient transmission
• Reduce latency and improve throughput in AR frame processing

Innovation

• Configurable tile-wise RAW image neural codec
• Dynamic transmission controller for DNN accuracy
• Optimized system pipelining for lower latency