RRTO: A High-Performance Transparent Offloading System for Model Inference in Mobile Edge Computing

2025-07-29

AI Summary

In mobile edge computing, transparent inference offloading suffers from high latency and poor energy efficiency due to frequent remote procedure calls (RPCs), while existing non-transparent approaches require invasive source-code modifications. Method: This paper proposes the first transparent offloading system that integrates a record-and-replay mechanism with a two-stage operator sequence search algorithm. It statically identifies batch-executable operator sequences within neural network models and applies noise filtering and pattern matching to eliminate redundant RPCs—achieving efficient remote scheduling without any source-code changes. Contribution/Results: Experiments demonstrate up to 98% reduction in per-inference latency and energy consumption. The system achieves performance comparable to non-transparent offloading and significantly outperforms prior transparent solutions, thereby delivering high performance, broad compatibility, and deployment simplicity simultaneously.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Interpretability, Explainability, and Transparency
• Search and Optimization: Distributed Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Deploying Machine Learning (ML) applications on resource-constrained mobile devices remains challenging due to limited computational resources and poor platform compatibility. While Mobile Edge Computing (MEC) offers offloading-based inference paradigm using GPU servers, existing approaches are divided into non-transparent and transparent methods, with the latter necessitating modifications to the source code. Non-transparent offloading achieves high performance but requires intrusive code modification, limiting compatibility with diverse applications. Transparent offloading, in contrast, offers wide compatibility but introduces significant transmission delays due to per-operator remote procedure calls (RPCs). To overcome this limitation, we propose RRTO, the first high-performance transparent offloading system tailored for MEC inference. RRTO introduces a record/replay mechanism that leverages the static operator sequence in ML models to eliminate repetitive RPCs. To reliably identify this sequence, RRTO integrates a novel Operator Sequence Search algorithm that detects repeated patterns, filters initialization noise, and accelerates matching via a two-level strategy. Evaluation demonstrates that RRTO achieves substantial reductions of up to 98% in both per-inference latency and energy consumption compared to state-of-the-art transparent methods and yields results comparable to non-transparent approaches, all without necessitating any source code modification.

Problem

• Overcoming high latency in transparent offloading for MEC inference
• Reducing energy consumption in mobile ML model offloading
• Achieving performance parity without source code modification

Innovation

• Record/replay mechanism to eliminate repetitive RPCs
• Operator Sequence Search algorithm for reliable identification
• Two-level strategy to accelerate pattern matching