Multimodal Remote Inference

2025-08-10

AI Summary

To address the problem of delayed feature transmission and consequent inference error inflation in bandwidth-constrained multimodal remote inference, this paper proposes a task-performance-oriented Age-of-Information (AoI) optimization framework. Unlike conventional latency- or throughput-driven scheduling, our method minimizes a non-monotonic, non-additive inference error function by designing an exponential-threshold-based dual-modal dynamic scheduling policy, supporting heterogeneous transmission delays and real-time modality switching. Theoretical analysis establishes its optimality under general error models and reveals a novel task-aware AoI mechanism—“error-sensitive freshness.” Experiments across diverse network conditions demonstrate that the proposed approach reduces inference error by up to 55% compared to polling and random baselines, significantly enhancing multimodal inference accuracy in resource-constrained settings.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Machine Learning: Multimodal Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We consider a remote inference system with multiple modalities, where a multimodal machine learning (ML) model performs real-time inference using features collected from remote sensors. As sensor observations may change dynamically over time, fresh features are critical for inference tasks. However, timely delivering features from all modalities is often infeasible due to limited network resources. To this end, we study a two-modality scheduling problem to minimize the ML model's inference error, which is expressed as a penalty function of AoI for both modalities. We develop an index-based threshold policy and prove its optimality. Specifically, the scheduler switches modalities when the current modality's index function exceeds a threshold. We show that the two modalities share the same threshold, and both the index functions and the threshold can be computed efficiently. The optimality of our policy holds for (i) general AoI functions that are emph{non-monotonic} and emph{non-additive} and (ii) emph{heterogeneous} transmission times. Numerical results show that our policy reduces inference error by up to 55% compared to round-robin and uniform random policies, which are oblivious to the AoI-based inference error function. Our results shed light on how to improve remote inference accuracy by optimizing task-oriented AoI functions.

Problem

• Scheduling multimodal sensor data transmission under network constraints
• Minimizing machine learning inference error through Age of Information optimization
• Developing optimal policy for non-monotonic AoI functions and heterogeneous transmissions

Innovation

• Index-based threshold policy for scheduling
• Optimizes non-monotonic non-additive AoI functions
• Handles heterogeneous transmission times efficiently