Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detection

2025-09-24

AI Summary

Traditional object detection methods suffer performance degradation in complex scenarios—such as low-light conditions and severe occlusion—due to insufficient semantic understanding. To address this, we propose an adaptive semantic-enhanced edge-cloud collaborative detection framework leveraging multimodal large language models (MLLMs). Our approach features three key contributions: (1) instruction-tuning an MLLM to generate structured scene descriptions; (2) designing a lightweight adaptive mapping module that dynamically translates semantic descriptions into parameter-adjustment signals for edge-side detectors; and (3) establishing a confidence-driven edge-cloud inference mechanism to optimally balance semantic enhancement and resource overhead. Experimental results demonstrate that our method maintains high detection accuracy while reducing latency by over 79% and computational cost by 70% compared to baseline approaches, significantly improving real-time performance and practicality in challenging environments.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Cloud, edge and content delivery systems for the Web
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Traditional object detection methods face performance degradation challenges in complex scenarios such as low-light conditions and heavy occlusions due to a lack of high-level semantic understanding. To address this, this paper proposes an adaptive guidance-based semantic enhancement edge-cloud collaborative object detection method leveraging Multimodal Large Language Models (MLLM), achieving an effective balance between accuracy and efficiency. Specifically, the method first employs instruction fine-tuning to enable the MLLM to generate structured scene descriptions. It then designs an adaptive mapping mechanism that dynamically converts semantic information into parameter adjustment signals for edge detectors, achieving real-time semantic enhancement. Within an edge-cloud collaborative inference framework, the system automatically selects between invoking cloud-based semantic guidance or directly outputting edge detection results based on confidence scores. Experiments demonstrate that the proposed method effectively enhances detection accuracy and efficiency in complex scenes. Specifically, it can reduce latency by over 79% and computational cost by 70% in low-light and highly occluded scenes while maintaining accuracy.

Problem

• Enhancing object detection in challenging conditions like low-light and occlusions
• Balancing detection accuracy with computational efficiency in edge-cloud systems
• Converting semantic understanding into real-time parameter adjustments for detectors

Innovation

• Adaptive mapping converts semantics to detector parameters
• Edge-cloud framework selects guidance based on confidence scores
• Multimodal LLM generates structured scene descriptions via fine-tuning