RMT-PPAD: Real-time Multi-task Learning for Panoptic Perception in Autonomous Driving

2025-08-02

AI Summary

To address the demand for holistic perception in autonomous driving, this paper proposes a lightweight, real-time multi-task Transformer model that jointly performs object detection, drivable area segmentation, and lane line segmentation. To tackle feature coupling across tasks and train-inference inconsistency, we introduce a novel gated adaptation module that enables adaptive fusion of shared and task-specific features, and design an adaptive multi-scale segmentation decoder for dynamic feature weighting. Additionally, a structured lane line loss is incorporated to mitigate training-inference discrepancy. Evaluated on BDD100K, the model achieves 84.9% mAP₅₀ for detection, 92.6% mIoU for drivable area segmentation, and 56.8% IoU for lane line segmentation, while running at 32.6 FPS on standard hardware. Extensive experiments demonstrate strong robustness in real-world driving scenarios.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Segmentation
• Application Domains: Mobility, Driving & Flight

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Autonomous driving systems rely on panoptic driving perception that requires both precision and real-time performance. In this work, we propose RMT-PPAD, a real-time, transformer-based multi-task model that jointly performs object detection, drivable area segmentation, and lane line segmentation. We introduce a lightweight module, a gate control with an adapter to adaptively fuse shared and task-specific features, effectively alleviating negative transfer between tasks. Additionally, we design an adaptive segmentation decoder to learn the weights over multi-scale features automatically during the training stage. This avoids the manual design of task-specific structures for different segmentation tasks. We also identify and resolve the inconsistency between training and testing labels in lane line segmentation. This allows fairer evaluation. Experiments on the BDD100K dataset demonstrate that RMT-PPAD achieves state-of-the-art results with mAP50 of 84.9% and Recall of 95.4% for object detection, mIoU of 92.6% for drivable area segmentation, and IoU of 56.8% and accuracy of 84.7% for lane line segmentation. The inference speed reaches 32.6 FPS. Moreover, we introduce real-world scenarios to evaluate RMT-PPAD performance in practice. The results show that RMT-PPAD consistently delivers stable performance. The source codes and pre-trained models are released at https://github.com/JiayuanWang-JW/RMT-PPAD.

Problem

• Real-time panoptic perception for autonomous driving
• Multi-task learning for object detection and segmentation
• Addressing negative transfer between tasks adaptively

Innovation

• Transformer-based multi-task model for panoptic perception
• Lightweight gate control with adaptive feature fusion
• Adaptive segmentation decoder for multi-scale features