MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detection

2024-06-21

arXiv.org

AI Summary

LiDAR-based 3D object detection suffers severe performance degradation under cross-dataset and multi-type corruptions—e.g., sensor degradation and adverse weather—simultaneously. Method: We propose a test-time online adaptation framework that operates batch-wise during inference. It dynamically computes ensemble weights over historical model checkpoints, updates the checkpoint pool incrementally, and performs weighted integration. Contribution/Results: First, we introduce Model-Oblivious Synergy (MOS), a learnable mechanism that selects and fuses checkpoints adaptively to mitigate catastrophic forgetting. Second, we jointly optimize feature independence and prediction similarity to construct low-redundancy, high-generalization ensembles. Evaluated across three datasets and eight corruption types, our method achieves a 67.3% relative mAP improvement in challenging cross-corruption settings—substantially outperforming existing test-time adaptation approaches—and establishes a new practical benchmark for real-world deployment.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

LiDAR-based 3D object detection is crucial for various applications but often experiences performance degradation in real-world deployments due to domain shifts. While most studies focus on cross-dataset shifts, such as changes in environments and object geometries, practical corruptions from sensor variations and weather conditions remain underexplored. In this work, we propose a novel online test-time adaptation framework for 3D detectors that effectively tackles these shifts, including a challenging cross-corruption scenario where cross-dataset shifts and corruptions co-occur. By leveraging long-term knowledge from previous test batches, our approach mitigates catastrophic forgetting and adapts effectively to diverse shifts. Specifically, we propose a Model Synergy (MOS) strategy that dynamically selects historical checkpoints with diverse knowledge and assembles them to best accommodate the current test batch. This assembly is directed by our proposed Synergy Weights (SW), which perform a weighted averaging of the selected checkpoints, minimizing redundancy in the composite model. The SWs are computed by evaluating the similarity of predicted bounding boxes on the test data and the independence of features between checkpoint pairs in the model bank. To maintain an efficient and informative model bank, we discard checkpoints with the lowest average SW scores, replacing them with newly updated models. Our method was rigorously tested against existing test-time adaptation strategies across three datasets and eight types of corruptions, demonstrating superior adaptability to dynamic scenes and conditions. Notably, it achieved a 67.3% improvement in a challenging cross-corruption scenario, offering a more comprehensive benchmark for adaptation. Source code: https://github.com/zhuoxiao-chen/MOS.

Problem

• address domain shifts in 3D detection
• tackle sensor and weather corruptions
• enhance adaptability in dynamic scenes

Innovation

• Dynamic checkpoint selection
• Synergy Weights averaging
• Efficient model bank updates