Open-set Anomaly Segmentation in Complex Scenarios

2025-04-28

AI Summary

To address the insufficient reliability of open-set anomaly segmentation under adverse weather conditions (e.g., rain, fog, low illumination) in safety-critical applications such as autonomous driving, this paper proposes the Energy-Entropy Joint Learning (EEL) framework. Methodologically, we introduce a novel anomaly discrimination mechanism that jointly leverages energy-based scores and information entropy; design a diffusion-based anomalous image synthesizer to enhance training data fidelity and diversity; and construct a plug-and-play semantic segmentation ensemble architecture for seamless integration. Evaluated on ComsAmy and multiple public benchmarks, EEL achieves an average AUPRC improvement of 4.96% and reduces FPR95 by 9.87% over state-of-the-art open-set segmentation methods. The framework establishes a scalable, deployable paradigm for robust perception in open-world environments.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Precise segmentation of out-of-distribution (OoD) objects, herein referred to as anomalies, is crucial for the reliable deployment of semantic segmentation models in open-set, safety-critical applications, such as autonomous driving. Current anomalous segmentation benchmarks predominantly focus on favorable weather conditions, resulting in untrustworthy evaluations that overlook the risks posed by diverse meteorological conditions in open-set environments, such as low illumination, dense fog, and heavy rain. To bridge this gap, this paper introduces the ComsAmy, a challenging benchmark specifically designed for open-set anomaly segmentation in complex scenarios. ComsAmy encompasses a wide spectrum of adverse weather conditions, dynamic driving environments, and diverse anomaly types to comprehensively evaluate the model performance in realistic open-world scenarios. Our extensive evaluation of several state-of-the-art anomalous segmentation models reveals that existing methods demonstrate significant deficiencies in such challenging scenarios, highlighting their serious safety risks for real-world deployment. To solve that, we propose a novel energy-entropy learning (EEL) strategy that integrates the complementary information from energy and entropy to bolster the robustness of anomaly segmentation under complex open-world environments. Additionally, a diffusion-based anomalous training data synthesizer is proposed to generate diverse and high-quality anomalous images to enhance the existing copy-paste training data synthesizer. Extensive experimental results on both public and ComsAmy benchmarks demonstrate that our proposed diffusion-based synthesizer with energy and entropy learning (DiffEEL) serves as an effective and generalizable plug-and-play method to enhance existing models, yielding an average improvement of around 4.96% in $ m{AUPRC}$ and 9.87% in $ m{FPR}_{95}$.

Problem

• Segmentation of anomalies in complex weather conditions
• Improving robustness in open-set anomaly detection
• Enhancing anomaly segmentation with energy-entropy learning

Innovation

• Energy-entropy learning for robust anomaly segmentation
• Diffusion-based anomalous training data synthesizer
• Plug-and-play method enhancing existing models