Physically Feasible Semantic Segmentation

2024-08-26

arXiv.org

AI Summary

Semantic segmentation models often produce geometrically implausible predictions—e.g., placing “road” above “sky”—due to a lack of physical commonsense reasoning. To address this, we propose a differentiable morphological loss that explicitly encodes and end-to-end optimizes spatial class-relational constraints, without requiring additional annotations or 3D priors. Our method learns inter-class spatial relationships in a data-driven manner and integrates multi-scale differentiable morphological operations, enabling plug-and-play incorporation into mainstream architectures (e.g., Mask2Former, SegFormer). Evaluated on ADE20K, Cityscapes, and ACDC, our approach consistently improves mean Intersection-over-Union (mIoU) across all benchmarks. It further enhances robustness under domain shift and effectively eliminates physically infeasible segmentation errors—such as topologically invalid object arrangements—thereby improving both geometric consistency and semantic plausibility of predictions.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion, minimizing solely per-pixel or per-segment classification objectives on their training data. This purely data-driven paradigm often leads to absurd segmentations, especially when the domain of input images is shifted from the one encountered during training. For instance, state-of-the-art models may assign the label ``road to a segment that is located above a segment that is respectively labeled as ``sky, although our knowledge of the physical world dictates that such a configuration is not feasible for images captured by forward-facing upright cameras. Our method, Physically Feasible Semantic Segmentation (PhyFea), first extracts explicit constraints that govern spatial class relations from the semantic segmentation training set at hand in an offline, data-driven fashion, and then enforces a morphological yet differentiable loss that penalizes violations of these constraints during training to promote prediction feasibility. PhyFea is a plug-and-play method and yields consistent and significant performance improvements over diverse state-of-the-art networks on which we implement it across the ADE20K, Cityscapes, and ACDC datasets. Code and models will be made publicly available.

Problem

• Image Segmentation
• Unseen Images
• Logic Discrepancy

Innovation

• PhyFea
• Spatial Rule Compliance
• Image Segmentation Enhancement