Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals

2025-10-10

AI Summary

Existing segmentation models often rely on spurious correlations—such as texture or contextual cues—rather than true object boundaries, leading to poor generalization in real-world scenarios. To address this, we propose an interactive learning framework grounded in human intervention feedback: user corrections are treated as causal interventions, enabling explicit identification and systematic mitigation of model bias. Our approach integrates visual similarity-based matching with cross-image feature propagation to iteratively refine erroneous predictions and transfer corrective knowledge. This enhances the model’s focus on semantic boundaries, improves interpretability, and strengthens domain generalization. On the Cubical Graph dataset, our method achieves a +9.0 mIoU gain (12–15% relative improvement) while reducing annotation cost by 3–4×. Crucially, it maintains state-of-the-art performance on standard benchmarks, demonstrating both robustness and practical efficiency.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Graphical Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Segmentation models achieve high accuracy on benchmarks but often fail in real-world domains by relying on spurious correlations instead of true object boundaries. We propose a human-in-the-loop interactive framework that enables interventional learning through targeted human corrections of segmentation outputs. Our approach treats human corrections as interventional signals that show when reliance on superficial features (e.g., color or texture) is inappropriate. The system learns from these interventions by propagating correction-informed edits across visually similar images, effectively steering the model toward robust, semantically meaningful features rather than dataset-specific artifacts. Unlike traditional annotation approaches that simply provide more training data, our method explicitly identifies when and why the model fails and then systematically corrects these failure modes across the entire dataset. Through iterative human feedback, the system develops increasingly robust representations that generalize better to novel domains and resist artifactual correlations. We demonstrate that our framework improves segmentation accuracy by up to 9 mIoU points (12-15% relative improvement) on challenging cubemap data and yields 3-4$ imes$ reductions in annotation effort compared to standard retraining, while maintaining competitive performance on benchmark datasets. This work provides a practical framework for researchers and practitioners seeking to build segmentation systems that are accurate, robust to dataset biases, data-efficient, and adaptable to real-world domains such as urban climate monitoring and autonomous driving.

Problem

• Addresses segmentation failures from spurious feature correlations
• Enables human corrections to guide robust feature learning
• Reduces annotation effort while improving cross-domain generalization

Innovation

• Human corrections guide segmentation model learning
• Propagates edits across visually similar images
• Iterative feedback builds robust domain-general representations