Open-World Semantic Segmentation with Sensitivity Modeling

2026-08-08

AI Summary

This work addresses the joint challenge of segmenting known classes and detecting unknown or anomalous content in open-world semantic segmentation by proposing a unified encoder–triple-decoder architecture. The first decoder performs closed-set segmentation using Gaussian prototypes, while the second isolates unknown regions via contrastive learning. The third decoder introduces a novel sensitivity modeling mechanism that leverages local cross-scale activation instability to capture fine-grained semantic anomalies—a capability absent in prototype- and contrast-based approaches. Evaluated on Cityscapes and BDD-Anomaly, the method achieves significant performance gains, improving AUROC by 2.4% and reducing FPR@95TPR by 2.5 percentage points on BDD-Anomaly, all while preserving closed-set segmentation accuracy.

Abstract

Modern vision systems must operate in "open-world" settings, where models must recognize known categories and detect unseen or anomalous content. Conventional semantic segmentation models operate under a "closed-world" assumption, often producing overconfident misclassifications on novel content. We address open-world semantic segmentation, the joint task of segmenting known classes while detecting and grouping novel or anomalous content without additional supervision, by extending a dual-decoder baseline with a third, complementary decoder within a unified encoder-decoder design. The first decoder performs closed-set segmentation using Gaussian prototypes for known categories. The second uses contrastive feature learning to isolate unknown regions in embedding space. The third, our key contribution, is a sensitivity decoder that captures fine-grained texture irregularities and activation instabilities indicative of semantic uncertainty, which neither semantic prototypes nor contrastive norms can reliably detect. The three decoders provide genuinely complementary signals: class-level OOD distance in logit space, global feature energy in embedding space, and local activation instability across encoder scales. Experiments on Cityscapes and BDD-Anomaly show that our method improves anomaly segmentation and novel-class discovery while maintaining competitive closed-set accuracy, with gains of +2.4% AUROC and a 2.5 pp. reduction in FPR@95TPR on BDD-Anomaly over the baseline.

Problem

• open-world
• semantic segmentation
• anomaly detection
• out-of-distribution
• novel class discovery

Innovation

• open-world semantic segmentation
• sensitivity decoder
• anomaly detection
• complementary decoders
• activation instability