Generative Semantic Segmentation via an Observable Semantic-Image Interface and Hierarchical Generator Evidence Alignment

2026-08-11

AI Summary

This work addresses color drift, boundary ambiguity, and the limitation of latent-feature decoders treating images merely as intermediate visualizations in generative semantic segmentation. To overcome these issues, the authors propose the Semantic Prism framework, which employs diffusion distillation to construct a single-step generator, integrates a fixed class-color codebook to establish an explicit probabilistic interface, and introduces a hierarchical evidence alignment mechanism in logit space to predict residuals while preserving the image-defined interface as the reference for the final distribution. The study also innovatively presents C-IHD, an error-ranking method that requires no additional predictor. Experiments demonstrate that the approach achieves 72.07% mIoU (+11.39) with an ECE of 0.41% on Cityscapes, and attains 62.22% and 46.89% mIoU on BDD100K and ACDC, respectively, while significantly improving pixel-wise error ranking AUPR—e.g., from 0.6580 to 0.7557 on ACDC.

Abstract

Generative semantic segmentation exposes structured predictions as images, but direct color decoding is susceptible to color drift and boundary mixing, whereas latent-feature decoders that predict a separate output distribution may relegate the rendered image to an intermediate visualization. We present Semantic Prism, a conditional semantic-image generation-and-refinement framework with deterministic inference. A diffusion-distilled one-step generator renders a semantic RGB image; per-pixel distances from the rendered colors to a fixed class-color codebook define an explicit probabilistic interface. Hierarchical Generator Evidence Alignment spatially aligns multi-level generator features and uses a zero-initialized output projection to predict an additive residual in the interface logit space, retaining the image-defined interface as the reference for the final distribution. The interface and refined distributions further enable Contextual Interface--Hierarchy Disagreement (C-IHD), a fixed readout for ranking remaining pixel errors without an auxiliary predictor or additional forward pass. On the 500-image Cityscapes validation set, Semantic Prism achieves 72.07% mean intersection over union, 11.39 mIoU points above direct-interface decoding, with 0.41% expected calibration error. Matched-capacity ablations over three seeds support the benefit of jointly aligned multi-level evidence. A separately trained model attains 62.22% mIoU on BDD100K, while the Cityscapes-trained model reaches 46.89\% mIoU under source-frozen transfer to the Adverse Conditions Dataset with Correspondences, without target-domain adaptation. Across all three datasets, C-IHD consistently improves the area under the precision--recall curve for pixel-error ranking over maximum softmax probability on the same segmentation predictions; on ACDC, it raises AUPR from 0.6580 to 0.7557.

Problem

• generative semantic segmentation
• color drift
• boundary mixing
• semantic-image interface
• pixel error ranking

Innovation

• Semantic Prism
• Observable Semantic-Image Interface
• Hierarchical Generator Evidence Alignment
• Contextual Interface--Hierarchy Disagreement
• Generative Semantic Segmentation