Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

2026-07-12

AI Summary

This study addresses the challenge of spatial misalignment between remote sensing imagery and external labels—such as those from OpenStreetMap—which hinders effective training of building segmentation models. To overcome this, the authors propose Align and Segment (AnS), an unsupervised framework that jointly performs high-quality building segmentation and automatic image-to-label alignment without requiring precisely registered annotations. The method employs a differentiable spatial transformation module to affinely align labels and incorporates a self-supervised regularization loss to prevent shortcut learning. This work is the first to simultaneously achieve accurate segmentation and alignment under misaligned supervision, thereby relaxing the conventional reliance on pixel-perfect ground truth. Extensive experiments on real and synthetic datasets across multiple cities demonstrate the superiority of AnS in both segmentation accuracy and alignment fidelity.

Abstract

Supervised learning for image segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment. Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network. We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time. Code and derived datasets are available at https://github.com/venkanna37/align-and-segment

Problem

• building segmentation
• misaligned labels
• unsupervised learning
• remote sensing
• image-label alignment

Innovation

• unsupervised learning
• misaligned labels
• spatial alignment
• building segmentation
• spatial transformer