A Unified Spatial Alignment Framework for Highly Transferable Transformation-Based Attacks on Spatially Structured Tasks

2026-03-26

AI Summary

This work addresses a critical yet previously overlooked issue in transformation-based adversarial attacks on spatially structured vision tasks—such as semantic segmentation and object detection—where geometric transformations applied to the input image are not consistently aligned with corresponding structural labels, leading to spatial misalignment that degrades attack efficacy. To resolve this, the authors propose the first unified Spatial Alignment Framework (SAF), which synchronously transforms both inputs and labels via a dedicated spatial alignment algorithm during attack generation, thereby preserving gradient fidelity. Extensive experiments demonstrate that SAF substantially enhances attack transferability, reducing model performance to mIoU 11.34 on Cityscapes, mIoU 31.80 on Kvasir-SEG, and mAP 5.25 on COCO, confirming its effectiveness and broad applicability across diverse structured prediction tasks.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Adversarial Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Transformation-based adversarial attacks (TAAs) demonstrate strong transferability when deceiving classification models. However, existing TAAs often perform unsatisfactorily or even fail when applied to structured tasks such as semantic segmentation and object detection. Encouragingly, recent studies that categorize transformations into non-spatial and spatial transformations inspire us to address this challenge. We find that for non-structured tasks, labels are spatially non-structured, and thus TAAs are not required to adjust labels when applying spatial transformations. In contrast, for structured tasks, labels are spatially structured, and failing to transform labels synchronously with inputs can cause spatial misalignment and yield erroneous gradients. To address these issues, we propose a novel unified Spatial Alignment Framework (SAF) for highly transferable TAAs on spatially structured tasks, where the TAAs spatially transform labels synchronously with the input using the proposed Spatial Alignment (SA) algorithm. Extensive experiments demonstrate the crucial role of our SAF for TAAs on structured tasks. Specifically, in non-targeted attacks, our SAF degrades the average mIoU on Cityscapes from 24.50 to 11.34, and on Kvasir-SEG from 49.91 to 31.80, while reducing the average mAP of COCO from 17.89 to 5.25.

Problem

• spatially structured tasks
• transformation-based adversarial attacks
• spatial misalignment
• semantic segmentation
• object detection

Innovation

• Spatial Alignment Framework
• Transformation-based Adversarial Attacks
• Structured Tasks
• Label Synchronization
• Transferability