SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks

2026-08-10

AI Summary

This work addresses the insufficient robustness of deep neural networks in safety-critical scenarios caused by rare semantic shifts and the difficulty of verifying high-level semantic requirements. To tackle these challenges, the authors propose SeFaR, a framework that integrates diffusion models with vision-language models to generate realistic, semantically consistent, and diverse perturbations guided by natural language specifications and valid inputs. SeFaR employs a hierarchical concept model to systematically explore the feature space, incorporates domain knowledge through user-defined concepts, and leverages a feedback mechanism to identify requirement-irrelevant features that influence model decisions. This enables the generation of interpretable failure-inducing semantic concepts along with corresponding test samples. Experimental results demonstrate that SeFaR effectively uncovers model failures and accurately attributes them to specific semantic factors.

Abstract

Deep neural networks are increasingly deployed in safety-critical domains as perception modules, where failures are often caused due to rare and under-represented scenarios. This necessitates the need to evaluate the semantic robustness of perception models; conformance of behavior to high-level requirements over real-world perceptual variability. To address this, we propose SeFaR, a framework for systematic semantic-feature-centric testing of vision models. Given a natural-language requirement and a set of satisfying inputs, SeFaR evaluates robustness with respect to diverse realistic semantic variations that preserve requirement satisfaction. The approach employs a novel hierarchical concept model enabling structured exploration of the feature space and incorporation of domain knowledge via user-defined concepts. State-of-the-art diffusion and vision-language models are leveraged to generate photorealistic semantics-preserving perturbations and identification of previously unknown features impacting behavior. A feedback-driven adaptive process is adopted to generate interpretable failure-inducing semantic concepts along with corresponding test inputs. Evaluation on case studies demonstrates that the proposed framework effectively satisfies requirement preconditions while identifying requirement-independent features that influence model decisions, enabling it to both uncover faults and relate them to such features.

Problem

• semantic robustness
• deep neural networks
• perception models
• safety-critical domains
• real-world perceptual variability

Innovation

• semantic robustness
• hierarchical concept model
• diffusion models
• vision-language models
• adaptive testing