Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization

2026-06-15

AI Summary

This work addresses the overconfidence of embodied agents in generating answers without sufficient visual grounding by proposing the Semantic Flip framework. Semantic Flip automatically synthesizes out-of-distribution (OOD) samples through semantic inversion of queries and video memory to train a lightweight rejection module. Notably, this approach requires neither external OOD annotations nor fine-tuning of the pretrained vision-language model, enabling, for the first time, a plug-and-play reliable refusal mechanism in embodied reasoning. To evaluate rejection performance in spatial grounding tasks, the authors introduce a new benchmark, SpaceReject. Experiments demonstrate that the proposed method substantially outperforms strong prompting baselines on both existing benchmarks and SpaceReject, achieving an F₁ score of 0.9559. The code and dataset are publicly released.

Abstract

Detecting unanswerable user queries remains essential for the reliable deployment of real-world embodied agents. However, modern vision-language models (VLMs) often generate overly confident answers even when the available visual memory cannot support the query. Such overconfidence poses various task-dependent risks. The agent may provide misleading information to the user in Embodied Question Answering and select an arbitrary coordinate and physically guide the user there in spatial reasoning for navigation. Despite these high stakes, only a few prior studies directly address when and how an embodied VLM should respond with "I do not know." This work proposes Semantic Flip, a simple yet effective framework that synthesizes auxiliary out-of-distribution (OOD) samples for embodied refusal without requiring external OOD annotations. The key idea is to independently transform the query and video memory to construct auxiliary OOD pairs that lack sufficient visual grounding. These synthesized pairs enable training a lightweight rejection module on top of a frozen pretrained VLM. The module attaches to any existing VLM-based pipeline without retraining the underlying model. Across two complementary benchmarks, Semantic Flip consistently outperforms strong prompting baselines. This work also introduces SpaceReject, a new refusal benchmark for spatial localization with deliberately unanswerable queries over long video memory, where Semantic Flip achieves an $F_1$ score of 0.9559. The source codes and datasets are publicly available at https://github.com/ndb796/SemanticFlip.

Problem

• out-of-distribution detection
• embodied question answering
• refusal mechanism
• spatial localization
• unanswerable queries

Innovation

• Semantic Flip
• out-of-distribution synthesis
• embodied refusal
• vision-language models
• spatial localization