SocRATES: Towards Automated Scenario-based Testing of Social Navigation Algorithms

2024-12-27

AI Summary

Current social navigation evaluation lacks contextualized testing methodologies that capture qualitative dimensions—such as social competence—thereby hindering algorithmic generalization and real-world deployment. To address this, we propose the first end-to-end automated framework for generating social navigation test scenarios: it leverages large language models (LLMs) to jointly perform social reasoning and code generation, translating high-level metadata—including social norms, agent roles, and spatial constraints—into context-aware, simulation-ready multi-agent interaction scenarios. This breaks the bottleneck of manual scenario construction, enabling a closed-loop transformation from abstract specifications to executable simulations. Experiments demonstrate high scenario fidelity and significantly improved semantic translation accuracy over baseline prompting methods. Validation via expert usability studies and evaluation across three representative navigation algorithms confirms the framework’s effectiveness in enabling systematic, quantitative assessment of social navigation capabilities.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Natural Language Processing: Generation
• Multiagent Systems: Agent Communication

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Current social navigation methods and benchmarks primarily focus on proxemics and task efficiency. While these factors are important, qualitative aspects such as perceptions of a robot's social competence are equally crucial for successful adoption and integration into human environments. We propose a more comprehensive evaluation of social navigation through scenario-based testing, where specific human-robot interaction scenarios can reveal key robot behaviors. However, creating such scenarios is often labor-intensive and complex. In this work, we address this challenge by introducing a pipeline that automates the generation of context-, and location-appropriate social navigation scenarios, ready for simulation. Our pipeline transforms simple scenario metadata into detailed textual scenarios, infers pedestrian and robot trajectories, and simulates pedestrian behaviors, which enables more controlled evaluation. We leverage the social reasoning and code-generation capabilities of Large Language Models (LLMs) to streamline scenario generation and translation. Our experiments show that our pipeline produces realistic scenarios and significantly improves scenario translation over naive LLM prompting. Additionally, we present initial feedback from a usability study with social navigation experts and a case-study demonstrating a scenario-based evaluation of three navigation algorithms.

Problem

• Robot Navigation
• Social Environment
• Testing Methodology

Innovation

• SocRATES
• Automated Scenario-based Testing
• Social Navigation Evaluation