Removing Planner Bias in Goal Recognition Through Multi-Plan Dataset Generation

2026-02-16

AI Summary

This work addresses the systematic bias in existing goal recognition datasets, which stems from their reliance on a single heuristic planner and limits the evaluation of recognizer robustness across diverse planning strategies. To overcome this limitation, the authors propose a top-k planning–based approach that generates multiple feasible plans for the same goal, thereby constructing a more representative benchmark dataset. They introduce a novel metric, Version Coverage Score (VCS), to quantify a goal recognizer’s adaptability to varying planning schemes. Experimental results demonstrate that state-of-the-art recognizers exhibit significant performance degradation under low observability when evaluated on data generated by multiple planners, highlighting the proposed method’s effectiveness in exposing model vulnerabilities and advancing research on robust goal recognition.

Technology Category

• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Motion and Path Planning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Autonomous agents require some form of goal and plan recognition to interact in multiagent settings. Unfortunately, all existing goal recognition datasets suffer from a systematical bias induced by the planning systems that generated them, namely heuristic-based forward search. This means that existing datasets lack enough challenge for more realistic scenarios (e.g., agents using different planners), which impacts the evaluation of goal recognisers with respect to using different planners for the same goal. In this paper, we propose a new method that uses top-k planning to generate multiple, different, plans for the same goal hypothesis, yielding benchmarks that mitigate the bias found in the current dataset. This allows us to introduce a new metric called Version Coverage Score (VCS) to measure the resilience of the goal recogniser when inferring a goal based on different sets of plans. Our results show that the resilience of the current state-of-the-art goal recogniser degrades substantially under low observability settings.

Problem

• goal recognition
• planner bias
• multi-agent systems
• dataset generation
• plan diversity

Innovation

• goal recognition
• multi-plan dataset generation
• top-k planning
• planner bias
• Version Coverage Score