MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factors

2026-06-15

AI Summary

This study addresses the challenge that user queries in map services often entail unstated decision factors influencing satisfaction, which current intelligent agents struggle to proactively identify and objectively evaluate. To bridge this gap, the authors propose a “restore–identify–filter” framework that reconstructs users’ complete intent from behavioral sequences, isolates implicit factors supportable by pre-query information, and—crucially—translates them into quantifiable, objective evaluation targets. This approach shifts the assessment paradigm from mere task completion to satisfaction-aware decision making. Leveraging large-scale, real-world anonymized user data, the work integrates behavioral analysis with an evidence-filtering mechanism to construct a multidimensional annotated benchmark. Experimental results demonstrate that while existing map agents perform adequately on explicit tasks, they remain markedly deficient in fulfilling implicit needs and proactively gathering satisfaction-relevant evidence.

Abstract

Large language model agents are increasingly integrated into map services. Since map services are embedded in everyday-life scenarios rather than professional task settings, users often express their needs informally, resulting in underspecified queries with many unspoken needs, namely, implicit decision factors that are critical for user satisfaction. Although clarification is an effective way to mitigate this issue, it increases user burden in daily interaction, and a capable agent should first proactively recover such factors from available information sources. However, evaluating this ability is challenging. The first challenge is to determine which implicit decision factors are suitable for evaluation. A factor is evaluable only if it affects user acceptance and can be recovered from information available to the agent before it responds. Second, user satisfaction cannot be reliably represented by a single reference answer, requiring a benchmark that converts satisfaction-relevant factors into objective and quantifiable evaluation targets. To address these challenges, we propose a restore-identify-filter framework that reconstructs complete user needs from behavior-chain evidence, identifies implicit decision factors, and retains only those supported by pre-query evidence. Building on this methodology, we construct MapSatisfyBench from large-scale, real-world anonymized user data and annotate ground truth from five dimensions and enables full-chain evaluation of satisfaction-aware map agents. Experiments show that current agents generally perform well on explicit task completion, but remain limited in satisfying implicit decision factors and proactively acquiring the evidence needed for satisfaction-aware decisions. These findings establish MapSatisfyBench as a benchmark for shifting map-agent evaluation from task completion toward satisfaction-aware spatial decision making.

Problem

• implicit decision factors
• user satisfaction
• map agents
• benchmarking
• satisfaction-aware evaluation

Innovation

• satisfaction-aware agents
• implicit decision factors
• behavior-grounded evaluation
• map agent benchmarking
• restore-identify-filter framework