Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents

2026-06-25

AI Summary

This study addresses the lack of effective evaluation for breadth-first search capabilities—specifically, exhaustive enumeration of members within a closed set and their structured attributes—in non-English contexts. It presents the first Korean-language benchmark for breadth-oriented agent assessment, leveraging an automated synthesis and validation pipeline to generate tasks that require agents to fully enumerate members of a given parent entity and populate their structured attribute tables. The work introduces a novel structured difficulty modulation mechanism, controlling table width and two-dimensional composite keys, alongside a unified normalized matcher and a multidimensional scoring framework (Item-, Column-, and Row-F1). Experiments reveal that while current agents achieve strong member identification (Item-F1: 92.8), they struggle significantly with complete row completion (Row-F1: 53.7), with performance degrading as task difficulty increases, thereby underscoring the benchmark’s necessity and challenge.

Abstract

Web-agent benchmarks overwhelmingly measure depth -- pinning one obscure answer behind a chain of constraints -- while breadth, exhaustively enumerating a closed set and filling each item's attributes, is barely evaluated, especially outside English. Breadth is also hard to build: certifying that a gold set is complete and every cell correct is far costlier than checking a single answer. I introduce \textsc{Ko-WideSearch}, a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity -- a TV season, a dynasty, a league, an administrative region, an election -- and asks for its full membership plus a per-item attribute table, graded by Item-, Column-, and Row-F1. It spans 228 tables over 190 entities and sixteen categories across three difficulty tiers, set by two structural knobs I dial independently -- table width and a 2-D composite key -- so cross-product membership climbs from 0\% to 100\% across the tiers. A single normalization-aware comparator is shared between gold construction and grading, so stable date and count columns are not over-dropped on formatting alone. Across twenty web agents, the failure is consistent: agents recover the set but not the rows (e.g.\ Item-F1 92.8 against Row-F1 53.7), accuracy falls steadily as the knobs harden, and neither more search nor more spend closes the gap. Broken down by cell, the hard part is finding the right value, not formatting it: open-ended free-text cells fail most, while cells with a standard answer such as a date or a name usually come out right.

Problem

• breadth-search
• web agents
• benchmark
• exhaustive set enumeration
• Korean

Innovation

• breadth-search benchmark
• synthesize-and-verify pipeline
• exhaustive set enumeration
• normalization-aware evaluation
• web agent evaluation