AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
本文提出AcCoRD基准，用于评估用户-代理协作中处理动态用户偏好的能力，通过在线购物和旅行规划两个领域测试五种前沿LLM，揭示了模型在应对偏好变化上的不足。
0 citationsRead paper
Selected work
Representative Papers
Document Optimization for Black-Box Retrieval via Reinforcement Learning
This work addresses the limitation of traditional document expansion methods, which often introduce noise that degrades the performance of modern retrievers. The authors reformulate document expansion as a black-box optimization problem and, for the first time, employ reinforcement learning—specifically the GRPO algorithm—to fine-tune language or vision-language models so that their generated document representations better align with the query distribution of a target retriever, using only ranking feedback. The approach is highly generalizable, supporting single-vector, multi-vector, and lexical retrievers, and can be combined with supervised fine-tuning for further gains. Experiments demonstrate that optimized small embedding models significantly outperform larger baselines on both code and visual document retrieval tasks; when integrated with fine-tuning, Jina-ColBERT-V2 achieves nDCG@5 scores of 63.3 and 61.8, respectively.
0 citationsRead paper
Reflective Context Learning: Studying the Optimization Primitives of Context Space
Existing in-context learning approaches lack systematic solutions to challenges such as credit assignment, overfitting, catastrophic forgetting, local optima, and high variance. This work proposes Reflective Contextual Learning (RCL), a novel framework that, for the first time, systematically introduces classical optimization primitives into contextual space. RCL enables agents to optimize their learning through interaction with the environment, reflection on actions and failures, and iterative refinement of context representations. The framework unifies prior methods while incorporating mechanisms including directional update signals, contextual perturbation, batch processing, failure replay, trajectory grouping, curriculum strategies, and variants of optimizer states. Experimental results demonstrate that RCL significantly outperforms strong baselines on AppWorld, BrowseComp+, and RewardBench2, validating both the efficacy and cross-task transferability of its constituent optimization primitives.
0 citationsRead paper
ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction
This work addresses the absence of an end-to-end evaluation benchmark and semantics-aware assessment framework for structured information extraction from PDFs under enterprise-grade, complex JSON schemas. We introduce ExtractBench, the first open-source benchmark comprising 35 high-value economic-domain PDF documents, human-annotated JSON schemas, and 12,867 evaluable fields. It features a novel fine-grained evaluation framework that treats JSON schemas as executable specifications, enabling field-level differentiated scoring—including exact match, tolerance-based, and semantic equivalence—and explicitly distinguishing omissions from hallucinations. Experiments on leading large language models (e.g., GPT-5/5.2, Gemini-3, Claude 4.5) reveal significant performance degradation in broad-schema scenarios, with effective output rates dropping to 0% on a 369-field financial statement schema, underscoring the current models’ severe unreliability in complex structured extraction tasks.
0 citationsRead paper
BLITZRANK: Principled Zero-shot Ranking Agents with Tournament Graphs
This work proposes the first tournament-graph-based zero-shot reranking framework to address the limitations of existing methods, which either rely on inefficient heuristics or fail to fully exploit preference signals from multi-document comparisons. The approach models k-wise comparisons as a complete tournament graph, aggregates them into a global preference graph, and infers implicit rankings via transitive closure. It further introduces certifiable ranking conditions, an information-gain-driven greedy query scheduling strategy, and an equivalence-class compression mechanism for handling non-transitive preferences, enabling efficient hierarchical reranking. Evaluated across 14 benchmarks and 5 large language models, the method achieves Pareto-optimal performance—matching or exceeding state-of-the-art accuracy while reducing token consumption by 25–40% compared to existing approaches, and cutting token usage by up to 7× relative to pairwise methods with negligible performance loss.
0 citationsRead paper
Recent publications
Latest Papers
AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
本文提出AcCoRD基准，用于评估用户-代理协作中处理动态用户偏好的能力，通过在线购物和旅行规划两个领域测试五种前沿LLM，揭示了模型在应对偏好变化上的不足。
0 citationsRead paper
Document Optimization for Black-Box Retrieval via Reinforcement Learning
This work addresses the limitation of traditional document expansion methods, which often introduce noise that degrades the performance of modern retrievers. The authors reformulate document expansion as a black-box optimization problem and, for the first time, employ reinforcement learning—specifically the GRPO algorithm—to fine-tune language or vision-language models so that their generated document representations better align with the query distribution of a target retriever, using only ranking feedback. The approach is highly generalizable, supporting single-vector, multi-vector, and lexical retrievers, and can be combined with supervised fine-tuning for further gains. Experiments demonstrate that optimized small embedding models significantly outperform larger baselines on both code and visual document retrieval tasks; when integrated with fine-tuning, Jina-ColBERT-V2 achieves nDCG@5 scores of 63.3 and 61.8, respectively.
0 citationsRead paper
Reflective Context Learning: Studying the Optimization Primitives of Context Space
Existing in-context learning approaches lack systematic solutions to challenges such as credit assignment, overfitting, catastrophic forgetting, local optima, and high variance. This work proposes Reflective Contextual Learning (RCL), a novel framework that, for the first time, systematically introduces classical optimization primitives into contextual space. RCL enables agents to optimize their learning through interaction with the environment, reflection on actions and failures, and iterative refinement of context representations. The framework unifies prior methods while incorporating mechanisms including directional update signals, contextual perturbation, batch processing, failure replay, trajectory grouping, curriculum strategies, and variants of optimizer states. Experimental results demonstrate that RCL significantly outperforms strong baselines on AppWorld, BrowseComp+, and RewardBench2, validating both the efficacy and cross-task transferability of its constituent optimization primitives.
0 citationsRead paper
ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction
This work addresses the absence of an end-to-end evaluation benchmark and semantics-aware assessment framework for structured information extraction from PDFs under enterprise-grade, complex JSON schemas. We introduce ExtractBench, the first open-source benchmark comprising 35 high-value economic-domain PDF documents, human-annotated JSON schemas, and 12,867 evaluable fields. It features a novel fine-grained evaluation framework that treats JSON schemas as executable specifications, enabling field-level differentiated scoring—including exact match, tolerance-based, and semantic equivalence—and explicitly distinguishing omissions from hallucinations. Experiments on leading large language models (e.g., GPT-5/5.2, Gemini-3, Claude 4.5) reveal significant performance degradation in broad-schema scenarios, with effective output rates dropping to 0% on a 369-field financial statement schema, underscoring the current models’ severe unreliability in complex structured extraction tasks.
0 citationsRead paper
BLITZRANK: Principled Zero-shot Ranking Agents with Tournament Graphs
This work proposes the first tournament-graph-based zero-shot reranking framework to address the limitations of existing methods, which either rely on inefficient heuristics or fail to fully exploit preference signals from multi-document comparisons. The approach models k-wise comparisons as a complete tournament graph, aggregates them into a global preference graph, and infers implicit rankings via transitive closure. It further introduces certifiable ranking conditions, an information-gain-driven greedy query scheduling strategy, and an equivalence-class compression mechanism for handling non-transitive preferences, enabling efficient hierarchical reranking. Evaluated across 14 benchmarks and 5 large language models, the method achieves Pareto-optimal performance—matching or exceeding state-of-the-art accuracy while reducing token consumption by 25–40% compared to existing approaches, and cutting token usage by up to 7× relative to pairwise methods with negligible performance loss.
0 citationsRead paper