Selected work

Representative Papers

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Jan 17, 2026

This work addresses the challenge that existing AI agent benchmarks inadequately evaluate performance on real-world, complex, and long-horizon command-line tasks. To bridge this gap, the authors introduce a novel evaluation benchmark comprising 89 high-difficulty terminal tasks, all derived from authentic workflows and accompanied by isolated execution environments, human-authored reference solutions, and automated verification tests. The benchmark is designed to ensure realism, verifiability, and diversity, substantially narrowing the disparity between practical scenarios and current model evaluation paradigms. Experimental results demonstrate that even state-of-the-art agents achieve success rates below 65% on this benchmark. The paper further provides comprehensive error analysis and publicly releases the dataset and evaluation toolchain to support future research in this domain.

9 citations1 influentialRead paper

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

Jun 03, 2026

Current large language models lack high-quality benchmarks for evaluating continual learning capabilities in realistic, stateful environments. To address this gap, this work proposes CL-Bench—a expert-validated benchmark spanning six professional domains, where tasks share latent structures to support online learning assessment. We introduce, for the first time, an evaluation framework that disentangles a model’s prior capabilities from its online learning performance, complemented by a state-aware protocol and gain metrics to systematically evaluate diverse agent architectures. Experiments reveal that state-of-the-art systems commonly suffer from overfitting to immediate observations and insufficient cross-task knowledge reuse, and that specialized memory mechanisms do not significantly outperform simple in-context learning.

0 citationsRead paper

JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

May 24, 2026

In highly specialized domains lacking ground-truth answers, it remains unclear how to effectively evaluate the quality of generated content and whether scoring rubrics or pairwise preferences constitute more suitable supervision signals. This work introduces JudgmentBench, a benchmark comprising 30 real-world legal tasks, where the same cohort of experienced lawyers provides both rubric-based scores and pairwise preference annotations for three quality tiers of outputs generated by large language models. Empirical analysis demonstrates that pairwise preference judgments substantially outperform rubric-based scoring in both validity—evidenced by a Spearman correlation coefficient of 0.908 versus 0.150—and efficiency, requiring less than half the annotation time. These findings hold consistently across both human and automated evaluators. The study delivers the first dual-modality expert-annotated dataset and methodological foundation for evaluation in high-expertise domains.

0 citationsRead paper

Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes

Apr 20, 2026

This work investigates how to efficiently fine-tune small language models under constraints of limited labeled data and computational resources by leveraging reinforcement learning with verifiable rewards (RLVR). To enable precise control over data scale, diversity, and complexity, the authors construct three procedurally generated datasets encompassing numerical counting, graph reasoning, and spatial reasoning tasks. Experimental results demonstrate that, in low-data regimes, training on a mixture of tasks with varying complexity yields up to a fivefold improvement in sample efficiency compared to training solely on simple tasks, while also significantly enhancing the model’s ability to generalize from simpler to more complex tasks. This study establishes a controllable and reproducible paradigm for efficient model fine-tuning in resource-constrained settings.

0 citationsRead paper

RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

Apr 01, 2026

Current evaluation rubrics for large language models lack systematic approaches to diagnosing quality issues. This work proposes RIFT—the first taxonomy of rubric failure modes grounded in grounded theory—encompassing eight distinct failure types across three dimensions: reliability, content validity, and consequential validity. Accompanying this framework is a suite of automated diagnostic metrics designed to detect such failures. Empirical evaluation demonstrates strong inter-annotator agreement, with 87% pairwise consistency (mean Cohen’s κ = 0.64), and high alignment between automated metrics and human judgments, achieving a peak F1 score of 0.86. This study represents the first systematic modeling and scalable diagnosis of quality problems in LLM evaluation rubrics.

0 citationsRead paper

Recent publications

Latest Papers

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

Jun 03, 2026

Current large language models lack high-quality benchmarks for evaluating continual learning capabilities in realistic, stateful environments. To address this gap, this work proposes CL-Bench—a expert-validated benchmark spanning six professional domains, where tasks share latent structures to support online learning assessment. We introduce, for the first time, an evaluation framework that disentangles a model’s prior capabilities from its online learning performance, complemented by a state-aware protocol and gain metrics to systematically evaluate diverse agent architectures. Experiments reveal that state-of-the-art systems commonly suffer from overfitting to immediate observations and insufficient cross-task knowledge reuse, and that specialized memory mechanisms do not significantly outperform simple in-context learning.

0 citationsRead paper

JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

May 24, 2026

In highly specialized domains lacking ground-truth answers, it remains unclear how to effectively evaluate the quality of generated content and whether scoring rubrics or pairwise preferences constitute more suitable supervision signals. This work introduces JudgmentBench, a benchmark comprising 30 real-world legal tasks, where the same cohort of experienced lawyers provides both rubric-based scores and pairwise preference annotations for three quality tiers of outputs generated by large language models. Empirical analysis demonstrates that pairwise preference judgments substantially outperform rubric-based scoring in both validity—evidenced by a Spearman correlation coefficient of 0.908 versus 0.150—and efficiency, requiring less than half the annotation time. These findings hold consistently across both human and automated evaluators. The study delivers the first dual-modality expert-annotated dataset and methodological foundation for evaluation in high-expertise domains.

0 citationsRead paper

Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes

Apr 20, 2026

This work investigates how to efficiently fine-tune small language models under constraints of limited labeled data and computational resources by leveraging reinforcement learning with verifiable rewards (RLVR). To enable precise control over data scale, diversity, and complexity, the authors construct three procedurally generated datasets encompassing numerical counting, graph reasoning, and spatial reasoning tasks. Experimental results demonstrate that, in low-data regimes, training on a mixture of tasks with varying complexity yields up to a fivefold improvement in sample efficiency compared to training solely on simple tasks, while also significantly enhancing the model’s ability to generalize from simpler to more complex tasks. This study establishes a controllable and reproducible paradigm for efficient model fine-tuning in resource-constrained settings.

0 citationsRead paper

RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

Apr 01, 2026

Current evaluation rubrics for large language models lack systematic approaches to diagnosing quality issues. This work proposes RIFT—the first taxonomy of rubric failure modes grounded in grounded theory—encompassing eight distinct failure types across three dimensions: reliability, content validity, and consequential validity. Accompanying this framework is a suite of automated diagnostic metrics designed to detect such failures. Empirical evaluation demonstrates strong inter-annotator agreement, with 87% pairwise consistency (mean Cohen’s κ = 0.64), and high alignment between automated metrics and human judgments, achieving a peak F1 score of 0.86. This study represents the first systematic modeling and scalable diagnosis of quality problems in LLM evaluation rubrics.

0 citationsRead paper

Benchmarking Agents in Insurance Underwriting Environments

Jan 31, 2026

This work proposes UNDERWRITE, the first multi-turn insurance underwriting evaluation benchmark co-developed with domain experts to address the limitations of existing AI agent benchmarks that focus predominantly on open-domain tasks and rely on single accuracy metrics. UNDERWRITE incorporates real-world complexities such as proprietary business knowledge, noisy tool interfaces, and imperfect user simulations. Leveraging multi-turn dialogue modeling, hallucination detection, and pass^k evaluation, comprehensive assessments of 13 state-of-the-art models reveal significant performance gaps and fragilities of general-purpose agents in specialized enterprise settings. Notably, the highest-accuracy model is not necessarily the most efficient, and robust tool usage alone fails to fully suppress hallucinations, highlighting critical shortcomings for real-world deployment in high-stakes domains like insurance underwriting.

0 citationsRead paper