Selected work

Representative Papers

Eliciting Harmful Capabilities by Fine-Tuning On Safeguarded Outputs

Jan 20, 2026

Although state-of-the-art large language models employ output-level safety mechanisms, they may still inadvertently leak harmful knowledge through indirect prompting, enabling open-source models—after fine-tuning—to reconstruct hazardous capabilities, thereby posing ecosystem-level risks. This work presents the first systematic investigation of such cross-model capability transfer threats and introduces a three-stage elicitation attack framework: by crafting benign prompts that are semantically proximate to harmful tasks, adversaries can extract implicit hazardous information from safeguarded models and use it to fine-tune open-source counterparts. Experiments on dangerous chemical synthesis tasks demonstrate that this approach recovers approximately 40% of the performance gap between protected and unrestricted models, with attack efficacy significantly amplified by both the capability of the frontier model and the scale of fine-tuning data, thereby challenging the adequacy of current safety paradigms.

3 citationsRead paper

SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

Feb 16, 2026arXiv.org

This work addresses the challenges large language models (LLMs) face in end-to-end spreadsheet generation—specifically, their difficulty in simultaneously satisfying users’ explicit and implicit constraints and the absence of a unified evaluation benchmark. To this end, we introduce SpreadsheetArena, the first systematic evaluation framework grounded in human preferences. The platform employs blind pairwise preference voting augmented by expert assessments and multidimensional feature analysis to comprehensively evaluate generated workbooks across structural, interactive, and layout dimensions. Our study uncovers significant differences in style, functionality, and professionalism among outputs produced under varying prompting strategies. Furthermore, we release the first large-scale spreadsheet dataset encompassing prompts, model-generated outputs, and human preference annotations, thereby establishing spreadsheets as a compelling domain for investigating complex LLM capabilities.

2 citationsRead paper

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

Jan 31, 2026

This work addresses the limitations of existing evaluation methods for assessing large language models’ ability to use external tools in complex real-world scenarios, which often suffer from oversimplified toolsets, rigid workflows, or subjective scoring. To this end, we present the first large-scale benchmark grounded in real Model Context Protocol (MCP) servers, encompassing 36 MCP services, 220 tools, and 1,000 multi-step natural language tasks that require agents to autonomously discover and orchestrate multiple tools. The evaluation employs a no-tool-name prompting strategy and a fine-grained, fact-based scoring mechanism, supported by a containerized framework and multidimensional diagnostic metrics—including tool discovery, parameterization, and error recovery. Experiments reveal that state-of-the-art models achieve pass rates exceeding 50%, with primary failure modes stemming from insufficient tool utilization and task comprehension errors. The benchmark framework, task schema, and a public subset of 500 tasks are openly released.

2 citationsRead paper

Agentic Rubrics as Contextual Verifiers for SWE Agents

Jan 07, 2026arXiv.org

This work addresses the lack of efficient, scalable, and context-aware validation mechanisms in current software engineering agents (SWE Agents). Traditional execution-based verification struggles to scale, while alternative approaches often neglect repository-level context. To overcome these limitations, we propose Agentic Rubrics—a novel framework in which expert agents interact with code repositories to automatically generate context-sensitive rubrics that evaluate the quality of candidate patches without requiring test execution. Our approach introduces, for the first time, agent-generated, fine-grained, and interpretable scoring criteria into the SWE validation pipeline, achieving both scalability and strong generalization. Evaluated on SWE-Bench Verified, our method attains scores of 54.2% and 40.6% using Qwen3-Coder-30B-A3B and Qwen3-32B, respectively, outperforming the strongest baseline by at least 3.5 percentage points.

1 citationsRead paper

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Aug 12, 2026

This work addresses reward hacking in rubric-based reinforcement learning, where policy models often overfit to fixed scoring criteria, leading to improved training scores but degraded output quality. To mitigate this, the authors propose Rubric Dropout, inspired by neuron dropout in neural networks, which randomly omits a subset of rubric items during each training step. This approach effectively prevents overfitting while preserving the validity of intra-group relative advantage computation. Integrated seamlessly into the Group Relative Policy Optimization (GRPO) framework without additional computational overhead, Rubric Dropout is evaluated using the Qwen3-8B large language model. Experiments on HealthBench-Hard and ResearchQA benchmarks demonstrate that with dropout rates of 30%–50%, the method substantially suppresses reward hacking, yielding gold-rater score improvements of 1–2 points and 6–7 points, respectively, and significantly enhancing out-of-distribution generalization.

0 citationsRead paper

Recent publications

Latest Papers

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Aug 12, 2026

This work addresses reward hacking in rubric-based reinforcement learning, where policy models often overfit to fixed scoring criteria, leading to improved training scores but degraded output quality. To mitigate this, the authors propose Rubric Dropout, inspired by neuron dropout in neural networks, which randomly omits a subset of rubric items during each training step. This approach effectively prevents overfitting while preserving the validity of intra-group relative advantage computation. Integrated seamlessly into the Group Relative Policy Optimization (GRPO) framework without additional computational overhead, Rubric Dropout is evaluated using the Qwen3-8B large language model. Experiments on HealthBench-Hard and ResearchQA benchmarks demonstrate that with dropout rates of 30%–50%, the method substantially suppresses reward hacking, yielding gold-rater score improvements of 1–2 points and 6–7 points, respectively, and significantly enhancing out-of-distribution generalization.

0 citationsRead paper

CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR

Aug 07, 2026

This work addresses the gap between strong performance of large language models on medical benchmarks and their limited capacity for defensible reasoning in real-world clinical settings using heterogeneous, longitudinal electronic health records (EHRs). To bridge this gap, the authors introduce CliniCARE-Bench—the first benchmark designed for clinical auditing—comprising 25 physician-validated scenarios and 750 real-world cases derived from MIMIC-IV. Agents must integrate structured and unstructured EHR data within a constrained tool environment to issue one of four policy-compliant rulings while fully tracing their investigative process. The framework uniquely unifies longitudinal EHR interrogation, evidence provenance, policy adherence, procedural compliance, and a calibrated abstention mechanism that distinguishes “missing data” from “medical ambiguity.” Reference rulings are established via multi-model arbitration and clinical committee calibration. Evaluation across 16 systems shows four-class accuracy of 65.3%–76.1%, yet defect-free accuracy—excluding shortcut-based responses—drops markedly by 4.8–14.8 percentage points, revealing substantial overestimation by conventional metrics.

0 citationsRead paper

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Aug 06, 2026

This work addresses the current lack of a standardized protocol for evaluating large language models’ (LLMs’) ability to optimize external components of intelligent agents—such as prompts, tools, and control flows. It introduces the first auditable, resource-constrained, and evaluation-isolated harness optimization benchmarking framework, which enforces assessment boundaries via trusted execution environments and incorporates standardized scoring, version tracking, and fixed-budget controls to enable systematic multi-model, multi-task, and multi-seed experimentation. Across 111 experimental runs, the study reveals that the optimizer model itself is more discriminative than the initial harness, that native harnesses exhibit no consistent advantage, and that optimization gains are highly dependent on both task and initial configuration. This work establishes harness optimization as a measurable and discriminative capability in AI systems.

0 citationsRead paper

The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

Aug 06, 2026

This work addresses a critical limitation in video-language models: their poor temporal reasoning on high-frequency or transient event counting tasks, which remains undetected by conventional evaluation protocols that rely solely on final answers. To diagnose this issue, the authors propose the first trajectory-grounded, parameterized evaluation framework, leveraging synthetically generated videos—such as bouncing balls and visual flickers—with executable event trajectories to enable timestamp-level, fine-grained assessment. Experiments reveal that current models are only reliable under low-frequency, persistent state transitions (≤12 events at 0.5–1.0 Hz), while their performance drastically degrades in high-frequency, high-count scenarios, achieving as low as 0.2% accuracy and 18.1% event recall. Although increasing frame rates marginally improves final scores, it fails to restore fidelity in event sequence reconstruction, exposing fundamental limitations in existing temporal modeling capabilities.

0 citationsRead paper

Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

Jul 30, 2026

Current agent evaluation practices often reduce failures to system-level outcomes, making it difficult to pinpoint root causes or guide effective remediation. This work proposes an interaction-centric failure taxonomy and introduces, for the first time, a cross-architectural and generalizable framework for failure localization. The framework maps 41 distinct failure modes onto interaction edges between components—such as models, toolchains, and environments—and explicitly delineates responsibility boundaries among them. By integrating component interaction graph attribution, multi-source trajectory analysis, and an independent reasoning agent-based evaluator, the approach enables reproducible validation. Experiments across four state-of-the-art models demonstrate that the strongest evaluator achieves a Cohen’s κ of 0.76 with human annotations, confirming the taxonomy’s generalizability and consensus alignment.

0 citationsRead paper