Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

2026-05-07

AI Summary

Current evaluations of large language models rely on static benchmarks, which are prone to ceiling and floor effects and thus struggle to precisely capture nuanced differences in model capabilities. This work proposes Dynamic Boundary Evaluation (DBE), a framework that enables fine-grained and comparable assessment across dimensions such as safety, competence, and truthfulness by identifying the boundary region where a model’s pass probability for prompts approaches 0.5. DBE integrates a difficulty-calibrated item bank, a skill-guided boundary search (SGBS) algorithm requiring only API access, and a unified evaluation protocol grounded in adaptive testing theory, supporting multi-turn interactive black-box evaluation. Experiments demonstrate that DBE achieves non-saturated, broad-coverage model assessment across four task categories, is compatible with existing datasets, and significantly enhances discriminative power for subtle inter-model capability differences.

Abstract

Evaluating large language models (LLMs) today rests on fixed benchmarks that apply the same set of items to any model, producing ceiling and floor effects that mask capability gaps. We argue that the most informative evaluation signal lies at the boundary, where the per-prompt pass probability is near $0.5$ under random-sampling decoding, and propose Dynamic Boundary Evaluation (DBE), which actively locates each model's boundary and places it on a globally comparable difficulty scale. DBE delivers three artifacts: (i) a calibrated item bank covering safety, capability, and truthfulness, with per-item difficulty labels validated across $9$ reference LLMs; (ii) Skill-Guided Boundary Search (SGBS), a search algorithm that finds boundary items for a given target LLM using only API-level query access; and (iii) an evaluation protocol that places a new LLM on a unified ability scale and grows the evaluation set adaptively when the target falls outside the bank's coverage. We instantiate DBE on four categories spanning safety (harmful request refusal and over-refusal), capability (constrained instruction following), and truthfulness (multi-turn sycophancy resistance). The resulting evaluation covers a broader model spectrum without saturation while remaining compatible with existing datasets.

Problem

• language model evaluation
• fixed benchmarks
• ceiling and floor effects
• capability gaps
• evaluation boundaries

Innovation

• Dynamic Boundary Evaluation
• Skill-Guided Boundary Search
• Calibrated Item Bank
• Adaptive Evaluation
• Language Model Benchmarking