ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation

2026-05-28

AI Summary

Although large language models (LLMs) can simulate personality in explicit self-reports, they often diverge from human patterns in implicit behavioral decisions, revealing a pronounced knowledge–decision gap that existing benchmarks struggle to evaluate effectively. To address this asymmetry, this work introduces ActTraitBench—the first evaluation framework that establishes one-to-one mappings between psychological constructs and behavioral tasks—and incorporates quantile mapping to align model outputs with human normative distributions. Furthermore, the authors propose CoCA (Chain of Cognitive Alignment), a novel inference-time intervention designed to bridge the gap between declarative knowledge and behavioral expression. Experiments across 14 mainstream LLMs demonstrate the pervasive nature of knowledge–decision asymmetry; CoCA substantially enhances behavioral consistency in state-of-the-art models while exposing inherent limitations in smaller architectures along this dimension.

Abstract

While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions, revealing a substantial Knowledge-Decision Gap ($G_{\text{KD}}$). Existing benchmarks struggle to measure this asymmetry due to limited construct validity, multi-dimensional entanglement, and distributional biases in LLM-based evaluation. To address these issues, we propose ActTraitBench, a human-grounded evaluation framework for measuring personality consistency in LLMs. Grounded in empirical human data, ActTraitBench establishes one-to-one mappings between psychometric facets and behavioral paradigms, and applies a Distributional Calibration via Quantile Mapping procedure to align LLM-judge score distributions with human norms. Experiments on 14 mainstream LLMs reveal a pervasive knowledge-decision asymmetry, where larger and more capable models often exhibit stronger behavioral divergence despite highly consistent self-reports. To mitigate this gap, we further introduce the Chain of Cognitive Alignment (CoCA), a plug-and-play inference-time intervention that improves alignment in reasoning-capable frontier models while exposing clear capability limitations in smaller architectures.

Problem

• Knowledge-Decision Gap
• Large Language Models
• Personality Consistency
• Behavioral Validation
• Construct Validity

Innovation

• Knowledge-Decision Gap
• Human-grounded evaluation
• Distributional Calibration
• Chain of Cognitive Alignment
• Personality consistency