Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation

2026-06-15

AI Summary

This study addresses a critical gap in evaluating large language models (LLMs) by proposing CEO-Bench, the first multi-agent benchmark designed to assess CEO-level strategic decision-making. Unlike existing evaluations that focus on isolated cognitive tasks, CEO-Bench simulates LLMs as CEOs integrating conflicting advice from four functional executives (CFO, CTO, COO, CMO), each providing private signals under information asymmetry, organizational constraints, and temporal dependencies. The framework evaluates performance across four dimensions: role integration, conditional assertiveness, historical sensitivity, and solution effectiveness, tested across 13 scenarios with five state-of-the-art models. Results reveal that while models generally exhibit structural validity, they display significant divergence in strategic calibration, often succumbing to overreliance on a single advisor, excessive conservatism in ambiguous contexts, and historical amnesia—highlighting a structural trade-off between integrative depth and decision assertiveness.

Abstract

Evaluating the decision-making capabilities of large language models (LLMs) is a growing research priority, yet existing benchmarks focus on isolated cognitive tasks such as reasoning, knowledge retrieval, and economic rationality in stylized settings. These evaluations overlook the defining challenge of real executive decision-making: integrating conflicting recommendations from specialized stakeholders under information asymmetry, organizational constraints, and temporal dependencies. We introduce \textsc{CEO-Bench}, a multi-agent benchmark that evaluates LLMs on CEO-level strategic resource reallocation -- the process of redirecting capital across business units in a multi-round, constraint-rich organizational environment. In \textsc{CEO-Bench}, LLM agents receive conflicting advice from four role-conditioned C-suite advisors (CFO, CTO, COO, CMO), each with private signals and distinct priorities, and must synthesize these into a concrete allocation plan evaluated along four dimensions: role integration, conditional boldness, history-sensitive judgment, and plan validity. Experiments across five frontier models on 13 scenarios reveal that all models achieve high structural validity but diverge sharply on strategic calibration -- the hardest capability layer. We identify systematic failure modes including single-advisor capture, conservative default under ambiguity, and historical amnesia, and uncover a structural integration-boldness tradeoff: models that engage more deeply with conflicting perspectives tend to produce less decisive action. These findings delineate the current capability boundary of LLMs as organizational decision-makers and inform the design of future AI-assisted executive systems.

Problem

• executive decision-making
• strategic resource reallocation
• multi-agent simulation
• information asymmetry
• organizational constraints

Innovation

• multi-agent simulation
• strategic decision-making
• resource reallocation
• role integration
• LLM evaluation