Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Models

2025-03-03

AI Summary

Current language models lack intrinsic risk-awareness mechanisms, rendering them incapable of autonomously deciding when to answer, refuse, or abstain from speculative responses under uncertainty. Method: We formally define the “risk-aware decision-making” task and propose a dynamic, skill-decomposition-based framework that employs chain-of-thought prompting to jointly model multi-level reasoning and refusal behavior, complemented by a risk-sensitive evaluation protocol. Contribution/Results: Experiments reveal that even state-of-the-art reasoning models reliably perform risk-informed decisions only when guided by explicit prompt chains; static refusal strategies exhibit fundamental limitations. Our work establishes the first principled benchmark for risk-adaptive decision-making, providing both theoretical foundations and scalable technical pathways toward building truly autonomous and safety-aware intelligent agents.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Knowing when to answer or refuse is crucial for safe and reliable decision-making language agents. Although prior work has introduced refusal strategies to boost LMs' reliability, how these models adapt their decisions to different risk levels remains underexplored. We formalize the task of risk-aware decision-making, expose critical weaknesses in existing LMs, and propose skill-decomposition solutions to mitigate them. Our findings show that even cutting-edge LMs--both regular and reasoning models--still require explicit prompt chaining to handle the task effectively, revealing the challenges that must be overcome to achieve truly autonomous decision-making agents.

Problem

• Investigating risk-aware decision-making in language models
• Exposing weaknesses in existing language models' refusal strategies
• Proposing skill-decomposition solutions for better decision-making

Innovation

• Formalize risk-aware decision-making task
• Propose skill-decomposition solutions
• Use explicit prompt chaining for effectiveness