Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment

2026-04-27

AI Summary

This work addresses the limitations of existing large language model alignment approaches, which predominantly focus on surface-level preferences or task utility and fail to effectively manage cognitive and normative risks while lacking explicit modeling of intervention timing and modality. The authors propose a risk-sensitive cognitive control framework that formalizes behaviors such as clarification, questioning, redirection, and refusal as explicit control actions to dynamically regulate the model’s evolving beliefs, commitments, and uncertainty. Key contributions include a taxonomy of friction-based interventions with associated structured functionals, a unified family of FPO (Friction-aware Policy Optimization) algorithms, and the first evaluation framework tailored to cognitive capabilities. By integrating reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions, the approach significantly enhances cognitive alignment across dimensions including clarifying behavior, calibration, contradiction resolution, proportional refusals, and informational efficiency.

Abstract

We propose Frictive Policy Optimization (FPO), a framework for learning language model policies that regulate not only what to say, but when and how to intervene in order to manage epistemic and normative risk. Unlike standard alignment methods that optimize surface-level preference or task utility, FPO treats clarification, verification, challenge, redirection, and refusal as explicit control actions whose purpose is to shape the evolution of belief, commitment, and uncertainty over time. We formalize alignment as a risk-sensitive epistemic control problem in which intervention decisions are selected based on their expected effect on downstream epistemic quality rather than on immediate reward alone. We introduce a compact taxonomy of frictive interventions, a structured friction functional that operationalizes multiple alignment failure modes, and a unified family of FPO methods spanning reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions. We further propose an evaluation framework that measures epistemic competence directly through clarification behavior, calibration, contradiction repair, refusal proportionality, and information efficiency. Together, these results provide a formal and algorithmic foundation for learning agents that are aligned not only in outcome, but in epistemic conduct.

Problem

• epistemic intervention
• risk-sensitive control
• reflective alignment
• frictive policy
• alignment failure

Innovation

• Frictive Policy Optimization
• Epistemic Control
• Risk-Sensitive Alignment
• Intervention Actions
• Cognitive Conduct