LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routing

2026-08-09

AI Summary

This work addresses the challenge of "reasoning collapse"—a phenomenon wherein large language models, when applied to subjective tasks such as content moderation, over-reason and fail to align with human preferences. The study reveals a strong correlation between reasoning style and verification efficacy, and for the first time formally identifies and characterizes reasoning collapse. To mitigate this issue, the authors propose a dynamic persona-aware reasoning routing mechanism, integrated with conditional length penalty fine-tuning, reinforcement learning with verifiable rewards (RLVR), and large-scale role simulation. Experiments across four real-world recommendation platforms demonstrate that the proposed approach effectively suppresses reasoning collapse, achieving up to a 0.38 improvement in macro-F1 solely through adjustments to reasoning persona style, thereby significantly enhancing alignment performance on subjective tasks.

Abstract

Recommendation systems thrive on personalization, where ''correctness'' is rarely a binary truth but a matter of subjective human preference. As Large Language Models (LLMs) are deployed as autonomous verifiers of safety and quality guidelines, they face a distinctive challenge: context-aware preference alignment. Recent gains in Reinforcement Learning with Verifiable Rewards (RLVR) are indexed mostly on objective, mathematical tasks. Through a large-scale study spanning both proprietary and open-source models on four real-world verification tasks from a production recommender platform, we ask whether explicit reasoning generalizes to subjective, human-centric industry rubrics. We expose a fundamental vulnerability: rigid, math-centric reasoning traces actively degrade verification, and applying standard RLVR triggers a phenomenon we term reasoning collapse, in which the policy abandons deliberation in favor of rapid heuristic guessing. We introduce a conditional length-penalized post-training algorithm that intertwines verification accuracy with bounded reasoning length, halting collapse and recovering performance. Finally, we show that a reasoning trace's efficacy is tightly coupled with its socio-linguistic framing: across 1500 synthesized personas, verification accuracy swings by nearly 0.38 macro-F1 depending solely on the adopted reasoning persona---evidence that much subjective-verification error is really reasoning-style mismatch. This observation motivates a mid-training architecture that routes reasoning through contextually aligned personas. This work offers both a scalable algorithmic patch and a long-term architectural blueprint for aligning reasoning models with real-world subjective constraints.

Problem

• subjective tasks
• preference alignment
• reasoning collapse
• verification
• large language models

Innovation

• reasoning collapse
• subjective alignment
• dynamic reasoning routing
• length-penalized post-training
• socio-linguistic persona