Reflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making

2025-10-05

AI Summary

Conversational AI systems, particularly large language models (LLMs), are increasingly deployed in high-stakes life decisions; however, their solution-oriented design undermines users’ metacognition and reflective capacity. Method: This paper introduces “Pre-Decisional Reflection” (PDR) and proposes PROBE—a novel, dual-dimensional quantitative framework that objectively measures reflection breadth (diversity of considerations) and depth (complexity of reasoning) in open-ended pre-decisional texts. Integrating LLM-based text analysis, intercoder reliability assessment, and qualitative content analysis, the study implements multidimensional, structured measurement of reflective discourse. Contribution/Results: Empirical validation confirms PROBE’s high reliability and construct validity. Crucially, it reveals a significant discrepancy between users’ subjective perception of reflection and objectively measured reflection levels, alongside systematic interindividual heterogeneity in reflective thinking patterns. This work advances decision support paradigms from problem-solving–oriented assistance toward reflection-enabling scaffolding.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.

Problem

• Quantifying thought patterns in personal decision making
• Assessing pre-decision reflection breadth and depth
• Enhancing self-awareness against AI-driven solution orientation

Innovation

• PROBE framework quantifies pre-decision reflection dimensions
• Measures reflection breadth and depth using coding reliability
• Reveals hidden thought patterns to enhance self-awareness