People Can Accurately Predict Behavior of Complex Algorithms That Are Available, Compact, and Aligned

2026-01-26

AI Summary

This study challenges the prevailing assumption that algorithmic simplicity is necessary for user predictability by proposing and empirically validating three conditions—cognitive accessibility, conceptual compactness, and human–algorithm execution alignment—under which users can construct accurate predictive mental models of complex algorithms, irrespective of their structural complexity. Through a preregistered experiment (N = 1,250) comparing 25 feed-ranking algorithms—including large language models and term-frequency-based methods—the research demonstrates that prediction accuracy significantly improves only when all three conditions are jointly satisfied, regardless of algorithmic complexity; conversely, the absence of any single condition renders even simple algorithms unpredictable. This work redefines the foundations of algorithmic interpretability by systematically identifying the necessary conditions for predictability in complex systems and elucidating their influence on the formation of user mental models.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Users trust algorithms more when they can predict the algorithms'behavior. Simple algorithms trivially yield predictively accurate mental models, but modern AI algorithms have often been assumed too complex for people to build predictive mental models, especially in the social media domain. In this paper, we describe conditions under which even complex algorithms can yield predictive mental models, opening up opportunities for a broader set of human-centered algorithms. We theorize that users will form an accurate predictive mental model of an algorithm's behavior if and only if the algorithm simultaneously satisfies three criteria: (1) cognitive availability of the underlying concepts being modeled, (2) concept compactness (does it form a single cognitive construct?), and (3) high alignment between the person's and algorithm's execution of the concept. We evaluate this theory through a pre-registered experiment (N=1250) where users predict behavior of 25 social media feed ranking algorithms that vary on these criteria. We find that even complex (e.g., LLM-based) algorithms enjoy accurate prediction rates when they meet all criteria, and even simple (e.g., basic term count) algorithms fail to be predictable when a single criterion fails. We also find that these criteria determine outcomes beyond prediction accuracy, such as which mental models users deploy to make their predictions.

Problem

• algorithm predictability
• mental models
• social media algorithms
• human-AI interaction
• algorithm transparency

Innovation

• predictive mental models
• algorithm interpretability
• human-centered AI
• cognitive alignment
• social media algorithms