On the Limits of Prompt-Conditioned Language Models as General-Purpose Learners

2026-06-22

AI Summary

This study investigates whether large language models relying solely on natural language prompts can serve as universal problem solvers. By modeling user–system interaction as a cheap-talk game and decoupling task inference from execution, the work integrates game theory, information theory, and PAC-Bayes generalization bounds to formally establish— for the first time—theoretical limits on prompt-based language models. These fundamental limitations arise from constraints imposed by the channel capacity of language and alignment requirements. The paper introduces two theoretical lower bounds: one on task expressibility and another on objective mismatch, demonstrating that certain families of tasks cannot be correctly solved by purely textual prompts, even with infinite data. The findings imply that achieving general intelligence necessitates moving beyond purely linguistic interfaces, such as by incorporating multimodal inputs or external memory mechanisms.

Abstract

Large Language Models (LLMs) are frequently portrayed as general-purpose solvers capable of solving arbitrary tasks. We argue that this view overlooks a fundamental constraint: language is a compressed and capacity-limited interface for conveying task information. Modelling User--System interaction as a bilevel \emph{cheap-talk} game, we analyse how latent tasks are encoded into prompts and reinterpreted under alignment and safety constraints. We introduce a conceptual decomposition separating task inference from execution and derive PAC-Bayes bounds that distinguish finite-sample estimation error from irreducible structural limitations. Our first main result establishes an \emph{expressivity floor}: language acts as a capacity-limited communication channel, and whenever the informational complexity of a task family exceeds the capacity of that channel, distinct tasks become unavoidably indistinguishable to the Solver, inducing a strictly positive error floor that cannot be eliminated by additional data, optimisation, or model scaling alone. We then establish an \emph{objective-misalignment floor}: when alignment constraints restrict the admissible output set, the User-ideal distribution may lie outside the feasible class, inducing an irreducible distortion. Together, these results yield a formal negative conclusion: prompt-conditioned LLMs are not universal problem solvers through prompting alone, as there exist task families for which correct behaviour is provably unattainable even in the infinite-data regime. More broadly, our analysis shows the limits of prompt-based generalisation arise from information-constrained communication and alignment-constrained objectives. This suggests that interfaces beyond natural language, including multimodal observations and, external memory, may reduce the inherent LLM limitations by increasing the task-relevant information available to the System.

Problem

• prompt-conditioned language models
• general-purpose learners
• information bottleneck
• alignment constraints
• task expressivity

Innovation

• cheap-talk game
• PAC-Bayes bounds
• expressivity floor
• objective-misalignment
• prompt-conditioned LLMs