The Value of a Prompt: An LLM-Relative Kolmogorov-Complexity Approach

2026-08-17

AI Summary

This study addresses the challenge of quantifying prompt value by proposing a novel paradigm grounded in LLM relative probability and Levin-Kolmogorov complexity. The proposed method defines prompt value as algorithmic mutual information, measuring it through efficient estimation of the reduction in generation difficulty attributable to the input. Experimental results validate this theoretical framework, demonstrating that a prompt with b bits of value increases the median token cost of unprompted generation by a factor of 2^b. Beyond providing a computable metric for value assessment, this work elucidates the fundamental role of prompts in reducing the generative complexity of large language models from the perspective of algorithmic information theory.

Abstract

In a world where valuable artifacts are increasingly created, completed, or processed by LLMs, the central economic question is not only what the LLM can produce, but what \emph{value} remains in the inputs (i.e., the prompts) we provide to it. Given a prompt, hint, critique, problem statement, or partial solution that helps an LLM produce an artifact $z$---a proof, program, design, or scientific hypothesis---how should we measure the value of that input? Intuitively, an input is valuable when it makes the target artifact easier for the model to generate: either by increasing its sampling probability, or by reducing the thinking time needed to find it. We propose a computational Levin--Kolmogorov complexity approach to this problem, by appropriately replacing the universal Turing machine in the classical definitions by the LLM itself. Concretely, we introduce an LLM-relative notion of \emph{probabilistic Levin--Kolmogorov complexity} $pKt$---treating the model's thinking as the random tape of the program, and charging logarithmically for it in Levin's manner---and define prompt value as algorithmic mutual information with respect to $pKt$. This captures the intuition above: a prompt having $b$ bits of value for an artifact $z$ makes $z$ $2^b$ times ``easier to obtain'', by multiplying the success probability by $2^b$, by dividing the required computation by $2^b$, or by any corresponding tradeoff between probability and computation. In contrast to the classical notion of algorithmic mutual information, ours is efficiently estimable. We additionally show that, under a natural reproduction experiment, a prompt value of \(b\) bits means that reproducing \(z\) without the prompt has median token cost \(2^b\) times that of reproducing it with the prompt.

Problem

• Prompt Value
• Large Language Models
• Kolmogorov Complexity
• Algorithmic Mutual Information

Innovation

• Probabilistic Levin-Kolmogorov Complexity
• Prompt Value
• Algorithmic Mutual Information
• LLM-relative Complexity