NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision

2026-07-09

AI Summary

This work addresses the fundamental challenge that ambiguous natural language task specifications render the supervision signal from large language models (LLMs) insufficient to uniquely identify the true target, thereby inducing irreducible identifiability error in learners. To tackle this, the authors propose the NL-PAC framework, which defines a set of acceptable labels via a fixed LLM threshold decoding rule and leverages unlabeled data to estimate the diameter of pointwise acceptable target classes. This approach yields the first provable minimax risk lower bound under LLM-mediated supervision with specification ambiguity. The bound is achieved by a data-independent strategy combining PAC-style finite-sample confidence bounds, target-blind supervision modeling, and stochastic minimax analysis. Auditing experiments on Qwen 2.5–3B reveal that only specific prompts yield positive model-relative certificates, whereas prompt rewrites or precise rule enforcement fail to satisfy acceptability conditions.

Abstract

Large language models increasingly provide labels, evaluations, and feedback for tasks specified in natural language. When a specification admits multiple readings but the supervision channel does not reveal which is operative, additional labels reduce sampling error without resolving the resulting identification problem. We introduce Natural Language PAC (NL-PAC), a framework that uses a fixed model's thresholded decoding law to define admissible labels and candidate targets. The probability that multiple labels are admissible equals the diameter of the pointwise-admissible target class, and under target-blind supervision every learner incurs worst-case risk of at least half this diameter, at every sample size; the exact randomized minimax risk over this class is attained by a data-independent strategy. Finite-sample confidence bounds make these quantities certifiable from held-out unlabeled inputs. In a frozen Qwen~2.5--3B audit, one prespecified prompt yields a positive model-relative certificate, whereas a paraphrase and exact-rule controls yield zero. A held-out bridge audit finds that supplied candidate reading clauses fail the admissibility condition needed to transfer the certificate to coherent readings. The guarantee is specific to the audited model, prompt, threshold, and input distribution; extending it to human interpretations requires external validation.

Problem

• specification ambiguity
• LLM-mediated supervision
• minimax risk
• identification problem
• natural language specifications

Innovation

• Natural Language PAC
• specification ambiguity
• minimax risk
• certifiable guarantees
• LLM-mediated supervision