Mitigating LLM-based p-Hacking by Preregistering for the Next LLM

2026-06-25

AI Summary

This study addresses the susceptibility of large language model (LLM) research to p-hacking, wherein researchers can manipulate statistical outcomes by tuning prompts, decoding parameters, or output formats. To mitigate this issue, the authors propose the first preregistration protocol designed for future, unknown LLMs: researchers finalize their analytical pipeline on currently available models, preregister both the analysis plan and a predefined set of qualifying future models, and then conduct confirmatory analyses on the first newly released model meeting those criteria. Leveraging the unpredictability of future model releases and the failure of configuration transfer across models, this approach fundamentally prevents p-hacking targeted at specific LLMs. Experiments across 20 LLMs from four providers show that the protocol blocks successful p-hacking migration in 73.9% and 72.7% of cases on two ground-truth tasks, with six out of seven adversarial configurations failing upon validation on a newly released model.

Abstract

Large language models (LLMs) are increasingly used to generate, classify, and annotate data whose outputs feed downstream hypothesis tests. However, LLM-based research is easy to p-hack: a researcher can tune the prompts, decoding parameters, or output format until a desired result is reached. We propose a protocol to mitigate p-hacking in LLM-based research: preregistering the experiment and eligible models, and then running it on the first eligible LLM that is released after the preregistration. The researcher finalizes the procedure on current models, preregisters the analysis plan together with a set of eligible future models, and runs the confirmatory analysis on the first eligible model released afterward. Because this model does not exist at commitment time, it cannot be hacked against; furthermore, configurations that hack one model frequently do not transfer to the next. We evaluate the protocol on two tasks whose true values are known. Across 20 models from four providers and 11 LLM-analysis configurations, the protocol would have blocked successful transfer of the p-hack in 73.9% and 72.7% of cases in the two tasks. Additional analyses reveal that mitigation remains substantial under several stress tests. Finally, putting money where our mouth is, we followed our own protocol and preregistered our experiment. The preregistered experiment confirmed the protocol's effectiveness: out of the 7 configurations that hacked the prior model, the hacking failed to carry over in 6 configurations on the first eligible model released afterward.

Problem

• p-hacking
• large language models
• preregistration
• hypothesis testing
• research reproducibility

Innovation

• p-hacking mitigation
• preregistration
• large language models
• reproducibility
• model transferability