Improving LLMs via Validator-to-Generator Alignment

2026-07-02

AI Summary

This work addresses the inconsistency between generation and verification in large language models, where generated answers are often deemed invalid upon re-evaluation. To mitigate this issue, the study introduces answer prior frequency into generator–verifier (G–V) consistency modeling for the first time, proposing a frequency-corrected consistency criterion and a corresponding Frequency-Corrected Pairwise Alignment (FCPA) training objective. Within a rational agent multi-answer question-answering framework, FCPA aligns the verifier’s judgments with frequency-adjusted generation scores. Experimental results demonstrate that FCPA improves Pearson correlation by up to 27 percentage points on IFEval and HumanEval benchmarks, significantly enhancing both G–V consistency and generation performance while preserving verifier quality.

Abstract

Large language models are inconsistent: varying prompts or including unrelated information can lead to unexpected changes in model outputs. The generator-validator (G-V) gap is one manifestation of this phenomenon, where LLMs generate responses that they then deem as invalid if re-queried to validate them. In this work, we introduce a new formulation of G-V consistency that involves a principled correction for utterance frequency. Specifically, generators often assign low likelihood to valid strings simply because those strings are a priori unlikely, which makes naive notions of G-V consistency unworkable. We show that under a natural model of rational agents answering questions with multiple answers, consistency of the validator with a frequency-corrected generator score emerges naturally. Our method, \emph{\FCPAname} (\FCPA), is a training objective implementing frequency-corrected G-V consistency for real-world LLMs. Our experimental results show that training with \FCPA{} substantially improves both G-V consistency and generator performance over prior methods, with gains of up to $+27$pp in Pearson correlation on IFEval and HumanEval, while preserving validator quality across all evaluated tasks.

Problem

• LLMs
• generator-validator gap
• consistency
• utterance frequency
• validation

Innovation

• Generator-Validator Alignment
• Frequency Correction
• LLM Consistency
• FCPA
• Rational Agent Modeling