Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

2026-06-20

AI Summary

This work addresses the degradation of complex reasoning performance in large language models (LLMs) caused by alignment optimization, which can perturb refined semantic representations in intermediate layers despite default decoding from the final layer. The authors propose Confident Decoding, a training-free method that reveals an intrinsic Guess-Refine-Perturb dynamic within LLMs and formulates layer selection as an optimal stopping problem. By employing entropy-guided conservative backtracking search, the approach dynamically identifies the most reliable near-final layer for decoding. Confident Decoding is applicable to both dense and mixture-of-experts architectures, achieving significant performance gains on challenging benchmarks such as GPQA-Diamond, Omni-MATH, and HLE, with zero additional memory overhead and less than 2% latency increase.

Abstract

Autoregressive generation in large language models (LLMs) conventionally decodes from the final layer, assuming that deeper representations yield more reliable next-token predictions. We revisit this assumption by revealing a recurring Guess-Refine-Perturb dynamic: early layers form coarse guesses, intermediate layers refine reasoning-relevant semantics, and final layers can perturb these refined predictions toward generic or alignment-preferred tokens. We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer through entropy-guided conservative backward search. We further provide a theoretical formulation of layer selection as an optimal stopping problem, showing that under bounded projection noise and dominant late-stage alignment perturbation, our search rule filters perturbation while bounding the loss relative to the oracle refinement layer. Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks, including GPQA-Diamond, Omni-MATH, and HLE, with zero memory overhead and less than 2% latency increase. These results suggest dynamically bypassing final-layer perturbations can unlock stronger reasoning behavior from aligned LLMs.

Problem

• alignment tax
• autoregressive generation
• layer decoding
• reasoning performance
• large language models

Innovation

• Confident Decoding
• layer selection
• alignment tax
• autoregressive generation
• optimal stopping