The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

2026-06-16

AI Summary

This study reveals that standard multiple-choice benchmarks overestimate the practical capabilities of pruned large language models in open-ended generation tasks. Through systematic evaluation on multilingual question-answering tasks, the authors compare model performance before and after pruning across both multiple-choice recognition and open-generation settings, uncovering a previously unreported “benchmark illusion”: highly sparse pruned models often retain the ability to recognize correct answers but struggle to generate them autonomously. Using pruning methods such as Wanda and evaluating under diverse decoding strategies—including greedy search, beam search, and sampling, with and without in-context examples—the analysis shows that most generation failures stem not from the absence of correct answers in the model’s output distribution but from their degraded ranking. Consequently, appropriate decoding can recover much of the lost performance, underscoring the necessity of prioritizing generative capability when assessing the utility of compressed models.

Abstract

Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.

Problem

• pruned LLMs
• benchmark illusion
• open generation
• multiple-choice evaluation
• model compression

Innovation

• pruned LLMs
• benchmark illusion
• open generation
• recognition-only errors
• model compression