MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration

2026-06-22

AI Summary

This work addresses the high computational cost of repeatedly evaluating large language models—especially across variants such as quantized or fine-tuned versions—on extensive benchmarks, a burden particularly acute on edge devices. The authors propose a parameter-free sampling strategy that leverages minimal per-item log outputs from a calibrated model to identify, via Monte Carlo simulation, the smallest evaluation subset guaranteeing accuracy drift within a specified bound. This fixed subset enables efficient assessment without requiring additional predictive layers. The approach drastically reduces evaluation data volume—by 54%, 89%, and 70% on IFEVAL, MMLU, and GSM8K, respectively—while limiting maximum accuracy drift to ≤2.62 percentage points for BF16 models and yielding average drifts of 0.77–3.59 percentage points on NPU-deployed models. Evaluation speedups reach 2.7–8.1× on GPUs and 1.7–2.0× on NPUs.

Abstract

Evaluating LLMs across many model variants -- quantized, fine-tuned, or deployment-specific -- requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing subset selection methods reduce this cost but depend on large calibration pools or learned prediction layers. We introduce MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed. MINCE reduces IFEVAL by 54\%, MMLU by 89\%, and GSM8K by 70\% with maximum drift $\leq$2.62\,pp on BF16 models and mean drift of 0.77--3.59\,pp on held-out NPU models, while delivering median GPU evaluation speedups of 2.7--8.1$\times$ and NPU evaluation speedups of 1.7--2.0$\times$. The method is robust to calibration pool size and achieves lower drift than tinyBenchmarks (12$\times$ lower on MMLU, 3.3$\times$ on GSM8K) while using 57$\times$ fewer calibration models.

Problem

• LLM evaluation
• dataset reduction
• edge hardware
• calibration models
• evaluation efficiency

Innovation

• Monte Carlo simulation
• dataset distillation
• LLM evaluation
• calibration-free subset selection
• edge deployment