SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization

2026-02-08

AI Summary

This work addresses the high computational cost of evaluating large language models, which typically relies on extensive inference samples. To enable efficient evaluation, the authors formulate the problem as a sparse optimization task, leveraging the inherent sparsity of the model–sample performance matrix. They introduce a novel approach that employs gradient descent to optimize anchor weights and integrates task-aware anchor selection with a candidate importance scoring mechanism, iteratively refining the set of anchors. By combining the representational capacity of multilayer perceptrons (MLPs) with the proposed optimization strategy, the method achieves low estimation error and high Kendall’s τ across multiple benchmarks, significantly improving the efficiency, robustness, and practicality of large language model evaluation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs. In this paper, we revisit the model-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated as a sparse optimization problem. Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP to handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task-aware refinement. Extensive experiments demonstrate the low estimation error and high Kendall's~$\tau$ of our method across a variety of benchmarks, showcasing its superior robustness and practicality in real-world scenarios. Code is available at {https://github.com/taolinzhang/SparseEval}.

Problem

• Large Language Models
• Efficient Evaluation
• Benchmarking
• Computational Cost
• Sparse Optimization

Innovation

• sparse optimization
• efficient evaluation
• anchor selection
• gradient descent
• large language models