LAG: LLM agents for Leaderboard Auto Generation on Demanding

2025-02-25

AI Summary

The exponential growth of AI-related publications has made it increasingly difficult to track performance results across papers and conduct fair, reproducible comparisons. Method: This paper introduces the first end-to-end LLM-based agent framework for fully automated construction of research-topic leaderboards. It integrates multi-document understanding, structured result extraction, cross-paper experimental metadata alignment, standardized leaderboard generation, and interpretable quality assessment—enabled by multi-role agent collaboration, retrieval-augmented generation (RAG), and consistency verification. Contribution/Results: Evaluated across multiple AI subfields, the framework achieves high-fidelity leaderboard generation with 92.3% accuracy in human evaluation—significantly outperforming baseline approaches. It supports dynamic leaderboard updates and ensures full traceability across the entire pipeline, enabling transparent, scalable, and reproducible benchmarking.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Teamwork

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.

Problem

• Automatic leaderboard generation in AI
• Multi-document summarization challenges
• Fair comparison of experimental results

Innovation

• Automated leaderboard generation framework
• LLM-based multi-document summarization
• Systematic experiment result integration