Modelpedia: A Catalog of Model Findings for the Meta-Science of AI

2026-09-01

AI Summary

为解决AI模型研究发现散乱问题，提出Modelpedia框架，利用LLM自动提取并整理论文中的模型发现，形成可搜索的公共目录。

Abstract

Scientific knowledge about AI models is produced faster than the community can organize it. Every few months a new foundation model reshapes the field and hundreds of papers, blogs, and technical reports document how each behaves or fails. Yet, these findings remain scattered and effectively unretrievable. To address this gap we present Modelpedia, an automated, LLM-assisted framework that extracts findings about models from published papers, links it to the model, dataset, method, and concept it concerns, and aggregates the result into a searchable public catalog. Applying the prototype to accepted ICLR 2024 and 2025 papers, we extract over a thousand findings and, treating the catalog itself as an object of study, run a meta-analysis of how the community investigates models. Now, we invite the community to explore, contribute to, and build on the open catalog, and to help establish model findings as a shared foundation for the meta-science of AI.

Problem

• AI models
• scientific knowledge
• organization
• retrieval

Innovation

• automated
• LLM-assisted framework
• findings extraction
• searchable catalog
• meta-analysis