🤖 AI Summary
This study addresses the lack of methodological transparency in chemical and biological (ChemBio) misuse risk assessment reports issued by leading AI developers—OpenAI, Anthropic, and Google DeepMind—in 2025. It conducts the first cross-organizational, systematic comparison grounded in the STREAM v1 framework. Using qualitative analysis and structured scoring across key dimensions—including test examples, prompt engineering conditions, and controllable elicitation strategies—the study reveals that all three reports omit concrete test materials and precise, reproducible triggering conditions. Its primary contributions are threefold: (1) introducing the first standardized transparency evaluation paradigm specifically designed for ChemBio risk assessments; (2) identifying shared methodological shortcomings across institutions; and (3) proposing an actionable, integrative pathway toward cross-organizational best practices. Collectively, these advances establish a foundational methodology to enhance scientific rigor, reproducibility, and industry alignment in AI safety evaluation.
📝 Abstract
Most frontier AI developers publicly document their safety evaluations of new AI models in model reports, including testing for chemical and biological (ChemBio) misuse risks. This practice provides a window into the methodology of these evaluations, helping to build public trust in AI systems, and enabling third party review in the still-emerging science of AI evaluation. But what aspects of evaluation methodology do developers currently include -- or omit -- in their reports? This paper examines three frontier AI model reports published in spring 2025 with among the most detailed documentation: OpenAI's o3, Anthropic's Claude 4, and Google DeepMind's Gemini 2.5 Pro. We compare these using the STREAM (v1) standard for reporting ChemBio benchmark evaluations. Each model report included some useful details that the others did not, and all model reports were found to have areas for development, suggesting that developers could benefit from adopting one another's best reporting practices. We identified several items where reporting was less well-developed across all model reports, such as providing examples of test material, and including a detailed list of elicitation conditions. Overall, we recommend that AI developers continue to strengthen the emerging science of evaluation by working towards greater transparency in areas where reporting currently remains limited.