Benchmarking LLM-based Agents for Single-cell Omics Analysis

2025-08-16

AI Summary

Conventional single-cell multi-omics analysis pipelines are rigid, and AI agents lack systematic benchmarking in this domain. Method: We introduce the first AI agent benchmark tailored to single-cell multi-omics, comprising 50 real-world biomedical tasks, a unified execution platform, and multidimensional evaluation metrics (planning, code generation, knowledge integration, collaboration). Our technical framework integrates large language models (LLMs), retrieval-augmented generation (RAG), self-reflection, and multi-agent coordination to support cross-species, multi-omics, and multi-technology scenarios. Contribution/Results: Experiments demonstrate that self-reflection and planning capabilities critically enhance performance; role-specialized multi-agent systems significantly improve task completion rates and execution efficiency. Grok-3-beta achieves state-of-the-art performance. Code generation quality and context-aware retrieval are identified as key bottlenecks. This work provides empirical foundations and methodological guidance for trustworthy AI agent deployment in biomedicine.

Technology Category

• Multiagent Systems: Multiagent Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

The surge in multimodal single-cell omics data exposes limitations in traditional, manually defined analysis workflows. AI agents offer a paradigm shift, enabling adaptive planning, executable code generation, traceable decisions, and real-time knowledge fusion. However, the lack of a comprehensive benchmark critically hinders progress. We introduce a novel benchmarking evaluation system to rigorously assess agent capabilities in single-cell omics analysis. This system comprises: a unified platform compatible with diverse agent frameworks and LLMs; multidimensional metrics assessing cognitive program synthesis, collaboration, execution efficiency, bioinformatics knowledge integration, and task completion quality; and 50 diverse real-world single-cell omics analysis tasks spanning multi-omics, species, and sequencing technologies. Our evaluation reveals that Grok-3-beta achieves state-of-the-art performance among tested agent frameworks. Multi-agent frameworks significantly enhance collaboration and execution efficiency over single-agent approaches through specialized role division. Attribution analyses of agent capabilities identify that high-quality code generation is crucial for task success, and self-reflection has the most significant overall impact, followed by retrieval-augmented generation (RAG) and planning. This work highlights persistent challenges in code generation, long-context handling, and context-aware knowledge retrieval, providing a critical empirical foundation and best practices for developing robust AI agents in computational biology.

Problem

• Lack of comprehensive benchmark for AI agents in single-cell omics analysis
• Evaluating agent capabilities across diverse frameworks and real-world tasks
• Addressing challenges in code generation, context handling, and knowledge retrieval

Innovation

• Unified platform for diverse agent frameworks
• Multidimensional metrics for cognitive program synthesis
• 50 real-world single-cell omics analysis tasks