Towards Human-Guided, Data-Centric LLM Co-Pilots

2025-01-17

AI Summary

Medical AI deployment is frequently hindered by poor raw data quality—including missing values, label noise, and domain heterogeneity—leading to degraded model performance. To address this, we propose CliMB-DC, a novel multi-agent framework that pioneers a “human-expert-guided + data-centric LLM-coordinated” architecture, comprising a strategic orchestrator and domain-specialized executors. It integrates human-in-the-loop (HITL) feedback, an automated data quality diagnostic toolkit, domain-knowledge-infused prompt engineering, and a modular, open-source design. CliMB-DC shifts the LLM co-pilot paradigm from model-centric to data-centric, systematically tackling data cleaning, label correction, and ML-readiness transformation in high-stakes domains like healthcare. Evaluated on real-world medical datasets, CliMB-DC significantly outperforms existing co-pilot baselines across key data curation metrics. Its extensible architecture enables cross-domain experts to autonomously drive AI deployment without deep ML expertise.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Machine learning (ML) has the potential to revolutionize healthcare, but its adoption is often hindered by the disconnect between the needs of domain experts and translating these needs into robust and valid ML tools. Despite recent advances in LLM-based co-pilots to democratize ML for non-technical domain experts, these systems remain predominantly focused on model-centric aspects while overlooking critical data-centric challenges. This limitation is problematic in complex real-world settings where raw data often contains complex issues, such as missing values, label noise, and domain-specific nuances requiring tailored handling. To address this we introduce CliMB-DC, a human-guided, data-centric framework for LLM co-pilots that combines advanced data-centric tools with LLM-driven reasoning to enable robust, context-aware data processing. At its core, CliMB-DC introduces a novel, multi-agent reasoning system that combines a strategic coordinator for dynamic planning and adaptation with a specialized worker agent for precise execution. Domain expertise is then systematically incorporated to guide the reasoning process using a human-in-the-loop approach. To guide development, we formalize a taxonomy of key data-centric challenges that co-pilots must address. Thereafter, to address the dimensions of the taxonomy, we integrate state-of-the-art data-centric tools into an extensible, open-source architecture, facilitating the addition of new tools from the research community. Empirically, using real-world healthcare datasets we demonstrate CliMB-DC's ability to transform uncurated datasets into ML-ready formats, significantly outperforming existing co-pilot baselines for handling data-centric challenges. CliMB-DC promises to empower domain experts from diverse domains -- healthcare, finance, social sciences and more -- to actively participate in driving real-world impact using ML.

Problem

• Data Handling Issues
• Machine Learning Performance
• Expertise Gap in Healthcare Industry

Innovation

• CliMB-DC Framework
• Data-Centric AI
• Multi-Role Collaboration