Count-Based Approaches Remain Strong: A Benchmark Against Transformer and LLM Pipelines on Structured EHR

2025-11-01

AI Summary

Clinical prediction from structured electronic health records (EHRs) remains challenging, with growing interest in leveraging large language models (LLMs) as surrogates despite their computational and interpretability trade-offs. Method: This work conducts the first systematic benchmark comparing traditional count-based tabular models (e.g., LightGBM, TabPFN) against emerging hybrid LLM-based pipelines—including CLMBR and table-to-text summarization—across eight clinical prediction tasks using the EHRSHOT dataset. Contribution/Results: Count-based models achieve superior balance among predictive accuracy, model interpretability, and computational efficiency, matching or exceeding hybrid LLM pipelines—especially under low-data and high-noise conditions. The study bridges a critical gap by providing the first direct empirical comparison between classical statistical methods and LLM surrogate paradigms for EHR prediction, offering evidence-based guidance and methodological insights for clinical AI deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Structured electronic health records (EHR) are essential for clinical prediction. While count-based learners continue to perform strongly on such data, no benchmarking has directly compared them against more recent mixture-of-agents LLM pipelines, which have been reported to outperform single LLMs in various NLP tasks. In this study, we evaluated three categories of methodologies for EHR prediction using the EHRSHOT dataset: count-based models built from ontology roll-ups with two time bins, based on LightGBM and the tabular foundation model TabPFN; a pretrained sequential transformer (CLMBR); and a mixture-of-agents pipeline that converts tabular histories to natural-language summaries followed by a text classifier. We assessed eight outcomes using the EHRSHOT dataset. Across the eight evaluation tasks, head-to-head wins were largely split between the count-based and the mixture-of-agents methods. Given their simplicity and interpretability, count-based models remain a strong candidate for structured EHR benchmarking. The source code is available at: https://github.com/cristea-lab/Structured_EHR_Benchmark.

Problem

• Benchmarking count-based models against LLM pipelines on EHR data
• Comparing structured EHR prediction methods using EHRSHOT dataset
• Evaluating performance of traditional versus modern clinical prediction approaches

Innovation

• Count-based models using LightGBM and TabPFN
• Transformer model CLMBR for sequential EHR data
• Mixture-of-agents pipeline converting EHR to text