🤖 AI Summary
This study addresses a common misconception in clinical machine learning that equates the sheer scale of large language model (LLM)-generated text with its informational content, often overlooking pervasive redundancy. The authors propose a source-based redundancy decomposition method to classify each token within 2.51 billion tokens extracted via multi-agent LLM processing from 167,034 patient narratives. At production scale, they reveal that only 10.9% of tokens represent unique content, while 79.4% are redundant—indicating that raw token counts overestimate true information by approximately ninefold. The work further distinguishes two distinct redundancy mechanisms and demonstrates that redundancy stems from pipeline design rather than inherent LLM properties. Through lossless compression analysis and controlled fine-tuning experiments, the study shows that deduplication significantly improves clinical encoder performance on external disease recognition tasks under identical token budgets.
📝 Abstract
Clinical machine learning increasingly relies on training corpora generated by large language models (LLMs) rather than annotated by clinicians, and such corpora are described and reused largely on the basis of their reported scale. We test whether volume reflects information content. Analysing the complete output of a multi-agent clinical extraction pipeline applied to 167,034 patient narratives, 2.51 billion generated tokens across the ten text-bearing channels of an eleven-channel pipeline, we introduce Provenance-based Redundancy Decomposition, a token-level classification of the entire output by source. Only 10.9% of the output is trainable-unique content while 79.4% is redundant; raw token count overstates information content by roughly ninefold. The redundancy arises through two distinct mechanisms, verbatim copying of source context into per-item fields, and duplication of generated text across records, of which only the former is losslessly removable. An independent, model-free analysis based on lossless compression confirms the redundancy, recovering the two mechanisms without reference to the provenance labels. One pipeline channel carries almost no redundancy, showing that the level of redundancy depends on how each channel is structured rather than being a fixed property of LLM extraction. Because uncorrected redundancy up-weights the longer, more complex presentations that generate the most items, it skews the token-level training distribution of the corpus, a property we measure directly. In a controlled downstream test, de-duplicating the corpus before adaptation improved a clinical encoder on external disease-recognition benchmarks at equal token budget, robustly across adaptation depths and replicated on a second benchmark, confirming that the redundancy carries a measurable cost beyond storage. The classification tool is released openly.