CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculation

2025-02-17

AI Summary

This paper addresses the weak evaluability and poor interpretability of large language models (LLMs) in Chinese Medical Quality Control Indicator Calculation (MQCIC). To this end, we introduce CMQCIC-Bench—the first Chinese MQCIC benchmark—comprising 785 real-world electronic medical records and 76 clinically validated quality indicators. We further propose CF-IR, a novel method that decouples clinical fact verification from reasoning rule application, enhanced by a semi-automatic rule representation strategy. CF-IR significantly improves both accuracy and interpretability in computing complex clinical indicators, outperforming chain-of-thought (CoT) baselines on MQCIC. Extensive experiments across 20 general-purpose and medical-domain Chinese LLMs systematically expose critical bottlenecks in clinical fact checking and rule-based reasoning. The benchmark dataset, evaluation protocols, and implementation code are publicly released to foster reproducible research.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Medical quality control indicators are essential to assess the qualifications of healthcare institutions for medical services. With the impressive performance of large language models (LLMs) like GPT-4 in the medical field, leveraging these technologies for the Medical Quality Control Indicator Calculation (MQCIC) presents a promising approach. In this work, (1) we introduce a real-world task MQCIC and propose an open-source Chinese electronic medical records (EMRs)-based dataset (CMQCIC-Bench) comprising 785 instances and 76 indicators. (2) We propose a semi-automatic method to enhance the rule representation. Then we propose the Clinical Facts-based Inferential Rule (CF-IR) method that disentangles the clinical fact verification and inferential rule reasoning actions. (3) We conduct comprehensive experiments on 20 representative LLMs, covering general and medical models. Our findings reveal that CF-IR outperforms Chain-of-Thought methods in MQCIC tasks. (4) We conduct an error analysis and investigate the capabilities of clinical fact verification and inferential rule reasoning, providing insights to improve performance in the MQCIC further. The dataset and code is available in this repo https://anonymous.4open.science/r/C-MQCIC-1151.

Problem

• Develops a Chinese benchmark for medical quality control
• Introduces a dataset for evaluating large language models
• Proposes a method for clinical fact verification and reasoning

Innovation

• Introduces CMQCIC-Bench dataset
• Proposes CF-IR method
• Evaluates 20 LLMs comprehensively