Selective Fine-Tuning of GPT Architectures for Parameter-Efficient Clinical Text Classification

2026-03-14

AI Summary

This study addresses the challenges of clinical text classification, which include scarce annotated data, complex medical terminology, and the high computational cost of full-parameter fine-tuning of large language models. To overcome these limitations, the authors propose a parameter-efficient selective fine-tuning approach for GPT-2 that updates only the final Transformer block, layer normalization modules, and a lightweight classification head. This strategy preserves the model’s pretrained linguistic representations while reducing trainable parameters to less than 6% of the original. Evaluated on 50,000 radiology reports from the MIMIC-IV-Note dataset with CheXpert-derived labels, the method achieves approximately 91% accuracy—significantly outperforming both classifier-only training and full-model fine-tuning baselines—thus offering an effective balance between performance and computational efficiency.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The rapid expansion of electronic health record (EHR) systems has generated large volumes of unstructured clinical narratives that contain valuable information for disease identification, patient cohort discovery, and clinical decision support. Extracting structured knowledge from these free-text documents remains challenging because clinical language is highly specialized, labeled datasets are limited, and full fine-tuning of large pretrained language models can require substantial computational resources. Efficient adaptation strategies are therefore essential for practical clinical natural language processing applications. This study proposes a parameter-efficient selective fine-tuning framework for adapting GPT-2 to clinical text classification tasks. Instead of updating the entire pretrained model, the majority of network parameters are frozen, and only the final Transformer block, the final layer normalization module, and a lightweight classification head are updated during training. This design substantially reduces the number of trainable parameters while preserving the contextual representation capabilities learned during pretraining. The proposed approach is evaluated using radiology reports from the MIMIC-IV-Note dataset with automatically derived CheXpert-style labels. Experiments on 50,000 radiology reports demonstrate that selective fine-tuning achieves approximately 91% classification accuracy while updating fewer than 6% of the model parameters. Comparative experiments with head-only training and full-model fine-tuning show that the proposed method provides a favorable balance between predictive performance and computational efficiency. These results indicate that selective fine-tuning offers an efficient and scalable framework for clinical text classification.

Problem

• clinical text classification
• parameter-efficient fine-tuning
• electronic health records
• natural language processing
• pretrained language models

Innovation

• selective fine-tuning
• parameter-efficient adaptation
• clinical text classification
• GPT-2
• MIMIC-IV-Note