feature engineering

Modeling heterogeneous, asynchronous, high-dimensional, and mixed-structured/unstructured clinical data in intensive care units (ICUs) remains a fundamental challenge. To address this, we propose ICU-BERT—the first Transformer-based pretraining framework specifically designed for critical care settings. Our method introduces (1) a multi-task self-supervised pretraining paradigm coupled with a multi-token asynchronous input strategy to accommodate irregular temporal dynamics, and (2) integration of dense semantic embeddings from biomedical large language models, enabling end-to-end, feature-engineering-free robust representation learning. Trained on MIMIC-IV, ICU-BERT achieves state-of-the-art performance across five diverse downstream clinical tasks—including mortality risk prediction and ICD coding—as well as on four independent external ICU datasets. Results demonstrate substantial improvements in generalizability and clinical utility of AI-driven decision support systems, validating its effectiveness in real-world critical care applications.

This study addresses the automatic extraction of physiologically or clinically meaningful features from intracranial pressure (ICP) waveforms to enhance diagnostic capabilities. Leveraging ICP data from 60 patients, the authors first segmented the signals into individual cardiac cycles and then developed a convolutional neural network to classify seven distinct body positions. Notably, they introduced a neural attention mechanism to localize critical regions within the ICP waveform—an approach that not only successfully identified waveform segments significantly associated with different body positions but also established a scalable, data-driven framework for feature discovery. This work provides a novel pathway toward deeper physiological interpretation of ICP dynamics and improved clinical decision-making.

Existing data pruning methods suffer from complex designs and poorly understood mechanisms of their key components, hindering progress in the field. Method: This paper pioneers a decoupled framework that separates data pruning into two orthogonal modules—“data representation” and “selection algorithm”—and systematically evaluates their independent contributions to instance selection efficacy in NLP model training. Through theoretical analysis and extensive empirical evaluation across multiple benchmarks (including gradient- and embedding-based representations and diverse selection algorithms), we assess their relative impact. Contribution/Results: We find that representation quality dominates selection algorithm choice: high-fidelity representations (e.g., training gradients) substantially improve pruning effectiveness, whereas no selection algorithm exhibits consistent superiority across tasks; even for identical objectives, different algorithms yield markedly divergent selected subsets. This work establishes an interpretable, reusable analytical framework for data pruning and identifies representation optimization—not algorithmic refinement—as the primary lever for enhancing pruning efficiency.

Existing approaches to modeling medical tabular data often neglect column-level contextual information (e.g., header semantics), require labor-intensive, manual preprocessing, and lack semantic interpretability. Method: We propose TabText—a novel framework that systematically encodes tabular structure into promptable natural language text, enabling end-to-end semantic representation learning via large language models (LLMs). TabText integrates structure-aware prompt engineering with a multi-task health prediction fine-tuning paradigm, supporting zero- or light-preprocessing modeling while remaining compatible with conventional feature fusion. Results: Evaluated on nine clinical prediction tasks, TabText alone establishes high-performance, lightweight baselines. When fused with traditional features, it yields an average AUC improvement of 6%, with even the worst-case gain reaching 6%, significantly enhancing model robustness and generalization across diverse healthcare scenarios.

This work addresses the disconnect between existing large language model (LLM)-based clinical tabular feature engineering approaches and downstream predictive models, which struggle with class imbalance, heterogeneous features, and stringent interpretability requirements in healthcare data. To bridge this gap, the authors propose MedFeat, a novel framework that, for the first time, integrates model-awareness and interpretability-driven mechanisms into LLM-assisted feature engineering. MedFeat iteratively generates new features that align with the inductive biases of downstream models and are inherently interpretable by fusing feature importance signals with real-time model feedback. Evaluated across multiple real-world clinical prediction tasks, MedFeat significantly outperforms current methods, achieving an average performance improvement exceeding 10%, while maintaining compatibility with diverse model architectures.

This work proposes a context-aware multi-model optimization approach to overcome the limitations of relying on a single optimal model. By systematically identifying multiple models that exhibit substantially different feature selections yet achieve comparable predictive performance, the method preserves overall accuracy while enhancing interpretability. Applied to the METABRIC gene expression dataset, the approach successfully generates a diverse ensemble of high-performing models, uncovering multiple plausible biological interpretation pathways underlying the data. Compared to baseline methods, the resulting models demonstrate superior trade-offs between feature dissimilarity and performance consistency, thereby significantly improving both model interpretability and scientific insight.

This study addresses a critical limitation in existing natural language explanation methods for tabular data: their neglect of users’ inherent understandability of input features, which undermines explanation effectiveness. To bridge this gap, the work introduces, for the first time, a feature understandability scale into a cost-sensitive feature selection framework and proposes a joint optimization algorithm that simultaneously maximizes model classification accuracy and the understandability of selected features. Experimental results on two real-world datasets demonstrate that the proposed approach significantly enhances the surface understandability of generated explanations while maintaining high predictive performance. These findings validate the feasibility of co-optimizing accuracy and understandability, marking a pivotal shift in explainable AI—from design-driven paradigms toward alignment with user cognition.

This work investigates the optimal choice of representation dimensionality to achieve maximal generalization performance in settings where pretraining and downstream task data exhibit significant imbalance. By modeling pretraining as principal component analysis on unlabeled data and downstream learning as linear regression on labeled data, the authors employ high-dimensional statistical analysis to derive, for the first time, exact expressions for training and generalization errors in the high-dimensional asymptotic limit. Their theoretical results establish a quantitative relationship among representation dimensionality, data scale, and task alignment, yielding an analytical condition for the optimal representation size: low-dimensional compressed representations are preferable when labeled downstream data are scarce but pretraining data are abundant, whereas high-dimensional representations are advantageous in the opposite regime. The study further quantifies the extent to which unlabeled data can substitute for labeled samples and validates the proposed mechanism in both autoencoders and large language models.

This study addresses the challenge that ICU time-series prediction models trained on large hospitals often fail to generalize to resource-constrained smaller facilities. To tackle this, the authors frame cross-institutional model transfer as a domain-incremental continual learning task, enabling adaptation to the target domain’s data distribution while preserving knowledge from the source domain. They establish the first benchmark for domain-incremental continual learning on ICU time-series data and systematically evaluate representative methods—such as experience replay and Elastic Weight Consolidation (EWC)—using real-world multicenter datasets to assess their capacity for knowledge retention and transfer. Their analysis reveals substantial differences in measurement distributions and frequencies across ICUs in distinct U.S. regions, thereby demonstrating both the promise and limitations of current continual learning approaches in clinical settings.

This study systematically investigates the mechanisms by which data scale, model complexity, and input modality influence the generalization performance of vision models. Within a unified experimental framework, the authors conduct controlled and large-scale ablation studies on synthetic functions and the CIFAR dataset, employing polynomial fitting, diverse CNN and Transformer architectures, and multimodal inputs—including RGB, grayscale, gradients, edges, and wavelet representations—to quantitatively compare the effects of these three core factors for the first time. The findings reveal that increasing training data consistently enhances generalization; greater model complexity yields non-monotonic improvements; removing color information substantially degrades performance; and the efficacy of explicit handcrafted priors is highly dependent on model architecture.