On the Role of Inductive Bias in Time-Series Pretraining: A Case Study in Learning Generalizable Representations for Clinical Time Series

2026-05-25

AI Summary

Clinical time series analysis faces significant challenges including limited sample sizes, data heterogeneity, and protocol drift, necessitating generalizable representation learning approaches that jointly support classification and regression tasks. This work proposes PathoFM, a framework that systematically investigates the impact of inductive biases on representation transfer using gait data from spinal cord injury patients. The approach introduces a multi-objective self-supervised pretraining strategy that integrates local structural reconstruction, causal temporal continuity, and individual-specific contextual conditioning, all jointly optimized within a Transformer encoder. Experimental results demonstrate that this dynamics-driven hybrid objective substantially outperforms single-objective methods, achieving superior generalization performance across both cross-task and cross-subject settings.

Abstract

Clinical time-series learning is routinely constrained by small, heterogeneous cohorts and protocol drift, while its downstream use spans both classification (e.g., pathology diagnosis) and regression (e.g., temporal forecasting). These constraints make foundation-model pretraining appealing, but raises an important question of which inductive biases should the pretraining objective impose so that representations transfer across task types and subjects. We study this question in pathological gait analysis for spinal cord injury (SCI) via PathoFM, an encoder-centric transformer pretrained on multivariate gait windows with three complementary objectives: Local Completion (reconstruct contiguous masked spans to enforce local structure), Temporal Continuity (predict a masked mid-horizon continuation from an observed prefix to enforce smoothness and causal consistency), and Unsupervised In-Context Dynamics (support-query reconstruction conditioned on subject exemplar windows via attention). Empirically comparing objective families (grouping/contrastive, dynamics-based, and generative reconstruction), we find that dynamics-centric mixtures produce the most balanced transfer: grouping objectives favor discriminative margins but can degrade magnitude fidelity needed for continuous targets, whereas reconstruction-only objectives preserve waveform structure but may underperform on classification. Overall, combining local reconstruction with temporal continuity, and adding in-context conditioning when exemplar access is realistic, yields robust subject-generalizing representations.

Problem

• inductive bias
• time-series pretraining
• clinical time series
• representation transfer
• foundation models

Innovation

• inductive bias
• time-series pretraining
• foundation model
• self-supervised learning
• clinical time series