Evaluating Document-Tuned Transformer Representations for Person-level Mental Health Assessment

2026-06-19

AI Summary

This study addresses the challenge of aggregating multiple texts from individuals for precise mental health assessment by systematically comparing base Transformers with document-level fine-tuned Transformers on longitudinal psychological data. It provides the first empirical validation of the advantages of document-level fine-tuned representations for this task. Through contrastive learning–based fine-tuning, layer-wise representation analysis, and robustness evaluations under various linguistic perturbations—including word deletion, synonym substitution, spelling errors, and back-translation—the fine-tuned models demonstrate significantly superior performance over baseline architectures, achieving a 13.4% improvement in Pearson correlation coefficient (p = 0.015). Moreover, these models exhibit enhanced capability in capturing linguistic uncertainty and greater robustness to textual variations.

Abstract

Person-level psychological assessment requires aggregating meaning across many messages from the same individual, a task that document-level training objectives were not explicitly designed for. We present a systematic, empirical comparison between architecturally matched traditional (a) base-transformers and (b) document-tuned-transformers (further contrastively fine-tuned at the document-level, sometimes referred to as "sentence transformers") under otherwise identical conditions. Comparing layer-wise and overall performance across two longitudinal mental health and psychological datasets, we find document-tuned models demonstrated a consistent improvement over base representations (increase in Pearson r of 13.4%, p=.015). Robustness analyses revealed document-tuned models remained more accurate under perturbations to word deletion, synonym replacement, typo injection, and back translation. Further, hedged language (e.g., `usually') was more characteristic of outcomes in document-tuned embeddings while abundance (e.g., `lot') was more characteristic of base-transformers, suggesting document-tuned models may better capture uncertainty. These results suggest representation choice impacts mental health prediction, document-tuned models often being more adept.

Problem

• mental health assessment
• document-level representation
• transformer models
• person-level prediction
• natural language processing

Innovation

• document-tuned transformers
• mental health assessment
• contrastive fine-tuning
• robustness analysis
• uncertainty modeling