SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest

2025-07-21

AI Summary

This work investigates how multi-task, multilingual, and multi-source learning jointly affect the robustness and generalization of pretrained language models. To this end, we propose the “Subset-of-Interest” (SOI) framework—the first systematic approach to identify and categorize six fine-grained learning behavior patterns during training. Integrating SOI partitioning, transition heatmaps, and dataset cartography, we uncover dynamic sample migration patterns across tasks, languages, and data sources. Building on SOI, we design a two-stage fine-tuning strategy that preserves in-distribution performance while substantially improving out-of-distribution (OOD) robustness: multi-source learning boosts OOD accuracy by up to 7%; multi-task learning yields significant gains on semantically related tasks; and the two-stage procedure further amplifies these benefits. Our framework provides an interpretable, reusable analytical paradigm and practical methodology for understanding and optimizing synergistic multi-paradigm learning.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This work investigates the impact of multi-task, multi-lingual, and multi-source learning approaches on the robustness and performance of pretrained language models. To enhance this analysis, we introduce Subsets of Interest (SOI), a novel categorization framework that identifies six distinct learning behavior patterns during training, including forgettable examples, unlearned examples, and always correct examples. Through SOI transition heatmaps and dataset cartography visualization, we analyze how examples shift between these categories when transitioning from single-setting to multi-setting configurations. We perform comprehensive experiments across three parallel comparisons: multi-task vs. single-task learning using English tasks (entailment, paraphrase, sentiment), multi-source vs. single-source learning using sentiment analysis datasets, and multi-lingual vs. single-lingual learning using intent classification in French, English, and Persian. Our results demonstrate that multi-source learning consistently improves out-of-distribution performance by up to 7%, while multi-task learning shows mixed results with notable gains in similar task combinations. We further introduce a two-stage fine-tuning approach where the second stage leverages SOI-based subset selection to achieve additional performance improvements. These findings provide new insights into training dynamics and offer practical approaches for optimizing multi-setting language model performance.

Problem

• Analyzing multi-setting training dynamics in pretrained language models
• Introducing Subsets of Interest (SOI) for learning behavior patterns
• Evaluating multi-task, multi-lingual, and multi-source learning impacts

Innovation

• Introduces Subsets of Interest (SOI) framework
• Uses SOI transition heatmaps for analysis
• Proposes two-stage SOI-based fine-tuning approach