Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets

2026-06-29

AI Summary

This work addresses the vulnerability of existing dataset distillation methods to noisy labels, which often compress both erroneous associations and useful signals, thereby compromising robustness. To mitigate this issue, the authors propose a novel optimization-trajectory-based distillation framework that integrates forgetting patterns with neighborhood consistency for the first time. This integration enables a progressive, selective guidance reweighting (SGR) mechanism, complemented by a teacher-model-generated residual auxiliary target (TIAT). Together, these components jointly suppress noise and preserve transferable knowledge without requiring a clean subset or relabeling. The proposed method consistently outperforms current distillation approaches across symmetric, asymmetric, and real-world label noise settings, yielding distilled datasets that are both cleaner and more informative.

Abstract

Dataset distillation (DD) condenses large corpora into compact, information-rich subsets for efficient training and reuse. However, under noisy supervision, DD risks condensing corrupted associations together with useful signals, degrading robustness. Conventional noisy-label remedies (sample selection, loss weighting, label correction) tightly couple noise estimation with model optimization, often require clean anchors, and can amplify confirmation bias-assumptions that are misaligned with DD's goal of compact, plug-and-play supervision. We therefore propose a trajectory-based DD framework that jointly suppresses noise and preserves transferable knowledge without relabeling or clean subsets. It comprises two complementary components: Selective Guidance Reweighting (SGR), which fuses global forgetting patterns (second-split forgetting) with local neighborhood consistency into a progressive reweighting scheme that prioritizes clean supervision along the teacher trajectory; and Teacher-Inspired Auxiliary Targets (TIAT), which inject auxiliary residual guidance distilled from intermediate teacher dynamics to reinforce informative signals while remaining internally consistent. Together, SGR and TIAT produce distilled datasets with cleaner and richer representations under noisy supervision. The framework is robust, label-preserving, computationally lightweight, and broadly applicable, yielding consistent gains over state-of-the-art DD baselines across symmetric, asymmetric, and real-world noise.

Problem

• Dataset Distillation
• Noisy Labels
• Robustness
• Trajectory-based Learning
• Supervision Compression

Innovation

• Dataset Distillation
• Noisy Labels
• Trajectory-Based Learning
• Selective Guidance Reweighting
• Teacher-Inspired Auxiliary Targets