See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning

2026-08-30

AI Summary

为了解决无监督动作分割问题，提出了一种基于谱-时序表示学习的方法SpecT-OT，通过优化传输伪标签的质量来提高动作分类和时间组织的准确性。

Abstract

Unsupervised action segmentation aims to discover latent action categories and their temporal organization without action annotations. Optimal transport-based methods provide structured frame-to-action assignments, however, their pseudo-label quality is fundamentally conditioned on the representation space used to construct the transport cost. We argue that reliable OT pseudo-labeling requires a representation geometry that is simultaneously sensitive to discriminative action changes and coherent along local temporal progressions. Based on this insight, we propose SpecT-OT, a spectral-temporal representation learning framework built upon an unbalanced optimal transport pseudo-labeling concept. SpecT-OT introduces a Spectral Reparameterization Projector (SRP), which parameterizes projector weights with fixed Fourier bases and learnable coefficients to improve the modeling of rapidly varying discriminative features, and Temporal Affinity Regularization (TAR), which imposes distance-aware, label-free constraints on pairwise frame affinities to stabilize local temporal structure. The two components jointly produce more discriminative and temporally stable transport costs, yielding more reliable pseudo-labels for iterative representation learning. Experiments on four benchmarks demonstrate strong performance compared with state-of-the-art methods. SpecT-OT achieves the best results on 13 of 15 metrics, including 4.1-point MoF and 7.4-point F1 gains over the baseline on Breakfast and Desktop Assembly, respectively.

Problem

• unsupervised action segmentation
• latent action categories
• temporal organization
• pseudo-labeling

Innovation

• Spectral Reparameterization Projector
• Temporal Affinity Regularization
• Unsupervised Action Segmentation
• Optimal Transport
• Representation Learning