Learning Probabilistic Embeddings for Unsupervised Action Segmentation

2026-07-06

AI Summary

This work addresses the issue of deterministic embeddings prone to local optima in unsupervised temporal action segmentation by introducing, for the first time, a probabilistic embedding framework. Each frame representation is modeled as a Gaussian distribution, and stochastic sampling is performed prior to pseudo-label estimation. By integrating optimal transport theory, the method alternately optimizes probabilistic embeddings and cluster assignments, enabling joint optimization of representation learning and clustering. Evaluated on multiple standard benchmarks, the proposed approach substantially outperforms existing methods, achieving improvements of up to 20.7% in accuracy and 19.0% in F1-score over the MoF baseline, thereby effectively mitigating the local optima problem inherent in iterative optimization.

Abstract

This paper concerns the problem of unsupervised temporal action segmentation for long, untrimmed videos. Recent successful approaches follow a joint representation learning and clustering paradigm, where optimal transport (OT) is adopted to produce pseudo labels for learning frame representations. These approaches alternate between estimating pseudo labels using OT and optimizing the parameters with gradient descent during training, where OT is used for obtaining the final temporal action segmentation. A major limitation of these works is that they learn a deterministic embedding for frame representations. The iterative procedure between learning deterministic embeddings based on pseudo labels and estimating pseudo labels from the learned embedding can thus get quickly stuck in a local optimum. As an alternative, we thus propose to learn a probabilistic embedding for frame representations. The embeddings are modeled by Gaussian distributions and we sample from the distributions before estimating the pseudo labels. We evaluate our approach on several challenging temporal action segmentation datasets and achieve results comparable to, and in some cases, better than the state of the art. Compared to baselines with deterministic embeddings, our approach improves MoF up to 20.7\% and F1-score up to 19.0\%. Our code is available at https://github.com/derkbreeze/PEOT.

Problem

• unsupervised action segmentation
• probabilistic embeddings
• temporal action segmentation
• deterministic embedding
• local optimum

Innovation

• probabilistic embedding
• unsupervised action segmentation
• optimal transport
• Gaussian distribution
• temporal action segmentation