Document Topic Alignment Metrics for Evaluating Topic Models of Short-Text Public Health Communications on Social Media

2026-09-12

AI Summary

为解决短文本主题模型评估问题，提出文档-主题对齐度量(DoTA)框架，衡量文档与分配主题间的语义一致性。

Abstract

Topic models are widely used to analyze public health-related social media short texts, yet their evaluation remains dominated by metrics that focus entirely on generated topics alone. There is a lack of metrics that quantitatively assess whether assigned topics meaningfully represent the corresponding short-text posts. We propose Document-Topic Alignment metrics (DoTA), an assignment-aware evaluation framework comprising metrics that measure semantic alignment between documents (posts) and their assigned topics. We also introduce margin-based and discriminative variants that capture topic assignment confidence and distinguishability. We evaluate DoTA across five topic models on three public health-related social media datasets from X and compare DoTA metrics with conventional topic-based metrics. Results show that DoTA provides complementary evaluation cues and aligns meaningfully with human evaluations. These findings establish the need for assignment-aware evaluation and demonstrate that the addition of DoTA enables a more comprehensive and practically meaningful evaluation for assessing short-text topic modeling performance.

Problem

• Document-Topic Alignment
• short-text topic modeling
• public health communications
• social media

Innovation

• Document-Topic Alignment
• assignment-aware evaluation
• semantic alignment
• short-text topic modeling