RELIANCE: Curating and Evaluating Reproductive Health Information on Social Media

2026-06-10

AI Summary

This study addresses growing concerns about the accuracy of reproductive health information on social media and the risks of deploying large language models (LLMs) for fact-checking without domain-specific evaluation. It presents the first high-quality dataset combining clinical expert annotations with real-world TikTok content on prenatal and postpartum health, alongside a novel LLM evaluation framework tailored to specific health domains. Through integrated expert review, natural language processing, and model assessment, the research finds that approximately 60% of the videos contain accurate information and reveals a 15% performance gap between LLM judgments at the overall content level versus specific factual claims. The authors publicly release their dataset and code to support scalable, trustworthy research on health information credibility.

Abstract

Social media platforms like TikTok have become a key source of health information, with studies reporting inaccuracies in posts. As Large Language Model (LLM) providers increasingly integrate LLMs into digital platforms to fact-check content (e.g., Grok and Perplexity on X and WhatsApp, respectively) and are being used by people to fact-check information, deploying these systems in critical areas such as reproductive health without rigorous evaluation can cause serious harm. We introduce RELIANCE, an expert-annotated dataset of health information on TikTok surrounding pregnancy and postpartum queries, serving as both an analysis of the reproductive health information landscape and an evaluation of LLMs' capabilities in fact-checking this content. Our dataset comprises 409 annotated sentences from 336 videos across 56 clinician-reviewed queries, annotated by three expert clinicians in Obstetrics, Gynecology, and Internal Medicine. Our findings reveal that nearly 60\% of the health information in the videos we sampled is accurate. Furthermore, LLM evaluations reveal a gap between evaluating specific claims and evaluating the entire content (15\%). We believe that our methodology, dataset, and tool will support the machine learning community in improving LLMs for important domains with real-world data, extending to other platforms and languages, and helping the health community further understand the information landscape on social media. Our dataset and code are made available at https://realize-lab.github.io/RELIANCE/.

Problem

• reproductive health
• social media
• misinformation
• fact-checking
• large language models

Innovation

• expert-annotated dataset
• reproductive health misinformation
• large language model evaluation
• fact-checking on social media
• clinical validation