Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy

2025-11-16

AI Summary

This study audits the information quality and consistency of Google’s AI Overview (AIO) and Featured Snippets (FS) for infant care and pregnancy-related queries—a high-stakes domain requiring clinical rigor. We construct a 1,508-query evaluation dataset derived from real user searches, integrating automated web crawling, expert annotation, and NLP-based analysis. Assessment spans five dimensions: factual consistency, relevance, medical safety warnings, source credibility, and affective alignment. Key findings reveal: (1) factual contradictions between AIO and FS in 33% of queries; (2) alarmingly low medical safety cue rates—11% for AIO and 7% for FS; and (3) high relevance scores lacking grounding in clinical consensus or evidence-based guidelines. To address these gaps, we propose a transferable algorithmic auditing framework that supports scalable, multi-dimensional evaluation of AI-generated health information. This work provides both methodological foundations and empirical evidence to inform regulatory policy and quality assurance for AI-assisted health communication.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Search and Optimization: Evaluation and Analysis
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Google Search increasingly surfaces AI-generated content through features like AI Overviews (AIO) and Featured Snippets (FS), which users frequently rely on despite having no control over their presentation. Through a systematic algorithm audit of 1,508 real baby care and pregnancy-related queries, we evaluate the quality and consistency of these information displays. Our robust evaluation framework assesses multiple quality dimensions, including answer consistency, relevance, presence of medical safeguards, source categories, and sentiment alignment. Our results reveal concerning gaps in information consistency, with information in AIO and FS displayed on the same search result page being inconsistent with each other in 33% of cases. Despite high relevance scores, both features critically lack medical safeguards (present in just 11% of AIO and 7% of FS responses). While health and wellness websites dominate source categories for both, AIO and FS, FS also often link to commercial sources. These findings have important implications for public health information access and demonstrate the need for stronger quality controls in AI-mediated health information. Our methodology provides a transferable framework for auditing AI systems across high-stakes domains where information quality directly impacts user well-being.

Problem

• Evaluating quality and consistency of Google's AI-generated health information displays
• Assessing medical safeguards and source reliability in AI Overviews and Featured Snippets
• Developing audit framework for AI systems in high-stakes health domains

Innovation

• Systematic algorithm audit of 1508 health queries
• Robust framework evaluating multiple quality dimensions
• Transferable methodology for auditing AI systems