AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results in a Domain-specific Search Engine

2026-07-03

AI Summary

This study addresses the high cognitive load associated with relevance assessment in academic search. Focusing on social science domains, it introduces AI-generated search engine results page (SERP)-level summaries within an interface designed according to information foraging theory and analyzes user behavior. The work proposes a six-category error taxonomy tailored to academic contexts and outlines five deployment safeguards, leveraging both commercial and open-source large language models to generate summaries. User experiments reveal that while AI-generated summaries do not significantly improve primary effectiveness metrics, they consistently reduce perceived mental workload and frustration, decrease click-through and query reformulation rates, and support early-stage screening through enhanced informational cues. These findings highlight their potential as context-sensitive assistive tools in scholarly search environments.

Abstract

Evaluating search engine results pages (SERPs) to assess result relevance is a demanding step in academic search. In a formative mixed-methods design study, we examine AI-generated SERP-level summaries as a support feature in an academic search engine for social science information. First, we manually evaluated summaries of the top five results for 10 queries using two general-purpose models, one commercial and one open, deriving an exploratory six-category error taxonomy and five safeguards for scholarly deployment. We then conducted a within-subjects user study (n = 30) comparing interfaces with and without AI summaries. Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand, with frustration also tending to be lower. Behaviorally, participants rarely expanded the summaries and descriptively made slightly fewer result clicks and query reformulations when summaries were available. Drawing on Information Foraging Theory and participant feedback, we suggest that AI summaries may concentrate SERP-level information scent to support early triage. Overall, the findings indicate that SERP-level AI summaries are a context- and user-dependent aid rather than a universal improvement, while contributing an error taxonomy, safeguard-aware deployment guidance, and concrete design implications for scholarly search.

Problem

• AI Overviews
• Academic Search
• Search Engine Results Pages
• AI-generated Summaries
• Information Foraging

Innovation

• AI-generated summaries
• search result evaluation
• error taxonomy
• information foraging theory
• scholarly search