Characterizing Web Search in The Age of Generative AI

2025-10-13

AI Summary

This study investigates fundamental differences between generative search and traditional web search across source provenance, knowledge dependency, and conceptual coverage. To address the lack of evaluation frameworks tailored to generative AI, we propose a three-dimensional assessment framework—measuring source diversity, internal vs. external knowledge utilization ratio, and cross-disciplinary concept coverage—and empirically compare leading generative search engines (e.g., Google AI Overviews, OpenAI’s SearchGPT) with conventional search across multi-domain queries. Results reveal that generative search significantly broadens source scope but exhibits structural divergence across systems in knowledge invocation strategies and conceptual depth. Its intrinsic generative mechanism reduces result traceability and deterministic grounding while enhancing serendipitous conceptual associations and cross-domain integration. This work pioneers a paradigm-specific reconfiguration of search evaluation for generative AI, offering both theoretical foundations and methodological tools for designing and assessing next-generation search systems.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Constraint Satisfaction and Optimization: Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

The advent of LLMs has given rise to a new type of web search: Generative search, where LLMs retrieve web pages related to a query and generate a single, coherent text as a response. This output modality stands in stark contrast to traditional web search, where results are returned as a ranked list of independent web pages. In this paper, we ask: Along what dimensions do generative search outputs differ from traditional web search? We compare Google, a traditional web search engine, with four generative search engines from two providers (Google and OpenAI) across queries from four domains. Our analysis reveals intriguing differences. Most generative search engines cover a wider range of sources compared to web search. Generative search engines vary in the degree to which they rely on internal knowledge contained within the model parameters v.s. external knowledge retrieved from the web. Generative search engines surface varying sets of concepts, creating new opportunities for enhancing search diversity and serendipity. Our results also highlight the need for revisiting evaluation criteria for web search in the age of Generative AI.

Problem

• Comparing generative search engines with traditional web search outputs
• Analyzing source coverage differences between AI and conventional search
• Investigating knowledge sources and diversity in generative search results

Innovation

• Generative search retrieves web pages and generates coherent responses
• Compares traditional and generative search engines across multiple domains
• Analyzes reliance on internal model knowledge versus external web retrieval