Beyond Text: Characterizing Domain Expert Needs in Document Research

2025-04-16

AI Summary

This study investigates whether current plain-text NLP systems adequately support expert-level document research practices. Method: We conducted semi-structured, in-depth interviews with 16 domain-expert researchers across disciplines, followed by thematic coding and cross-domain comparative analysis. Contribution/Results: We empirically establish— for the first time—a “document-centric” (rather than text-centric) design paradigm, identifying four core capabilities essential for expert workflows: accessibility, customizability, iterativity, and social awareness. Our findings reveal a systemic gap in how NLP models document work practices, particularly neglecting documents as socio-structural artifacts and under-supporting iterative, context-sensitive, and socially embedded exploration. Building on this, we propose the first user-centered evaluation framework and design principles for document intelligence, grounded in empirical evidence. These contributions provide both theoretical foundations and practical guidelines for developing next-generation document intelligence systems.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Intelligent User Interfaces

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Working with documents is a key part of almost any knowledge work, from contextualizing research in a literature review to reviewing legal precedent. Recently, as their capabilities have expanded, primarily text-based NLP systems have often been billed as able to assist or even automate this kind of work. But to what extent are these systems able to model these tasks as experts conceptualize and perform them now? In this study, we interview sixteen domain experts across two domains to understand their processes of document research, and compare it to the current state of NLP systems. We find that our participants processes are idiosyncratic, iterative, and rely extensively on the social context of a document in addition its content; existing approaches in NLP and adjacent fields that explicitly center the document as an object, rather than as merely a container for text, tend to better reflect our participants' priorities, though they are often less accessible outside their research communities. We call on the NLP community to more carefully consider the role of the document in building useful tools that are accessible, personalizable, iterative, and socially aware.

Problem

• Understanding expert document research processes vs NLP capabilities
• Analyzing gaps in NLP systems for social-context-aware document use
• Advocating accessible, iterative tools centered on documents not just text

Innovation

• Interviews domain experts for document research insights
• Compares expert processes with current NLP systems
• Advocates document-centric, socially-aware NLP tools