Research Community Perspectives on"Intelligence"and Large Language Models

2025-05-27

AI Summary

This study addresses the lack of a precise, empirically grounded definition of “intelligence” in NLP. Through a mixed-methods survey involving 303 interdisciplinary researchers—including structured questionnaires, cross-disciplinary sampling, and statistical analysis—it systematically investigates scholarly consensus on intelligence and its role in NLP research practice. The study identifies and empirically validates three core consensus dimensions: generalization capability, adaptability, and reasoning ability. It further reveals that only 29% of respondents attribute intelligence to current large language models (LLMs), and merely 16.2% explicitly pursue intelligent system construction as their primary research goal—this subgroup exhibits significantly higher endorsement of LLM intelligence. By bridging a critical gap in empirical foundational AI research, the work establishes an essential cognitive benchmark for NLP theory development and evaluation paradigms.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Emotional Intelligence

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Despite the widespread use of ''artificial intelligence'' (AI) framing in Natural Language Processing (NLP) research, it is not clear what researchers mean by ''intelligence''. To that end, we present the results of a survey on the notion of ''intelligence'' among researchers and its role in the research agenda. The survey elicited complete responses from 303 researchers from a variety of fields including NLP, Machine Learning (ML), Cognitive Science, Linguistics, and Neuroscience. We identify 3 criteria of intelligence that the community agrees on the most: generalization, adaptability,&reasoning. Our results suggests that the perception of the current NLP systems as ''intelligent'' is a minority position (29%). Furthermore, only 16.2% of the respondents see developing intelligent systems as a research goal, and these respondents are more likely to consider the current systems intelligent.

Problem

• Defining 'intelligence' in AI and NLP research
• Assessing community agreement on intelligence criteria
• Evaluating perception of current NLP systems as 'intelligent'

Innovation

• Surveyed researchers on intelligence criteria
• Identified generalization, adaptability, reasoning
• Minority view current NLP as intelligent