Can AI mimic the human ability to define neologisms?

2025-02-18

AI Summary

This study investigates the alignment between large language models (LLMs)—exemplified by ChatGPT—and human-defined semantic interpretations of Greek neologisms, including blends, compounds, and derivations. Method: Human definitions were collected via online behavioral experiments; inter-annotator agreement between human and model outputs was quantified using Krippendorff’s alpha and majority-voting consensus. Contribution/Results: We report the first systematic finding that LLMs achieve only moderate agreement with humans on blends and derivations (α ≈ 0.4–0.5), but near-zero agreement on compounds (α < 0.1). When human majority responses serve as the gold standard, accuracy on blends and derivations significantly exceeds that on compounds. These results expose a fundamental limitation of current LLMs in cross-constituent semantic integration—particularly for compound structures—and propose an interpretable enhancement pathway integrating semantic networks with contextual learning. The work establishes a novel, morphology-driven paradigm for evaluating and refining language models.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

One ongoing debate in linguistics is whether Artificial Intelligence (AI) can effectively mimic human performance in language-related tasks. While much research has focused on various linguistic abilities of AI, little attention has been given to how it defines neologisms formed through different word formation processes. This study addresses this gap by examining the degree of agreement between human and AI-generated responses in defining three types of Greek neologisms: blends, compounds, and derivatives. The study employed an online experiment in which human participants selected the most appropriate definitions for neologisms, while ChatGPT received identical prompts. The results revealed fair agreement between human and AI responses for blends and derivatives but no agreement for compounds. However, when considering the majority response among humans, agreement with AI was high for blends and derivatives. These findings highlight the complexity of human language and the challenges AI still faces in capturing its nuances. In particular, they suggest a need for integrating more advanced semantic networks and contextual learning mechanisms into AI models to improve their interpretation of complex word formations, especially compounds.

Problem

• AI mimics human neologism definition
• Agreement between human and AI responses
• Challenges in AI interpreting complex word formations

Innovation

• AI defines Greek neologisms
• ChatGPT prompts for definitions
• Semantic networks improve AI