Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agents

2025-09-09

AI Summary

Existing LLM evaluation paradigms neglect interactive language acquisition—the core human capacity to autonomously acquire new languages through pattern recognition and real-time feedback. Method: We propose the first benchmark framework for interactive language learning, built upon real-time conversational feedback. We design Tinkatongue, a novel artificial language, and an embodied robotic interaction environment to enable controlled, grounded language acquisition experiments. Contribution/Results: While LLM agents fail to achieve functional dialogue within 100 interaction rounds, they exhibit human-like learning behaviors—including tentative induction, hypothesis testing, and adaptive strategy revision—demonstrating emergent inductive reasoning under interaction constraints. This work exposes fundamental limitations of current LLMs in embodied, feedback-driven language acquisition and establishes the first process-oriented evaluation paradigm—shifting focus from static linguistic competence to dynamic learning trajectories. It provides both theoretical foundations and empirical infrastructure for developing next-generation embodied language learning models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Intelligent Robots: Embodied AI
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Existing evaluation studies on linguistic competence of large language models (LLM agents) have focused primarily on vocabulary learning, morphological rule induction, syntactic generalization, pragmatic inference, and cross-linguistic transfer. However, none assess whether LLM agents can acquire a language through pattern recognition and interactive feedback, a central feature of human language acquisition. We propose a novel experimental framework in which an LLM agent is evaluated on its ability to acquire and use a newly constructed language (Tinkatongue) in conversation with a bot that understands only Tinkatongue. Our findings show that LLM agents fail to establish a conversation within 100 responses, yet they adopt distinct strategies that mirror human approaches to language learning. The results suggest a new direction for evaluation benchmarks and open pathways to model designs that learn more effectively from interactive feedback.

Problem

• Evaluating LLM agents' language acquisition through interactive feedback
• Assessing pattern recognition in linguistic competence of language models
• Testing conversational ability in a novel constructed language

Innovation

• Novel framework for LLM language acquisition evaluation
• Tests pattern recognition and interactive feedback abilities
• Uses constructed Tinkatongue language for conversational assessment