Is deeper always better? Replacing linear mappings with deep learning networks in the Discriminative Lexicon Model

2024-10-05

Linguistics Vanguard

AI Summary

This study investigates whether deep learning can surpass linear methods in modeling form–meaning mappings for language cognition. Within the Discriminative Lexical Modeling (DLM) framework, we systematically introduce Deep Dense Neural Networks (DDL) and their frequency-informed variant (FIDDL) as replacements for conventional linear mappings. Results on large English and Dutch datasets show that DDL significantly outperforms Linear Discriminative Learning (LDL), while FIDDL substantially exceeds Frequency-Informed Linear (FIL) models; this advantage stems from superior modeling of pseudo-morphological words and a synergistic interaction between frequency information and deep architecture. However, gains are limited on morphologically complex, low-frequency–dominant languages—Estonian and Taiwanese Southern Min—and DDL exhibits weaker incremental learning capacity than linear models. These findings provide empirical evidence delineating the applicability boundaries of deep architectures in cognitive modeling of linguistic structure.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recently, deep learning models have increasingly been used in cognitive modelling of language. This study asks whether deep learning can help us to better understand the learning problem that needs to be solved by speakers, above and beyond linear methods. We utilize the Discriminative Lexicon Model introduced by Baayen and colleagues, which models comprehension and production with mappings between numeric form and meaning vectors. While so far, these mappings have been linear (linear discriminative learning; LDL), in the present study we replace them with deep dense neural networks (deep discriminative learning; DDL). We find that DDL affords more accurate mappings for large and diverse datasets from English and Dutch, but not necessarily for Estonian and Taiwan Mandarin. DDL outperforms LDL in particular for words with pseudo-morphological structure such as chol + er . Applied to average reaction times, we find that DDL is outperformed by frequency-informed linear mappings (FIL). However, DDL trained in a frequency-informed way (“frequency-informed” deep learning; FIDDL) substantially outperforms FIL. Finally, while linear mappings can very effectively be updated from trial-to-trial to model incremental lexical learning, deep mappings cannot do so as effectively. At present, both linear and deep mappings are informative for understanding language.

Problem

• Evaluating deep learning vs linear mappings in cognitive language models
• Assessing deep networks' effectiveness across diverse linguistic datasets
• Comparing incremental learning capabilities between linear and deep mappings

Innovation

• Replaced linear mappings with deep dense neural networks
• Applied frequency-informed training to deep learning models
• Enhanced accuracy for morphologically complex word structures