Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences

2024-10-27

International Conference on Learning Representations

AI Summary

Human sequence learning relies on abstract pattern extraction and cross-contextual generalization, yet existing models—including large language models (LLMs)—exhibit limited generalization and poor memory efficiency. To address this, we propose the nonparametric Hierarchical Variable Model (HVM), the first framework enabling context-driven, tunable-level variable abstraction: it automatically clusters similar sequence segments into reusable variables, facilitating efficient memory organization and cross-sequence generalization. HVM integrates sequence chunking, nonparametric hierarchical learning, and neural-behavioral alignment evaluation—specifically calibrated to human recall latency. Experiments show that HVM achieves higher dictionary compression than LZ on babyLM; its sequence likelihood strongly correlates with human recall times; and it significantly outperforms state-of-the-art LLMs on abstract variable transfer tasks. This work establishes a novel, cognition-inspired paradigm for sequence abstraction modeling.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Humans excel at learning abstract patterns across different sequences, filtering out irrelevant details, and transferring these generalized concepts to new sequences. In contrast, many sequence learning models lack the ability to abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables. HVM efficiently organizes memory while uncovering abstractions, leading to compact sequence representations. When learning on language datasets such as babyLM, HVM learns a more efficient dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence recall task requiring the acquisition and transfer of variables embedded in sequences, we demonstrate HVM's sequence likelihood correlates with human recall times. In contrast, large language models (LLMs) struggle to transfer abstract variables as effectively as humans. From HVM's adjustable layer of abstraction, we demonstrate that the model realizes a precise trade-off between compression and generalization. Our work offers a cognitive model that captures the learning and transfer of abstract representations in human cognition and differentiates itself from the behavior of large language models.

Problem

• Learning abstract patterns from sequences efficiently
• Transferring generalized concepts to new sequences effectively
• Balancing compression and generalization in sequence models

Innovation

• Non-parametric hierarchical variable learning model
• Abstracts contextually similar chunks as variables
• Adjustable layer balancing compression and generalization