Banyan: Improved Representation Learning with Explicit Structure

2024-07-25

arXiv.org

AI Summary

To address the low learning efficiency and poor interpretability of pretrained models for low-resource languages, this paper proposes an explicit hierarchical semantic representation framework. Methodologically, it introduces (1) an entangled hierarchical tree structure that sparsely and efficiently models cross-granularity semantic relations with minimal overhead, and (2) a diagonalized graph neural message-passing mechanism that achieves effective hierarchical information aggregation using only 14 non-embedding parameters. Empirically, the approach significantly outperforms existing structured models on under-resourced languages and computationally constrained settings, matching the performance of large-scale Transformers while offering strong interpretability and ultra-low parameter cost (e.g., negligible parameter overhead beyond embeddings). This work establishes a novel paradigm for semantic modeling in resource-constrained environments, balancing expressiveness, efficiency, and transparency.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Structured Learning
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.

Problem

• Language Models
• Limited Resources
• Rare Languages

Innovation

• Banyan Model
• Tree-like Structure
• Efficient Information Propagation