Omni TM-AE: A Scalable and Interpretable Embedding Model Using the Full Tsetlin Machine State Space

2025-05-22

AI Summary

The black-box nature of large language models hinders their interpretability and reusability. Traditional embedding models (e.g., Word2Vec, GloVe) are scalable but inherently uninterpretable; Tsetlin Machines (TMs) offer strong interpretability but have historically suffered from poor scalability and limited reusability. To bridge this gap, we propose Omni TM-AE—the first scalable, interpretable embedding model built upon a full-state-space Tsetlin Machine. It systematically leverages *all* literals in the TM state matrix—including those conventionally ignored—via Boolean logic encoding and gradient-free discrete optimization, achieving linear scalability, end-to-end interpretability, and embedding reusability within a single-stage training paradigm. Empirically, Omni TM-AE matches or surpasses Word2Vec, GloVe, and leading neural embedding models on semantic similarity, sentiment classification, and document clustering tasks. This work provides the first empirical validation that high performance, full interpretability, and linear scalability can be simultaneously achieved in embedding learning.

Technology Category

• Machine Learning: Transparent, Interpretable, Explainable ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

The increasing complexity of large-scale language models has amplified concerns regarding their interpretability and reusability. While traditional embedding models like Word2Vec and GloVe offer scalability, they lack transparency and often behave as black boxes. Conversely, interpretable models such as the Tsetlin Machine (TM) have shown promise in constructing explainable learning systems, though they previously faced limitations in scalability and reusability. In this paper, we introduce Omni Tsetlin Machine AutoEncoder (Omni TM-AE), a novel embedding model that fully exploits the information contained in the TM's state matrix, including literals previously excluded from clause formation. This method enables the construction of reusable, interpretable embeddings through a single training phase. Extensive experiments across semantic similarity, sentiment classification, and document clustering tasks show that Omni TM-AE performs competitively with and often surpasses mainstream embedding models. These results demonstrate that it is possible to balance performance, scalability, and interpretability in modern Natural Language Processing (NLP) systems without resorting to opaque architectures.

Problem

• Balancing performance and interpretability in NLP models
• Overcoming scalability limits of Tsetlin Machine embeddings
• Creating reusable embeddings via single-phase TM training

Innovation

• Utilizes full Tsetlin Machine state matrix
• Single training for reusable embeddings
• Balances performance, scalability, interpretability