Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

2026-06-18

AI Summary

This work addresses the lack of interpretability in pretrained language models in high-stakes scenarios and the inability of conventional Tsetlin Machines to effectively capture semantic information. To bridge this gap, the authors propose a novel approach that replaces static word embeddings with semantic clusters derived from pretrained language models—generated via K-means or Top2Vec clustering—to pretrain a non-negated Tsetlin Machine. This is further enhanced by an augmented Type I feedback mechanism to facilitate knowledge transfer. The method represents the first integration of semantic clustering into the Tsetlin Machine framework. Evaluated across five datasets, it significantly outperforms both traditional and embedding-based Tsetlin Machine variants, achieving performance comparable to BERT while preserving rule-level interpretability.

Abstract

Pre-trained language models such as BERT achieve strong text classification performance but lack transparency, limiting their use in high-stakes settings. The Tsetlin Machine (TM) offers fully interpretable, clause-based reasoning but captures little semantic information, and prior attempts to bridge the two rely on static word embeddings that miss contextual meaning. We propose a semantic pre-training framework that transfers knowledge from a pre-trained language model into a TM without using embeddings. Text samples are grouped into semantically coherent clusters with K-means or Top2Vec, and the resulting cluster-sample pairs pre-train a non-negated TM with enhanced Type I feedback. The TM thereby learns interpretable semantic keywords that are fine-tuned on downstream tasks. Across five datasets, our method substantially outperforms vanilla and embedding-based TMs and reaches performance competitive with BERT while remaining interpretable.

Problem

• interpretability
• semantic representation
• Tsetlin Machine
• pre-trained language models
• text classification

Innovation

• Tsetlin Machine
• semantic clustering
• interpretable AI
• pre-training
• language model distillation