Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors

2026-06-30

AI Summary

This work addresses the limited logical interpretability of conventional Transformer feedforward layers, which struggle to model syntactic licensing and quantifier reasoning. The authors propose a novel feedforward architecture restructured as an explicit fuzzy logic operator unit, incorporating a sequential quantifier module with a self-forgetting mechanism. This module supports soft existential and proportional quantifiers with learnable forgetting rates, grounded in fundamental fuzzy operations—intersection (A·B) and set difference (A·(1−B)). For the first time under parameter neutrality, this design yields a feedforward layer with explicit logical form and syntactic interpretability. Evaluated on a 125M-parameter model, it achieves perplexity comparable to GELU-based baselines, slightly outperforms them on the LAMBADA task, and enables direct interpretation of feedforward units as syntactic licensing detectors.

Abstract

A transformer's feed-forward (FFN) sublayer materializes the distinctions attention gathers, yet gives no account of what it computes. In a parameter-neutral replacement, each hidden unit is an explicit fuzzy set operation on sigmoid-bounded [0,1] memberships: intersection A*B and set-difference A*(1-B), the latter a bounded positive negation ("A but not B") that gated/bilinear units lack -- a negation-capable FFN (NC-FFN). On N-bit parity they are the most parameter-efficient reasoning basis at shallow depth; at scale (125M, OpenWebText) NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form. Two limits share one cause: two-operand logic localizes to layer 0 and erodes under training, and the one robust grammatical deficit concentrates in licensing and quantifiers, beyond within-token operators. We resolve both with a small block of sequence quantifiers: a soft existential and a soft proportion, each with a per-unit learned forgetting rate from a sticky init. This recovers the deficit at epoch one (halving the wider epoch-two gap), modestly leads on LAMBADA, and makes the FFN legible: the structure now holds and migrates into depth; the decay un-learns its stickiness (median half-life ~1.5 tokens; zero latch units); and at the semantic layers the units read, without dictionary learning, as grammatical licensing detectors: each fires on a licensor (a comparative, a passive participle, a negative-polarity item) and carries its memory forward to predict the licensed word (than, by, nor). This legibility is localized and free only up to a partition (a fully Boolean FFN diverges in training), but the result is a parameter-neutral, language-model-quality transformer with a readable, interpretable-by-construction grammatical mechanism -- an account not just of what a feed-forward layer represents but how it licenses.

Problem

• feed-forward network
• grammatical licensing
• fuzzy logic
• interpretability
• quantifiers

Innovation

• Negation-Capable FFN
• Fuzzy Logic
• Grammatical Licensing
• Self-Forgetting Quantifiers
• Interpretable Transformers