The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

2026-01-09

arXiv.org

AI Summary

This study systematically investigates whether and how Transformer-based language models acquire syntactic knowledge. Through a large-scale, systematic literature review synthesizing findings from 337 studies and over 3,000 data points, the work presents the first integrated quantitative assessment of syntactic capabilities across multiple languages and model architectures by combining behavioral experiments, representation probing, and mechanistic interpretability methods. The analysis reveals that Transformers possess substantial syntactic knowledge, yet exhibit limitations in phenomena at the syntax–semantics interface and in low-resource languages. It also highlights a pronounced research bias toward English and BERT-family models, with insufficient coverage of linguistic and architectural diversity. This work provides comprehensive empirical evidence and new directions for understanding the mechanisms and boundaries of syntactic generalization in neural language models.

Abstract

We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models, reporting on 1,015 model results from a range of syntactic phenomena and interpretability methods. Our analysis shows that the state of the art presents a healthy variety of methods and data, but an over-focus on a single language (English), a single model (BERT), and phenomena that are easy to get at (like part of speech and agreement). Results also suggest that TLMs capture these form-oriented phenomena well, but show more variable and weaker performance on phenomena at the syntax-semantics interface, like binding or filler-gap dependencies. We provide recommendations for future work, in particular reporting complete data, better aligning theoretical constructs and methods across studies, increasing the use of mechanistic methods, and broadening the empirical scope regarding languages and linguistic phenomena.

Problem

• syntactic knowledge
• Transformer language models
• interpretability
• syntax-semantics interface
• cross-lingual

Innovation

• systematic review
• syntactic knowledge
• Transformer-based language models
• interpretability
• cross-linguistic analysis