A Systematic Study of Compositional Syntactic Transformer Language Models

2025-06-28

AI Summary

This work addresses the insufficient compositional modeling in syntax-augmented language models. We propose a unified framework grounded in constituent parse trees, jointly encoding linearized parse trees and surface-level token sequences into a Transformer architecture, while incorporating an explicit bottom-up composition mechanism. Through systematic ablation studies and multi-task evaluation—including language modeling, syntactic generalization, summarization, dialogue understanding, and reasoning efficiency—we identify three critical design factors: tree encoding methodology, compositional granularity, and architectural integration strategy. Compared to existing compositional syntax-aware language models (SLMs), our framework significantly improves syntactic sensitivity and cross-task generalization without compromising computational efficiency. Moreover, it provides empirically grounded guidance and reproducible design principles for structural optimization of syntax-enhanced models.

Technology Category

• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This paper focuses on compositional SLMs that are based on constituency parse trees and contain explicit bottom-up composition of constituent representations. We identify key aspects of design choices in existing compositional SLMs and propose a unified framework encompassing both existing models and novel variants. We conduct a comprehensive empirical evaluation of all the variants in our framework across language modeling, syntactic generalization, summarization, dialogue, and inference efficiency. Based on the experimental results, we make multiple recommendations on the design of compositional SLMs. Our code is released at https://github.com/zhaoyd1/compositional_SLMs.

Problem

• Enhancing Transformers with syntactic biases via parse trees
• Developing a unified framework for compositional syntactic models
• Evaluating syntactic models across multiple NLP tasks

Innovation

• Incorporates syntactic biases via parse trees
• Unifies compositional SLMs with novel variants
• Evaluates models across multiple NLP tasks