Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars

2026-06-15

AI Summary

This work addresses the common practice in syntactic Transformer models of linearizing parse trees via depth-first traversal (DFT), which overlooks the impact of traversal order on model performance. The study systematically investigates breadth-first traversal (BFT) and introduces a novel production-rule traversal (PRT), integrating diverse tree representations, traversal strategies, and attention masking mechanisms within a Transformer-based syntactic architecture. PRT combines BFT’s global foresight with DFT’s advantage in early lexical prediction, revealing a fundamental trade-off between nested compositional structure and access to global syntactic information. Empirical results across language modeling, syntactic generalization, and summarization tasks demonstrate that traversal strategies yield markedly different performance profiles, offering both empirical grounding and practical guidance for designing task-specific, syntax-aware Transformer models.

Abstract

Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic trees are linearized in TGs, existing studies rely solely on Depth-First Traversal (DFT) for linearization. In this paper, we expand the traversal design space by exploring Breadth-First Traversal (BFT) and a novel hybrid traversal strategy, Production-Rule Traversal (PRT), which combines the structural lookahead of BFT with the early lexical generation of DFT. We integrate these traversal methods with varying tree configurations and masking strategies, and empirically evaluate their performance on language modeling, syntactic generalization and summarization. We reveal the inherent trade-offs between nested composition and global lookahead, providing actionable recommendations for designing task-aware Transformer Grammars.

Problem

• Tree Traversal
• Transformer Grammars
• Linearization
• Syntactic Structure
• Language Modeling

Innovation

• Tree Traversal
• Transformer Grammars
• Production-Rule Traversal
• Syntactic Generalization
• Linearization Strategy