LACE: Lattice Attention for Cross-thread Exploration

2026-04-16

AI Summary

This work addresses the limitation of existing large language models in parallel reasoning, where independent inference paths often lead to redundant errors. To overcome this, the authors propose LACE, a novel framework that enables real-time interaction and collaborative error correction among multiple reasoning threads for the first time. LACE introduces a lattice attention mechanism and a cross-thread attention architecture, allowing paths to dynamically share intermediate insights during inference. Additionally, the framework incorporates a synthetic data generation pipeline specifically designed to train models for effective collaboration. Experimental results demonstrate that LACE improves reasoning accuracy by over 7 percentage points compared to standard parallel search methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Current large language models reason in isolation. Although it is common to sample multiple reasoning paths in parallel, these trajectories do not interact, and often fail in the same redundant ways. We introduce LACE, a framework that transforms reasoning from a collection of independent trials into a coordinated, parallel process. By repurposing the model architecture to enable cross-thread attention, LACE allows concurrent reasoning paths to share intermediate insights and correct one another during inference. A central challenge is the absence of natural training data that exhibits such collaborative behavior. We address this gap with a synthetic data pipeline that explicitly teaches models to communicate and error-correct across threads. Experiments show that this unified exploration substantially outperforms standard parallel search, improving reasoning accuracy by over 7 points. Our results suggest that large language models can be more effective when parallel reasoning paths are allowed to interact.

Problem

• parallel reasoning
• cross-thread interaction
• reasoning collaboration
• redundant failure
• independent inference

Innovation

• cross-thread attention
• parallel reasoning
• collaborative inference
• synthetic training data
• lattice attention