Selected work

Representative Papers

Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention

Aug 11, 2026

This work establishes fundamental limitations of non-negative kernel attention mechanisms, showing that their feature dimension must grow exponentially even to handle contexts as short as three tokens. Focusing on the exact modeling of Boolean inputs under the Min-IP task, the authors employ rank analysis, information-theoretic lower bounds, and constructive counterexamples within a framework of causal queries and position-dependent mappings. Their key contribution is the first proof that standard Softmax attention solves this task with only linear feature dimensionality, whereas non-negative kernel attention heads require at least $2^{\Omega(m)}$ dimensions. Furthermore, they derive an information transmission lower bound for multi-head, multi-layer models operating over finite alphabets.

0 citationsRead paper

Indexing: the Beginning and the End

Jul 24, 2026

This work investigates the information bottleneck inherent in modern deep learning architectures when performing indexing tasks, with a particular focus on performance disparities between indexing at the beginning versus the end of a sequence. By introducing the notion of causal complexity, the authors theoretically demonstrate that low-parameter RNNs, state space models (SSMs), and masked linear-attention Transformers cannot solve the tail-indexing task within a constant number of layers, whereas a Softmax-based Transformer can accomplish it in a single layer. Combining theoretical analysis under infinite precision with empirical validation for sequence lengths up to \( n \leq 64 \), the study shows that architectures with low causal complexity learn indexing tasks efficiently, while those with higher causal complexity degrade significantly as sequence length increases.

0 citationsRead paper

Polynomial-Time Mistake-Bounded Language Generation

Jun 14, 2026

This study addresses the problem of achieving error-bounded language generation in polynomial time, with a focus on specific classes of Boolean functions. The work proposes a polynomial-time framework grounded in combinatorial game theory and establishes, for the first time, that all monotone Boolean functions with a polynomial number of maximal terms—encompassing every monotone Boolean function computable by a decision tree of polynomial size—admit error-bounded language generation within this framework. Furthermore, the paper demonstrates the polynomial-time learnability under bounded error of variable parity functions, conjunctions of literals, and a broad class of monotone Boolean functions, thereby significantly extending the known boundaries for both generation and learning of Boolean functions under constrained error conditions.

0 citationsRead paper

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

May 29, 2026

This study investigates how Transformers leverage attention mechanisms to generalize on structured multi-hop reasoning tasks, with a particular focus on their divergent performance in long-sequence extrapolation. By training GPT-J models on equivalent numeric and alphabetic tasks and employing controlled experiments, attention head behavior classification, and geometric analysis of Rotary Position Embeddings (RoPE), the work provides the first clear distinction and theoretical characterization of positional versus symbolic attention heads in terms of their computational roles. The authors introduce a “discrepancy” metric that quantitatively demonstrates the superior extrapolation robustness of symbolic mechanisms over positional ones. Furthermore, they establish that the presence of purely typed attention heads is critical for successful learning—a finding consistently validated across both controlled setups and real-world models.

0 citationsRead paper

Language Generation: Complexity Barriers and Implications for Learning

Nov 07, 2025

This paper investigates the practical feasibility of language generation in terms of sample efficiency. It establishes that, for classical language classes—including regular and context-free languages—successful generation may require a number of positive examples exceeding the bound of any computable function, rendering sample complexity uncomputable—even though these classes are theoretically learnable. Method: Integrating formal language theory, the PAC learning framework, and the Kleinberg–Mullainathan generative model, the paper rigorously derives and proves strong information-theoretic lower bounds on sample complexity. Contribution/Results: The work provides the first systematic characterization, from a computational complexity perspective, of fundamental sample barriers inherent to language generation. Crucially, it demonstrates that the empirical success of modern large language models cannot be fully explained by classical learnability theory alone; instead, it must rely on structural constraints unique to natural language. This insight offers a novel conceptual bridge between theoretical guarantees and practical performance.

0 citationsRead paper

Recent publications

Latest Papers

Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention

Aug 11, 2026

This work establishes fundamental limitations of non-negative kernel attention mechanisms, showing that their feature dimension must grow exponentially even to handle contexts as short as three tokens. Focusing on the exact modeling of Boolean inputs under the Min-IP task, the authors employ rank analysis, information-theoretic lower bounds, and constructive counterexamples within a framework of causal queries and position-dependent mappings. Their key contribution is the first proof that standard Softmax attention solves this task with only linear feature dimensionality, whereas non-negative kernel attention heads require at least $2^{\Omega(m)}$ dimensions. Furthermore, they derive an information transmission lower bound for multi-head, multi-layer models operating over finite alphabets.

0 citationsRead paper

Indexing: the Beginning and the End

Jul 24, 2026

This work investigates the information bottleneck inherent in modern deep learning architectures when performing indexing tasks, with a particular focus on performance disparities between indexing at the beginning versus the end of a sequence. By introducing the notion of causal complexity, the authors theoretically demonstrate that low-parameter RNNs, state space models (SSMs), and masked linear-attention Transformers cannot solve the tail-indexing task within a constant number of layers, whereas a Softmax-based Transformer can accomplish it in a single layer. Combining theoretical analysis under infinite precision with empirical validation for sequence lengths up to \( n \leq 64 \), the study shows that architectures with low causal complexity learn indexing tasks efficiently, while those with higher causal complexity degrade significantly as sequence length increases.

0 citationsRead paper

Polynomial-Time Mistake-Bounded Language Generation

Jun 14, 2026

This study addresses the problem of achieving error-bounded language generation in polynomial time, with a focus on specific classes of Boolean functions. The work proposes a polynomial-time framework grounded in combinatorial game theory and establishes, for the first time, that all monotone Boolean functions with a polynomial number of maximal terms—encompassing every monotone Boolean function computable by a decision tree of polynomial size—admit error-bounded language generation within this framework. Furthermore, the paper demonstrates the polynomial-time learnability under bounded error of variable parity functions, conjunctions of literals, and a broad class of monotone Boolean functions, thereby significantly extending the known boundaries for both generation and learning of Boolean functions under constrained error conditions.

0 citationsRead paper

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

May 29, 2026

This study investigates how Transformers leverage attention mechanisms to generalize on structured multi-hop reasoning tasks, with a particular focus on their divergent performance in long-sequence extrapolation. By training GPT-J models on equivalent numeric and alphabetic tasks and employing controlled experiments, attention head behavior classification, and geometric analysis of Rotary Position Embeddings (RoPE), the work provides the first clear distinction and theoretical characterization of positional versus symbolic attention heads in terms of their computational roles. The authors introduce a “discrepancy” metric that quantitatively demonstrates the superior extrapolation robustness of symbolic mechanisms over positional ones. Furthermore, they establish that the presence of purely typed attention heads is critical for successful learning—a finding consistently validated across both controlled setups and real-world models.

0 citationsRead paper

Language Generation: Complexity Barriers and Implications for Learning

Nov 07, 2025

This paper investigates the practical feasibility of language generation in terms of sample efficiency. It establishes that, for classical language classes—including regular and context-free languages—successful generation may require a number of positive examples exceeding the bound of any computable function, rendering sample complexity uncomputable—even though these classes are theoretically learnable. Method: Integrating formal language theory, the PAC learning framework, and the Kleinberg–Mullainathan generative model, the paper rigorously derives and proves strong information-theoretic lower bounds on sample complexity. Contribution/Results: The work provides the first systematic characterization, from a computational complexity perspective, of fundamental sample barriers inherent to language generation. Crucially, it demonstrates that the empirical success of modern large language models cannot be fully explained by classical learnability theory alone; instead, it must rely on structural constraints unique to natural language. This insight offers a novel conceptual bridge between theoretical guarantees and practical performance.

0 citationsRead paper