Selected work

Representative Papers

Targeted Neuron Modulation via Contrastive Pair Search

May 12, 2026

Current language models lack transparency in their mechanisms for rejecting harmful requests, and prevailing intervention techniques often degrade output coherence under strong intervention intensities. This work proposes Contrastive Neuron Attribution (CNA), a method that precisely identifies, during the forward pass, the key MLP neurons responsible for distinguishing harmful from benign inputs and applies targeted modulation to steer model behavior. The study reveals, for the first time, that alignment fine-tuning transforms the inherent discriminative structures of base models into sparse, localized rejection gating mechanisms. Requiring neither gradients nor additional training, CNA enables cross-architecture analysis (e.g., Llama/Qwen) and reduces refusal rates by over 50% on standard jailbreaking benchmarks while preserving output fluency and non-degeneracy across all intervention strengths.

0 citationsRead paper

Efficient Pre-Training with Token Superposition

May 07, 2026

This work addresses the high computational cost and low data throughput efficiency of large language model pretraining, which often necessitates modifications to system architecture in existing approaches. The authors propose Token-Superposition Training (TST), a two-stage pretraining method that requires no changes to parallelization strategies, optimizers, tokenizers, data pipelines, or model architectures. In the first stage, consecutive tokens are compressed into bags and trained efficiently using multi-hot cross-entropy loss with superposition; standard training is then resumed in the second stage. Evaluated across models ranging from 270M to 10B parameters, TST significantly improves data throughput per FLOP, reduces pretraining time by up to 2.5× at equivalent loss, and consistently outperforms baseline methods on downstream tasks, demonstrating strong generality and plug-and-play compatibility.

0 citationsRead paper

Long Context Pre-Training with Lighthouse Attention

May 07, 2026

This work addresses the quadratic computational complexity of standard dot-product attention, which hinders the training of causal Transformers on extremely long sequences. The authors propose Lighthouse Attention—a symmetric, selection-based hierarchical attention mechanism used exclusively during training—that adaptively compresses sequences by jointly pooling queries, keys, and values through a gradient-agnostic hierarchical selection and symmetric compression strategy. This approach enhances parallelism while strictly preserving causality. Integrated within a two-stage training paradigm—initial pretraining with Lighthouse Attention followed by fine-tuning with full attention—the method substantially reduces training costs. Experiments on small-scale language models demonstrate that, under identical settings, the proposed approach achieves shorter total training time and yields lower final loss after the recovery phase.

0 citationsRead paper

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

Apr 29, 2026

The precise mechanisms by which subword tokenization enhances language model performance remain unclear. This work systematically disentangles the independent effects of subword tokenization on training efficiency and linguistic priors within a controlled byte-level pretraining framework, by simulating subword boundaries, vocabulary scaling, and throughput variations. For the first time, subword boundaries are explicitly modeled as an inductive bias, revealing that their combination with increased training throughput constitutes the core advantage of subword-based models. These findings provide both theoretical grounding and practical guidance for hybrid pretraining approaches that integrate byte-level and subword representations.

0 citationsRead paper

Hermes 4 Technical Report

Aug 25, 2025

To address the challenge of balancing complex multi-step reasoning with open-ended instruction following in large language models, this paper introduces the Hermes-4 series of hybrid reasoning models. Methodologically, we propose a unified training framework that jointly optimizes structured multi-turn reasoning and open instruction comprehension, leveraging large-scale cleaned corpora and high-quality synthetic data—including behavior-guided reasoning trajectories. A multi-dimensional evaluation suite ensures training alignment and fidelity. Our key contribution is the first end-to-end unified modeling of structured reasoning chains (e.g., mathematical derivation, program debugging) and general-purpose instruction understanding. Hermes-4 achieves significant improvements over same-scale baselines on benchmarks including GSM8K, HumanEval, and MMLU. We fully open-source the model weights and detailed training configurations to foster reproducible research in hybrid reasoning.

0 citationsRead paper

Recent publications

Latest Papers

Targeted Neuron Modulation via Contrastive Pair Search

May 12, 2026

Current language models lack transparency in their mechanisms for rejecting harmful requests, and prevailing intervention techniques often degrade output coherence under strong intervention intensities. This work proposes Contrastive Neuron Attribution (CNA), a method that precisely identifies, during the forward pass, the key MLP neurons responsible for distinguishing harmful from benign inputs and applies targeted modulation to steer model behavior. The study reveals, for the first time, that alignment fine-tuning transforms the inherent discriminative structures of base models into sparse, localized rejection gating mechanisms. Requiring neither gradients nor additional training, CNA enables cross-architecture analysis (e.g., Llama/Qwen) and reduces refusal rates by over 50% on standard jailbreaking benchmarks while preserving output fluency and non-degeneracy across all intervention strengths.

0 citationsRead paper

Efficient Pre-Training with Token Superposition

May 07, 2026

This work addresses the high computational cost and low data throughput efficiency of large language model pretraining, which often necessitates modifications to system architecture in existing approaches. The authors propose Token-Superposition Training (TST), a two-stage pretraining method that requires no changes to parallelization strategies, optimizers, tokenizers, data pipelines, or model architectures. In the first stage, consecutive tokens are compressed into bags and trained efficiently using multi-hot cross-entropy loss with superposition; standard training is then resumed in the second stage. Evaluated across models ranging from 270M to 10B parameters, TST significantly improves data throughput per FLOP, reduces pretraining time by up to 2.5× at equivalent loss, and consistently outperforms baseline methods on downstream tasks, demonstrating strong generality and plug-and-play compatibility.

0 citationsRead paper

Long Context Pre-Training with Lighthouse Attention

May 07, 2026

This work addresses the quadratic computational complexity of standard dot-product attention, which hinders the training of causal Transformers on extremely long sequences. The authors propose Lighthouse Attention—a symmetric, selection-based hierarchical attention mechanism used exclusively during training—that adaptively compresses sequences by jointly pooling queries, keys, and values through a gradient-agnostic hierarchical selection and symmetric compression strategy. This approach enhances parallelism while strictly preserving causality. Integrated within a two-stage training paradigm—initial pretraining with Lighthouse Attention followed by fine-tuning with full attention—the method substantially reduces training costs. Experiments on small-scale language models demonstrate that, under identical settings, the proposed approach achieves shorter total training time and yields lower final loss after the recovery phase.

0 citationsRead paper

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

Apr 29, 2026

The precise mechanisms by which subword tokenization enhances language model performance remain unclear. This work systematically disentangles the independent effects of subword tokenization on training efficiency and linguistic priors within a controlled byte-level pretraining framework, by simulating subword boundaries, vocabulary scaling, and throughput variations. For the first time, subword boundaries are explicitly modeled as an inductive bias, revealing that their combination with increased training throughput constitutes the core advantage of subword-based models. These findings provide both theoretical grounding and practical guidance for hybrid pretraining approaches that integrate byte-level and subword representations.

0 citationsRead paper

Hermes 4 Technical Report

Aug 25, 2025

To address the challenge of balancing complex multi-step reasoning with open-ended instruction following in large language models, this paper introduces the Hermes-4 series of hybrid reasoning models. Methodologically, we propose a unified training framework that jointly optimizes structured multi-turn reasoning and open instruction comprehension, leveraging large-scale cleaned corpora and high-quality synthetic data—including behavior-guided reasoning trajectories. A multi-dimensional evaluation suite ensures training alignment and fidelity. Our key contribution is the first end-to-end unified modeling of structured reasoning chains (e.g., mathematical derivation, program debugging) and general-purpose instruction understanding. Hermes-4 achieves significant improvements over same-scale baselines on benchmarks including GSM8K, HumanEval, and MMLU. We fully open-source the model weights and detailed training configurations to foster reproducible research in hybrid reasoning.

0 citationsRead paper