ByteDance · Agora

Selected work

Representative Papers

Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model

Jan 22, 2026

This work addresses the longstanding performance gap between diffusion-based and autoregressive large language models for code, where diffusion models typically underperform under comparable resource constraints. Building upon the Seed-Coder architecture and dataset, the authors introduce block-wise diffusion with continuous pretraining, enhanced by a tailored warm-up strategy and a block-level truncated noise schedule to enable efficient and stable training. Under identical architectural and data conditions, this approach enables diffusion models to consistently surpass autoregressive baselines, achieving substantial improvements in structured code editing, reasoning, and low-resource language modeling. Notably, using only pretraining and supervised fine-tuning, the proposed method outperforms a range of 8B-scale autoregressive and diffusion models across multiple code benchmarks.

4 citations1 influentialRead paper

HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction

Jan 19, 2026

This work addresses the challenge of jointly modeling long user behavior sequences and heterogeneous non-sequential features under stringent efficiency constraints in industrial-scale recommender systems. To this end, we propose HyFormer, a unified hybrid Transformer architecture that, for the first time, integrates sequence modeling and feature interaction within a single backbone network. The core innovation lies in an alternating optimization mechanism comprising Query Decoding—leveraging hierarchical key-value representations for efficient long-sequence decoding—and Query Boosting, which enables cross-query and cross-sequence token mixing for dynamic semantic enhancement. Evaluated on billion-scale industrial datasets, HyFormer significantly outperforms state-of-the-art methods LONGER and RankMixer under identical parameter and FLOPs budgets, with online A/B tests confirming substantial performance gains in high-traffic deployment scenarios.

4 citationsRead paper

Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst

May 20, 2025

To address the bottleneck of large language models (LLMs) relying heavily on extensive human-annotated reasoning examples (e.g., Chain-of-Thought, CoT), this paper proposes Self-Reasoning Language Models (SRLMs). Our method introduces a meta-reasoning framework based on self-training, integrating prompt engineering, sampling augmentation, and cognitive skill modeling to enable iterative, annotation-free improvement of reasoning capabilities. Crucially, SRLMs leverage only 1,000 high-quality, few-shot reasoning examples as “catalysts” to trigger autonomous generation of long-horizon, multi-step implicit reasoning chains. The core innovation lies in the novel mechanism of “few-shot reasoning catalysis + model-autogenerated long-chain data,” which substantially enhances reasoning depth, diversity, and stability. Experiments across five benchmarks—including MMLU and GSM8K—demonstrate an average gain of +2.5 points, rising to +7.89 points under 64 sampling iterations.

3 citationsRead paper

Real-Time Aligned Reward Model beyond Semantics

Jan 30, 2026

This work addresses the challenge in reinforcement learning where continuous policy distribution shifts often lead to reward model over-optimization and alignment failure. To mitigate this, the paper proposes the R2M framework, which introduces a policy feedback mechanism into the reinforcement learning from human feedback (RLHF) pipeline. For the first time, R2M leverages real-time hidden states from the policy model to dynamically adjust the reward model, enabling online alignment between the two. Unlike conventional approaches that rely on static, pre-trained semantic representations, R2M employs a lightweight architecture and dynamic alignment strategy to significantly alleviate reward misalignment caused by distributional shifts. This approach effectively suppresses over-optimization and enhances both the stability and alignment performance of the reward model during policy updates.

2 citationsRead paper

LoL: Longer than Longer, Scaling Video Generation to Hour

Jan 23, 2026

This work addresses the "sink frame collapse" phenomenon in autoregressive long-video generation, where excessive reliance on specific sink frames by attention mechanisms leads to abrupt scene transitions and repetitive motion patterns. To mitigate this issue without retraining, the authors propose a lightweight intervention: injecting random perturbations into the rotary positional embeddings (RoPE) within multi-head attention layers. This simple modification effectively breaks the homogeneity across attention heads, thereby suppressing sink frame collapse. The method enables real-time, streaming video generation of unlimited length while maintaining high visual fidelity. In public demonstrations, it successfully produced a continuous 12-hour video—the longest streaming-generation result reported to date—with minimal quality degradation over time.

2 citationsRead paper

Recent publications