Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers

2026-06-18

AI Summary

Existing fixed-compression-ratio tokenizers struggle to balance generation quality and computational cost, while variable-length tokenization methods suffer from positional dependencies that cause inconsistent representations across token lengths, hindering effective processing by a single diffusion model. This work proposes a learnable global merging–based variable-length tokenization approach that employs a data-agnostic, adaptive token merging mechanism to maintain semantic alignment across varying compression ratios while remaining compatible with diffusion Transformers. Notably, it achieves cross-scale representation consistency during generation without requiring prior knowledge of the merging pattern—a first in the field. Evaluated on ImageNet 256×256 image generation, the method significantly outperforms existing variable-length tokenization strategies, achieving a superior trade-off between generation fidelity (gFID) and computational overhead.

Abstract

Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality-compute trade-off is largely constrained by the tokenizer's fixed compression ratio. Variable-length tokenizers (VLTs) promise adaptive compression by varying token counts, allowing diffusion models to flexibly balance quality and compute. However, conventional VLTs modulate length by truncating ordered token sequences, which makes token semantics depend on token position and breaks representational alignment across lengths. This leads to a cross-length shift in the latent distribution that hinders a single variable-length diffusion model from operating effectively. To address this, we propose a novel variable-length tokenizer that modulates length by merging tokens. We show that encouraging similar tokens to merge enables direct cross-length representation alignment when the diffusion transformer operates according to the merging pattern. Since conventional merging methods are data-dependent, making the merging pattern inaccessible during generation, we introduce learnable global merging, which is data-independent, to ensure compatibility with diffusion transformers. On ImageNet 256$\times$256 generation, our merging-based variable-length tokenizer integrated with a diffusion transformer achieves a superior gFID-compute trade-off compared to prior VLT methods. Code is available at [this https URL](https://github.com/movinghoon/lgm)

Problem

• Variable-Length Tokenization
• Latent Diffusion Models
• Representation Alignment
• Cross-Length Shift
• Token Merging

Innovation

• variable-length tokenization
• learnable global merging
• diffusion transformer
• latent alignment
• adaptive compression