Separating Representation from Reconstruction Enables Scalable Text Encoders

2026-07-04

AI Summary

This work addresses the limitation of conventional BERT, where representation learning is tightly coupled with token reconstruction, thereby constraining representation usability under frozen probing and hindering model scalability. To overcome this, the authors propose CrossBERT, the first approach to decouple these two objectives through a dual-branch architecture and complementary masking strategies. This design enables high masking ratios (≥50%) while allowing gradient computation over all tokens, substantially improving training efficiency and sample utilization. Evaluated on the MTEB (English, v2) and frozen GLUE benchmarks, CrossBERT demonstrates superior performance, achieving 1.5–2× higher throughput, 2× better sample efficiency, and monotonic scalability with respect to model size.

Abstract

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despite improved perplexity. The misalignment originates in BERT's flat design, which couples representation learning to the token reconstruction loss. We propose $\textbf{CrossBERT}$, a two-part architecture that separates the learning of high-quality encoded representations from the rigid grounding of token reconstruction. This design further enables high masking ratios ($\ge 50\%$) and gradient collection over all tokens via a $\textit{Complementary Masking Strategy}$, respectively increasing throughput by $1.5$ to $2\times$ and sample efficiency by $2\times$. Overall, CrossBERT demonstrates monotonic scaling and superior performance on MTEB(eng, v2) and frozen GLUE benchmarks.

Problem

• text encoders
• representation learning
• token reconstruction
• scalability
• frozen probing

Innovation

• representation-reconstruction decoupling
• CrossBERT
• complementary masking strategy
• scalable text encoders
• frozen probing