HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

2026-07-02

AI Summary

Existing data mixing approaches are constrained by fixed semantic labels and a single granularity level, limiting their ability to flexibly explore combinatorial effects across multiple granularities. This work proposes a hierarchical, data-driven multi-granularity annotation framework that leverages learnable semantic transformations and a three-stage residual vector quantization scheme to generate up to 130,000 reusable hierarchical document codes, enabling dynamic navigation from coarse to fine granularities. Evaluated in a pretraining setting with 1B parameters and 25B tokens, the method—combined with an equal-subbucket coverage strategy—achieves an average performance gain of +0.0253 across 16 tasks at specific granularities, demonstrating a significant interaction effect between granularity selection and mixing strategy.

Abstract

Most data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels, including provenance, topic or format taxonomies, and flat embedding clusters, commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to approximately 130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: at one prefix length, a combined Stage-2 rule contrast, equal-subbucket coverage versus size-proportional within-bucket quality top-30%, lifts a 16-task capability macro-average by +0.0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract approximately 5x. HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.

Problem

• data mixing
• label granularity
• pre-training
• semantic hierarchy
• mixture design

Innovation

• multi-granularity labeling
• hierarchical vector quantization
• data mixture
• pre-training
• learned semantic transform