TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance

2026-08-08

AI Summary

This work addresses the challenge of tracing the provenance of language models and identifying overlaps in their training data using existing metadata, which often proves inadequate. The authors propose a training-free model fingerprinting method that leverages 250 fixed knowledge probes to elicit hidden states from a model, from which top-k vocabulary projections are extracted and decoded into token strings to compute Jaccard similarity. This approach establishes the first calibratable fingerprint directly in token space. The method reveals signals of shared training data prior to the emergence of task-specific capabilities and robustly identifies model lineages across different organizations, tokenizers, and architectures. Evaluated on 32 open-source models, it accurately distinguishes hierarchical model relationships, achieves an average retrieval rank of 1.8 for R1-distilled base models, yields an output-layer AUC of 0.90, and maintains a fingerprint similarity of 0.92 under int8 quantization.

Abstract

Establishing the provenance of a language model---including its base checkpoint and possible overlap in training distributions---is a governance challenge that metadata alone cannot resolve. We introduce a training-free fingerprint based on the top-$k$ vocabulary projections of late hidden states elicited by 250 fixed knowledge probes, compared using Jaccard overlap over decoded token strings. We evaluate the method on 32 open-weight models from nine families (0.6B--32B) with documented relationships. (1)~A \emph{similarity ladder} broadly follows model relatedness: independently trained models on identical data score 0.48 raw (0.35 vocabulary-corrected), followed by shared-base fine-tunes (0.39/0.33), same-developer relatives (0.38/0.28), and models with no documented relationship (0.22/0.17). This identical-data signal persists across three organizations, two tokenizer families, and two architecture classes, and emerges within the first 1\% of training before measurable task competence, suggesting a contribution from shared training data beyond capability convergence. (2)~As a nearest-neighbor \emph{lineage-retrieval} method, the fingerprint ranks the exact documented base among the top two candidates for all five R1 distillations (mean rank 1.8, MRR 0.60), including a math-specialized base not identifiable from coarse metadata. (3)~A \emph{depth ablation} shows that lineage group discrimination strengthens toward the output distribution, with AUC increasing from 0.72 at quarter depth to 0.90 at the output; using only the top 5 output tokens retains AUC 0.87. (4)~The fingerprint remains stable under quantization, with Jaccard similarity of 0.92 under int8 and 0.82--0.85 under int4, compared with a maximum cross-model similarity of 0.81 in the calibration pool. We release the probes, code, and fingerprints.

Problem

• language model provenance
• training data overlap
• model fingerprinting
• base checkpoint identification
• metadata limitations

Innovation

• model provenance
• training-free fingerprint
• token-space representation
• lineage retrieval
• Jaccard similarity