The Moving Target: A Longitudinal Audit of Trustworthiness Drift Across Twelve Checkpoints of Open-Source Chat LLMs

2026-07-01

AI Summary

This study addresses a critical oversight in current model cards: the common practice of reusing trustworthiness scores across model checkpoints despite potential performance drift during version updates. We conduct a longitudinal audit of three consecutive checkpoints from four open-source chat models—Yi, Qwen, Mistral, and Gemma—evaluating them under a fixed trust benchmark and multiple prompt templates. Through multi-template testing, quantification of inter-generational drift, and robustness checks, we systematically demonstrate significant trustworthiness drift across successive versions of open large language models. Our findings reveal that the mean absolute drift between adjacent generations is substantially higher than under the null hypothesis, indicating that trust scores cannot be reliably transferred across checkpoints. Consequently, trustworthiness should be treated as a timestamped, checkpoint-specific attribute, necessitating the adoption of longitudinal model cards.

Abstract

Model cards quote trust-benchmark scores without recording when they were measured, and the same number is routinely carried across successive checkpoints of one release line as if the model behind it had not shifted. We test whether it has shifted by auditing four open-source release lines, Yi, Qwen, Mistral, and Gemma, at three successive generations each, on a fixed basket of trust benchmarks under multiple prompt templates. Mean absolute adjacent-generation drift lands well above an independence-based no-drift reference null, and the gap persists when we drop a benchmark, drop a release line, or switch to strict scoring. We therefore conclude that a trust score attached to a release line should not be carried forward to the next checkpoint without remeasurement; it should instead be reported as a checkpoint-bound, dated artefact, which we package as a longitudinal model card. Closed APIs, larger models, canonical benchmark protocols, and fixed month-cadence rules lie outside the audited scope and require their own evaluation.

Problem

• trustworthiness drift
• model cards
• checkpoints
• open-source LLMs
• longitudinal audit

Innovation

• trustworthiness drift
• longitudinal audit
• checkpoint-bound evaluation
• model cards
• open-source LLMs