An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face

2025-12-08

AI Summary

How to ensure semantic preservation during intelligent component optimization in Learning-Enabled Software Systems (LESS), avoiding unintended semantic drift? Method: We propose the first semantic-preserving assessment framework for large-scale model evolution. Leveraging the Hugging Face platform, we construct a reproducible data pipeline that collects 1.7 million commit records, Model Cards from 536 models, and over 4,000 performance metrics. We jointly analyze inter-version functional behavior consistency to identify canonical refactoring patterns and semantic drift signals, innovatively coupling commit histories, documentation metadata, and multidimensional performance metrics to formally define semantic preservation boundaries. Contribution/Results: We release HF-Evol—the largest publicly available ML evolution dataset to date—enabling detectable, quantifiable, and attributable semantic drift analysis. Our empirical study reveals cross-domain performance change patterns, establishing a practical, evidence-based evaluation paradigm for trustworthy LESS.

Technology Category

• Machine Learning: Evolutionary Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evolutionary Computation

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As machine learning (ML) becomes an integral part of high-autonomy systems, it is critical to ensure the trustworthiness of learning-enabled software systems (LESS). Yet, the nondeterministic and run-time-defined semantics of ML complicate traditional software refactoring. We define semantic preservation in LESS as the property that optimizations of intelligent components do not alter the system's overall functional behavior. This paper introduces an empirical framework to evaluate semantic preservation in LESS by mining model evolution data from HuggingFace. We extract commit histories, $ extit{Model Cards}$, and performance metrics from a large number of models. To establish baselines, we conducted case studies in three domains, tracing performance changes across versions. Our analysis demonstrates how $ extit{semantic drift}$ can be detected via evaluation metrics across commits and reveals common refactoring patterns based on commit message analysis. Although API constraints limited the possibility of estimating a full-scale threshold, our pipeline offers a foundation for defining community-accepted boundaries for semantic preservation. Our contributions include: (1) a large-scale dataset of ML model evolution, curated from 1.7 million Hugging Face entries via a reproducible pipeline using the native HF hub API, (2) a practical pipeline for the evaluation of semantic preservation for a subset of 536 models and 4000+ metrics and (3) empirical case studies illustrating semantic drift in practice. Together, these contributions advance the foundations for more maintainable and trustworthy ML systems.

Problem

• Evaluates semantic preservation in machine learning systems
• Detects semantic drift across model versions via metrics
• Establishes baselines for trustworthy learning-enabled software

Innovation

• Framework evaluates semantic preservation via HuggingFace data mining
• Pipeline analyzes commit histories and performance metrics for drift detection
• Large-scale dataset from 1.7 million entries supports empirical case studies