Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed

2026-08-12

AI Summary

It remains unclear how to effectively construct small language models (SLMs) that exhibit high trustworthiness across multiple dimensions—fairness, robustness, privacy, and ethics. This work presents the first systematic comparison between training SLMs from scratch and compressing large language models (LLMs), introducing a comprehensive evaluation framework that assesses trustworthiness along these four axes. The study investigates the impact of pruning, quantization, and knowledge distillation on the trust-related properties of SLMs. Findings reveal that quantization preserves trustworthiness significantly better than pruning. Moreover, SLMs derived via quantization from trustworthy LLMs outperform natively trained small models in both trustworthiness and task adaptability. Further gains in reliability are achievable by incorporating knowledge distillation into the compression pipeline.

Abstract

Small Language Models (SLMs) have emerged as a more efficient alternative to traditional Large Language Models (LLMs), offering promising potential in resource-constrained scenarios. Existing approaches to building SLMs typically follow two paths: training compact models from scratch, or compressing larger pre-trained models using methods such as pruning, quantization, or distillation. As language models become increasingly integrated into real-world applications, ensuring their trustworthiness has become a critical concern. However, how to build trustworthy SLMs remains an underexplored question. In this work, we present a comprehensive evaluation of SLM trustworthiness across multiple dimensions, including fairness, robustness, privacy, and ethics. We first examine the effects of pruning and quantization, and find that quantization is significantly more effective in preserving trustworthiness compared to pruning. More importantly, we demonstrate that compressing a reliable large model via quantization can produce SLMs with superior trustworthiness and adaptability compared to using small models trained from scratch. Furthermore, knowledge distillation from trustworthy teacher models can further enhance the reliability of SLMs. We hope our findings provide practical guidance and a foundation for future research into the development and deployment of trustworthy small language models.

Problem

• Small Language Models
• Trustworthiness
• Model Compression
• Quantization
• Knowledge Distillation

Innovation

• Small Language Models
• Trustworthiness
• Quantization
• Knowledge Distillation
• Model Compression