TELLME: Test-Enhanced Learning for Language Model Enrichment

2026-08-12

AI Summary

This work addresses the challenges of large-scale data acquisition and high computational costs in domain adaptation through continual pretraining by introducing, for the first time, Test-Enhanced Learning (TEL) into the continual pretraining framework. By integrating an embedded quiz mechanism, the proposed approach enhances the model’s efficiency in acquiring domain-specific knowledge and its ability to retain long-term memory. Empirical results demonstrate that this method significantly improves domain adaptation efficiency, achieving up to a 23.6% performance gain on financial-domain tasks and a 9.8% improvement in long-term memory retention. The study thus establishes a novel paradigm for effective and cost-efficient domain adaptation.

Abstract

Continual pre-training (CPT) has been widely adopted as a method for domain adaptation in large language models. However, CPT has consistently been accompanied by challenges, such as the difficulty of acquiring large-scale domain-specific datasets and high computational costs. In this study, we propose a novel method called Test-Enhanced Learning for Language Model Enrichment (TELLME) to alleviate these issues. TELLME leverages the TestEnhanced Learning (TEL) principle, whereby the model's training efficiency is improved using quizzes during training. It integrates this principle with CPT, thereby promoting efficient domain-specific knowledge acquisition and long-term memory retention. Experimental results demonstrate that TELLME outperforms existing methods by up to 23.6% in the financial domain and achieves a 9.8% improvement in long-term memory retention.

Problem

• continual pre-training
• domain adaptation
• large language models
• computational cost
• domain-specific data

Innovation

• Test-Enhanced Learning
• Continual Pre-training
• Domain Adaptation
• Long-term Memory Retention
• Language Model Enrichment