Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training

2026-08-31

AI Summary

研究通过继续预训练大型语言模型以适应瑞典新闻领域，使用高质量数据集和特定基准评估，结合经验回放提升生成质量和事实知识。

Abstract

Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models' generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the models' in-domain performance gains.

Problem

• Large Language Models
• Swedish Journalism
• Continued Pre-Training
• Domain-Specific Benchmark
• Instruction Following

Innovation

• continued pre-training
• experience replay
• parameter-efficient fine-tuning
• low-rank adaptation
• targeted evaluation