Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models
This work investigates whether plan-guided summarization improves faithfulness of small language models (SLMs) on long narrative texts. Addressing the susceptibility of existing fine-grained plans to hallucination, we propose a high-level planning method grounded in narrative structure. Through automated evaluation and rigorous human assessment—specifically targeting faithfulness and hallucination—we find that neither fine-grained nor our novel high-level plan guidance significantly outperforms the plan-free baseline. The root cause is high hallucination rates inherent in the plans themselves, which undermine guidance efficacy and even propagate factual errors. To our knowledge, this is the first systematic study exposing the limitations of plan-guided summarization for complex narratives. Our findings caution against uncritical adoption of planning in long-text and low-resource settings, where plan hallucinations critically compromise reliability. The study provides key empirical evidence for developing trustworthy abstractive summarization systems, highlighting the necessity of hallucination-robust planning mechanisms.