Papers-to-Posts: Supporting Detailed Long-Document Summarization with an Interactive LLM-Powered Source Outline

2024-06-14

AI Summary

To address insufficient content controllability when compressing lengthy technical documents (e.g., research papers) into concise formats (e.g., blog posts), this paper proposes an interactive reverse-source outline mechanism. It explicitly models LLM-based summarization as an editable, traceable, structured outline—comprising hierarchical, semantically grounded nodes—enabling iterative user refinement of functional points to precisely govern information coverage. The method integrates large language models, dynamic content-to-outline mapping, and an interactive UI to realize an end-to-end, fine-grained summarization system. Empirical evaluation and real-world deployment demonstrate significant improvements: author satisfaction with content coverage increases markedly; information change per editing operation rises by 37%; and retention of critical research insights improves 2.1×. This work achieves, for the first time, bidirectional controllability—both selection and synthesis—over content in technical document summarization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Compressing long and technical documents (e.g.,>10 pages) into shorter-form articles (e.g.,<2 pages) is critical for communicating information to different audiences, for example, blog posts of scientific research paper or legal briefs of dense court proceedings. While large language models (LLMs) are powerful tools for condensing large amounts of text, current interfaces to these models lack support for understanding and controlling what content is included in a detailed summarizing article. Such capability is especially important for detail- and technical-oriented domains, in which tactical selection and coherent synthesis of key details is critical for effective communication to the target audience. For this, we present interactive reverse source outlines, a novel mechanism for controllable long-form summarization featuring outline bullet points with automatic point selections that the user can iteratively adjust to obtain an article with the desired content coverage. We implement this mechanism in Papers-to-Posts, a new LLM-powered system for authoring research-paper blog posts. Through a within-subjects lab study (n=20) and a between-subjects deployment study (n=37 blog posts, 26 participants), we compare Papers-to-Posts to a strong baseline tool that provides an LLM-generated draft and access to free-form prompting. Under time constraints, Papers-to-Posts significantly increases writer satisfaction with blog post quality, particularly with respect to content coverage. Furthermore, quantitative results showed an increase in editing power (change in text for an amount of time or writing actions) while using Papers-to-Posts, and qualitative results showed that participants found incorporating key research-paper insights in their blog posts easier while using Papers-to-Posts.

Problem

• Enabling controlled summarization of long technical documents
• Improving content coverage in detailed long-form summaries
• Facilitating interactive adjustment of key details in summaries

Innovation

• Interactive reverse source outlines for summarization
• LLM-powered system for detailed content coverage
• User-adjustable outline bullet points for control