Process-Oriented Evaluation of AI-Assisted Scientific Writing

2026-06-14

AI Summary

This study addresses the critical yet underexplored dynamics of human–AI collaboration in scientific abstract writing, where the quality of scientific communication hinges on effective drafting and revision. Leveraging a large-scale keystroke-level log dataset comprising 869 abstracts and over 240,000 editing actions, we propose the first process-oriented evaluation framework to compare revision trajectories between human- and AI-generated abstracts across dimensions of sentential agency, global coherence, and editing strategies. Integrating behavioral labeling, linguistic analysis of editing bursts, and fusion of local–global features from language models within a mixed-methods approach, we find that AI-generated text exhibits stronger sentential agency but weaker coherence; human experts adapt their editing strategies upon knowing the AI origin; and both humans and AI tend to correct weaknesses while neglecting refinement of strengths. These findings illuminate current language models’ limitations in structural coherence and offer empirical grounding for optimizing human–AI collaborative writing.

Abstract

Bad writing hinders the publication of science. The role of artificial intelligence (AI) in generating and editing scientific texts remains unsettled. Abstracts serve as the critical gateway to scientific manuscripts, often shaping readers' interest. We inspect how individuals revise AI-generated abstracts compared to human-authored abstracts when incentivized to communicate scientific content. Using 869 keystroke-level edit logs with 240k total edits, we construct behavioral labels and measure linguistic properties of edit bursts to investigate the edit trajectories. AI abstracts exhibit higher sentence-level agency, whereas human-authored abstracts outperform in global coherence, even with edits. Experts engage in stigmatic behavior, switching their strategy from predominantly restructuring to substitution when AI source is disclosed. Language Models (LMs) improve edit outcomes through a mix of local and global features, but still actively struggle with global coherence. Both humans and LMs often target the weakest sections of abstracts, but fail to improve stronger areas. Our large-scale process-oriented evaluation highlights the perks and pitfalls of both human and LM editing processes as machine-generated texts emerge in scientific communication.

Problem

• AI-assisted writing
• scientific abstracts
• global coherence
• editing behavior
• language models

Innovation

• process-oriented evaluation
• AI-assisted writing
• edit logs
• global coherence
• language models