Scaling In, Not Up? Testing Thick Citation Context Analysis with GPT-5 and Fragile Prompts

2026-02-25

AI Summary

This study proposes an “inward-expansion” paradigm for citation context analysis, moving beyond conventional large-scale label classification by leveraging a large language model (GPT-5) to conduct thick, verifiable interpretive analyses of individual complex cases. The methodology employs a two-stage prompting pipeline with a 2×3 balanced prompt design, integrating both the citation and its full-text context to systematically investigate how prompt structure influences the model’s interpretive pathways and lexical choices. Across 90 reconstructions yielding 450 hypotheses, the study identifies 21 recurrent explanatory strategies: surface-level classifications remain stably “supplementary,” whereas deeper interpretations are markedly shaped by prompt formulation, revealing a structured yet highly sensitive interpretive space. This work establishes prompt sensitivity as a core methodological concern, offering a novel pathway toward explainable citation analysis.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

This paper tests whether large language models (LLMs) can support interpretative citation context analysis (CCA) by scaling in thick, text-grounded readings of a single hard case rather than scaling up typological labels. It foregrounds prompt-sensitivity analysis as a methodological issue by varying prompt scaffolding and framing in a balanced 2x3 design. Using footnote 6 in Chubin and Moitra (1975) and Gilbert's (1977) reconstruction as a probe, I implement a two-stage GPT-5 pipeline: a citation-text-only surface classification and expectation pass, followed by cross-document interpretative reconstruction using the citing and cited full texts. Across 90 reconstructions, the model produces 450 distinct hypotheses. Close reading and inductive coding identify 21 recurring interpretative moves, and linear probability models estimate how prompt choices shift their frequencies and lexical repertoire. GPT-5's surface pass is highly stable, consistently classifying the citation as "supplementary". In reconstruction, the model generates a structured space of plausible alternatives, but scaffolding and examples redistribute attention and vocabulary, sometimes toward strained readings. Relative to Gilbert, GPT-5 detects the same textual hinges yet more often resolves them as lineage and positioning than as admonishment. The study outlines opportunities and risks of using LLMs as guided co-analysts for inspectable, contestable interpretative CCA, and it shows that prompt scaffolding and framing systematically tilt which plausible readings and vocabularies the model foregrounds.

Problem

• citation context analysis
• large language models
• prompt sensitivity
• interpretative analysis
• GPT-5

Innovation

• citation context analysis
• prompt scaffolding
• scaling in
• interpretative reconstruction
• large language models