Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs

2026-02-17

AI Summary

This study addresses the challenge posed by the unstructured nature of large-scale historical interview transcripts—comprising 350,000 activity and organization mentions—which has hindered quantitative research on social integration. To overcome this, the authors develop a participation taxonomy that systematically categorizes activities along dimensions of type, sociability, frequency, and physical demand. For the first time, they apply open-source large language models to historical oral archives, employing a multi-round voting strategy to automatically annotate a human-labeled gold-standard dataset. The model’s outputs demonstrate high agreement with expert judgments, effectively transforming unstructured narrative text into structured social indicators. This approach not only enables scalable quantification of historical social participation but also offers a methodological innovation that expands data resources for social integration research.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Web Mining and Content Analysis: Bridging structured and unstructured data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Digitized historical archives make it possible to study everyday social life on a large scale, but the information extracted directly from text often does not directly allow one to answer the research questions posed by historians or sociologists in a quantitative manner. We address this problem in a large collection of Finnish World War II Karelian evacuee family interviews. Prior work extracted more than 350K mentions of leisure time activities and organizational memberships from these interviews, yielding 71K unique activity and organization names -- far too many to analyze directly. We develop a categorization framework that captures key aspects of participation (the kind of activity/organization, how social it typically is, how regularly it happens, and how physically demanding it is). We annotate a gold-standard set to allow for a reliable evaluation, and then test whether large language models can apply the same schema at scale. Using a simple voting approach across multiple model runs, we find that an open-weight LLM can closely match expert judgments. Finally, we apply the method to label the 350K entities, producing a structured resource for downstream studies of social integration and related outcomes.

Problem

• social integration
• historical archives
• activity categorization
• displaced populations
• quantitative history

Innovation

• large language models
• social integration
• historical archives
• activity categorization
• structured annotation