GS-BrainText: A Multi-Site Brain Imaging Report Dataset from Generation Scotland for Clinical Natural Language Processing Development and Validation

2026-03-27

AI Summary

This study addresses the scarcity of multicenter brain imaging report datasets in the UK, which has hindered the development of generalizable clinical natural language processing (NLP) systems. To this end, we introduce GS-BrainText, a novel dataset comprising 8,511 brain radiology reports from five NHS Health Boards in Scotland, with 2,431 reports independently and blindly annotated by a multidisciplinary team for 24 brain disease phenotypes using a standardized annotation protocol and rigorous quality assurance mechanisms. As the first UK-based brain imaging text resource spanning multiple centers and a broad age distribution, GS-BrainText enables systematic evaluation of NLP model generalizability across institutions, phenotypes, and demographic groups. Benchmarking with EdIE-R reveals substantial performance variation across health boards (F1: 86.13–98.13), phenotypes (F1: 22.22–100), and age groups (F1: 87.01–98.13), highlighting critical challenges in achieving robust clinical NLP generalization.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We present GS-BrainText, a curated dataset of 8,511 brain radiology reports from the Generation Scotland cohort, of which 2,431 are annotated for 24 brain disease phenotypes. This multi-site dataset spans five Scottish NHS health boards and includes broad age representation (mean age 58, median age 53), making it uniquely valuable for developing and evaluating generalisable clinical natural language processing (NLP) algorithms and tools. Expert annotations were performed by a multidisciplinary clinical team using an annotation schema, with 10-100% double annotation per NHS health board and rigorous quality assurance. Benchmark evaluation using EdIE-R, an existing rule-based NLP system developed in conjunction with the annotation schema, revealed some performance variation across health boards (F1: 86.13-98.13), phenotypes (F1: 22.22-100) and age groups (F1: 87.01-98.13), highlighting critical challenges in generalisation of NLP tools. The GS-BrainText dataset addresses a significant gap in available UK clinical text resources and provides a valuable resource for the study of linguistic variation, diagnostic uncertainty expression and the impact of data characteristics on NLP system performance.

Problem

• clinical natural language processing
• generalisation
• linguistic variation
• diagnostic uncertainty
• multi-site dataset

Innovation

• multi-site clinical NLP
• expert-annotated radiology reports
• generalisation evaluation
• brain disease phenotypes
• clinical text dataset