🤖 AI Summary
This study addresses the challenge of underreported and undiagnosed foodborne gastrointestinal (GI) illnesses—cases often missed by conventional public health surveillance. We propose a passive disease signal detection paradigm leveraging online restaurant reviews and large language models (LLMs). Methodologically, we introduce the first multi-level expert annotation schema tailored for GI diseases, enabling joint extraction of disease mentions, symptom expressions, and implicated foods. Using open-source LLMs (e.g., Llama, Phi), we implement end-to-end information extraction via zero-shot and few-shot prompting, and rigorously evaluate model robustness across gender, geographic, and cuisine-related biases. Experimental results show micro-F1 scores exceeding 90% across all three tasks, with prompting outperforming fine-tuned RoBERTa baselines. Our key contribution is demonstrating that lightweight prompting strategies achieve high accuracy and strong generalizability in fine-grained health information extraction—establishing a scalable, low-cost technical foundation for broad-coverage digital epidemiological surveillance.
📝 Abstract
Foodborne gastrointestinal (GI) illness is a common cause of ill health in the UK. However, many cases do not interact with the healthcare system, posing significant challenges for traditional surveillance methods. The growth of publicly available online restaurant reviews and advancements in large language models (LLMs) present potential opportunities to extend disease surveillance by identifying public reports of GI illness. In this study, we introduce a novel annotation schema, developed with experts in GI illness, applied to the Yelp Open Dataset of reviews. Our annotations extend beyond binary disease detection, to include detailed extraction of information on symptoms and foods. We evaluate the performance of open-weight LLMs across these three tasks: GI illness detection, symptom extraction, and food extraction. We compare this performance to RoBERTa-based classification models fine-tuned specifically for these tasks. Our results show that using prompt-based approaches, LLMs achieve micro-F1 scores of over 90% for all three of our tasks. Using prompting alone, we achieve micro-F1 scores that exceed those of smaller fine-tuned models. We further demonstrate the robustness of LLMs in GI illness detection across three bias-focused experiments. Our results suggest that publicly available review text and LLMs offer substantial potential for public health surveillance of GI illness by enabling highly effective extraction of key information. While LLMs appear to exhibit minimal bias in processing, the inherent limitations of restaurant review data highlight the need for cautious interpretation of results.