Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI

2026-08-06

AI Summary

This work addresses the challenge of efficiently and accurately extracting complex nested structures from unstructured text and enabling semantic-level automated evaluation. The authors propose a schema-guided, end-to-end framework that integrates domain-specific knowledge schemas with generative AI models—such as Claude Opus 3—to perform zero-shot, one-pass extraction of hierarchical attributes with variable cardinality. Automated evaluation is achieved through path alignment and fine-grained semantic matching algorithms. The framework demonstrates strong transferability across models, institutions, and languages, successfully extracting 12 out of 14 attributes in NICE documents with F1 scores exceeding 90%. It achieves a 30-fold speedup over manual annotation while significantly improving extraction efficiency, consistency, and generalizability.

Abstract

We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.

Problem

• information extraction
• structured data
• semantic evaluation
• hierarchical information
• generative AI

Innovation

• schema-guided extraction
• hierarchical information extraction
• generative AI
• semantic evaluation
• zero-shot learning