LLM-OREF: An Open Relation Extraction Framework Based on Large Language Models

2025-09-18

AI Summary

Open relation extraction (OpenRE) aims to generalize models to unseen relations during training, yet existing clustering-based approaches rely on manual cluster labeling, limiting practical applicability. This paper proposes the first end-to-end large language model (LLM)-driven OpenRE framework that eliminates explicit clustering and human intervention. Instead, it leverages in-context learning to guide the LLM in directly identifying and naming novel relations. We introduce a novel relation discovery and denoising mechanism, along with a three-stage self-correcting reasoning strategy that employs cross-validation to select high-confidence instances, thereby enhancing robustness. Experiments on three standard benchmarks demonstrate substantial improvements over prior state-of-the-art methods, achieving, for the first time, fully automated relation discovery and naming without human annotations. The code is publicly available.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: Information Extraction
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The goal of open relation extraction (OpenRE) is to develop an RE model that can generalize to new relations not encountered during training. Existing studies primarily formulate OpenRE as a clustering task. They first cluster all test instances based on the similarity between the instances, and then manually assign a new relation to each cluster. However, their reliance on human annotation limits their practicality. In this paper, we propose an OpenRE framework based on large language models (LLMs), which directly predicts new relations for test instances by leveraging their strong language understanding and generation abilities, without human intervention. Specifically, our framework consists of two core components: (1) a relation discoverer (RD), designed to predict new relations for test instances based on extit{demonstrations} formed by training instances with known relations; and (2) a relation predictor (RP), used to select the most likely relation for a test instance from $n$ candidate relations, guided by extit{demonstrations} composed of their instances. To enhance the ability of our framework to predict new relations, we design a self-correcting inference strategy composed of three stages: relation discovery, relation denoising, and relation prediction. In the first stage, we use RD to preliminarily predict new relations for all test instances. Next, we apply RP to select some high-reliability test instances for each new relation from the prediction results of RD through a cross-validation method. During the third stage, we employ RP to re-predict the relations of all test instances based on the demonstrations constructed from these reliable test instances. Extensive experiments on three OpenRE datasets demonstrate the effectiveness of our framework. We release our code at https://github.com/XMUDeepLIT/LLM-OREF.git.

Problem

• Open relation extraction without human annotation
• Predicting new relations using large language models
• Self-correcting inference strategy for relation discovery

Innovation

• LLM-based framework for OpenRE
• Self-correcting inference strategy
• Automatic relation prediction without human intervention