Zero-Shot End-to-End Relation Extraction in Chinese: A Comparative Study of Gemini, LLaMA and ChatGPT

2025-02-08

AI Summary

This paper addresses the underexplored task of zero-shot end-to-end relation extraction for Chinese—simultaneously identifying entities and extracting relations without labeled data. We conduct the first systematic evaluation of three major large language models (LLMs): ChatGPT, Gemini, and LLaMA. To this end, we propose a Chinese-oriented zero-shot prompting inference method and establish a domain-adapted evaluation protocol with an end-to-end joint assessment framework. Experimental results show that ChatGPT achieves the highest overall accuracy; Gemini exhibits the fastest inference speed, making it suitable for real-time applications; and LLaMA demonstrates significant limitations in fine-grained Chinese semantic modeling. Our study reveals inherent accuracy–efficiency trade-offs among LLMs in zero-shot Chinese relation extraction, providing empirical evidence and methodological guidance for model selection and optimization of Chinese semantic understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study investigates the performance of various large language models (LLMs) on zero-shot end-to-end relation extraction (RE) in Chinese, a task that integrates entity recognition and relation extraction without requiring annotated data. While LLMs show promise for RE, most prior work focuses on English or assumes pre-annotated entities, leaving their effectiveness in Chinese RE largely unexplored. To bridge this gap, we evaluate ChatGPT, Gemini, and LLaMA based on accuracy, efficiency, and adaptability. ChatGPT demonstrates the highest overall performance, balancing precision and recall, while Gemini achieves the fastest inference speed, making it suitable for real-time applications. LLaMA underperforms in both accuracy and latency, highlighting the need for further adaptation. Our findings provide insights into the strengths and limitations of LLMs for zero-shot Chinese RE, shedding light on trade-offs between accuracy and efficiency. This study serves as a foundation for future research aimed at improving LLM adaptability to complex linguistic tasks in Chinese NLP.

Problem

• Evaluate Chinese zero-shot relation extraction
• Compare ChatGPT, Gemini, LLaMA performance
• Assess accuracy, efficiency, and adaptability

Innovation

• Zero-shot end-to-end relation extraction
• Comparative study of ChatGPT, Gemini, LLaMA
• Evaluation based on accuracy, efficiency, adaptability