Comparative Analysis of Listwise Reranking with Large Language Models in Limited-Resource Language Contexts

2024-12-28

AI Summary

This study addresses the challenge of listwise re-ranking for low-resource African languages, where labeled data and domain-specific ranking models are scarce. Method: We propose a lightweight LLM-based re-ranking framework leveraging listwise prompt engineering and cross-lingual transfer inference, systematically evaluating models—including RankGPT-3.5 and Rank4o-mini—under standardized IR metrics (nDCG@10, MRR@100). Contribution/Results: All LLM rankers significantly outperform the BM25-DT baseline, achieving 32–57% improvements in both nDCG@10 and MRR@100. These results demonstrate strong zero-shot generalization to under-resourced languages and highlight the framework’s viability for low-cost, scalable deployment. To our knowledge, this is the first systematic evaluation of LLMs for listwise re-ranking in low-resource African languages, establishing a novel, resource-efficient re-ranking paradigm for information retrieval in data-scarce linguistic settings.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have demonstrated significant effectiveness across various NLP tasks, including text ranking. This study assesses the performance of large language models (LLMs) in listwise reranking for limited-resource African languages. We compare proprietary models RankGPT3.5, Rank4o-mini, RankGPTo1-mini and RankClaude-sonnet in cross-lingual contexts. Results indicate that these LLMs significantly outperform traditional baseline methods such as BM25-DT in most evaluation metrics, particularly in nDCG@10 and MRR@100. These findings highlight the potential of LLMs in enhancing reranking tasks for low-resource languages and offer insights into cost-effective solutions.

Problem

• Large Language Models
• African Languages
• Text Ranking

Innovation

• Large Language Models
• Low-resource African Languages
• Ranking Performance