CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?

2026-07-04

AI Summary

This study addresses the limited cross-lingual and cross-domain generalization of existing hallucination detection methods for large language models, which predominantly rely on English-centric internal representations. For the first time, it systematically evaluates the transferability of internal representation–based hallucination detection across six large language models in generative question-answering tasks involving both Arabic and English. Leveraging the TruthfulQA benchmark (including its newly developed Arabic version) and the HalluScore dataset, the work conducts comprehensive cross-lingual (Arabic ↔ English) and cross-domain experiments. The findings reveal that internal hallucination signals in most models exhibit notable cross-lingual and cross-domain transferability; however, cross-lingual performance critically depends on language alignment and class separability in the feature space, while cross-domain effectiveness within Arabic is significantly influenced by dataset-specific discrepancies.

Abstract

Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations and training a classifier on these features to detect hallucinations, demonstrating promising results. Notwithstanding this advancement, most internal-state hallucination detection techniques have been explored predominantly in English, raising the question of whether such internal signals generalize across different languages and domains. To address this gap, we present CrossHallu, the first study to evaluate the cross-lingual and cross-domain generalization of hallucination detection using internal representations from six LLMs on the generative question-answering task. We conduct a systematic Arabic <-> English evaluation using TruthfulQA, an Arabic translated version of TruthfulQA, and HalluScore. This evaluation encompasses monolingual training and testing, cross-lingual transfer, cross-domain transfer, and combined cross-lingual and cross-domain transfer. The results reveal that internal-state hallucination signals in LLMs transfer across languages and domains for most models, with cross-lingual performance highly dependent on both class separability and language alignment in the feature space, whereas cross-domain transfer within Arabic varies depending on the training and testing datasets used for the hallucination detector. The code is publicly available at https://github.com/aishaalansari57/CrossHal.

Problem

• hallucination detection
• cross-lingual generalization
• cross-domain generalization
• large language models
• internal representations

Innovation

• cross-lingual generalization
• hallucination detection
• internal representations
• large language models
• cross-domain transfer