Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment

2026-09-09

AI Summary

研究通过比较传统模型和Transformer在多种语言上的可读性评估，使用SHAP和TCAV方法分析了两者对语言特征的捕捉情况。

Abstract

Transformer-based models excel at Automatic Readability Assessment (ARA), yet feature-based models remain in active use because their predictions tie back to linguistic properties. This matters because readability labels are subjective and rater-dependent, so high accuracy on noisy ground truth may reflect surface patterns rather than the linguistic structure that defines difficulty. We test whether transformers internalize the same features as traditional models across Arabic, English, French, Hindi, and Russian using the ReadMe++ dataset. Shapley Additive Explanations (SHAP) identify the features driving traditional classifiers, which we then use as TCAV concept sets to probe multilingual XLM-R and language-specific encoders. Transformers recover surface-length, syntactic, and lexical-diversity signals, and reflect the ordinal CEFR structure of the traditional models. Alignment varies by model family, language, and layer, with language-specific encoders tracking traditional models more clearly than XLM-R. High linear separability does not always imply directional influence, limiting linear probing for count-based readability features.

Problem

• Automatic Readability Assessment
• Transformer-based models
• feature-based models
• linguistic properties
• ReadMe++ dataset

Innovation

• Transformer-based models
• Shapley Additive Explanations (SHAP)
• Testing with Concept Activation Vectors (TCAV)
• Automatic Readability Assessment (ARA)
• multilingual