The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation

2020-11-01

Findings

AI Summary

This work investigates neural machine translation (NMT) models’ capacity to resolve lexical and syntactic ambiguities via commonsense reasoning. To this end, we introduce CReM—the first commonsense reasoning evaluation benchmark tailored for NMT—comprising 1,200 triplets spanning seven categories of commonsense knowledge. We propose a dual-dimensional evaluation framework assessing both accuracy (where mainstream models achieve only 60.1%) and cross-context consistency (with merely 31% consistency), enabling the first systematic quantification of NMT’s commonsense reasoning capability. Extensive comparative experiments are conducted using models including BERT and GPT-2; statistical analysis and error attribution reveal that contextual modeling and effective commonsense integration remain critical bottlenecks. The CReM benchmark is publicly released to serve as a standardized evaluation tool for future research.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Does neural machine translation yield translations that are congenial with common sense? In this paper, we present a test suite to evaluate the commonsense reasoning capability of neural machine translation. The test suite consists of three test sets, covering lexical and contextless/contextual syntactic ambiguity that requires commonsense knowledge to resolve. We manually create 1,200 triples, each of which contain a source sentence and two contrastive translations, involving 7 different common sense types. Language models pretrained on large-scale corpora, such as BERT, GPT-2, achieve a commonsense reasoning accuracy of lower than 72% on target translations of this test suite. We conduct extensive experiments on the test suite to evaluate commonsense reasoning in neural machine translation and investigate factors that have impact on this capability. Our experiments and analyses demonstrate that neural machine translation performs poorly on commonsense reasoning of the three ambiguity types in terms of both reasoning accuracy ( 6 60.1%) and reasoning consistency (6 31%). We will release our test suite as a machine translation commonsense reasoning testbed to promote future work in this direction.

Problem

• Evaluates commonsense reasoning in neural machine translation.
• Tests lexical and syntactic ambiguity requiring commonsense knowledge.
• Assesses reasoning accuracy and consistency in translation models.

Innovation

• Developed a commonsense reasoning test suite
• Evaluated neural machine translation using BERT, GPT-2
• Analyzed impact factors on translation reasoning accuracy