Zero-Shot Commonsense Validation and Reasoning with Large Language Models: An Evaluation on SemEval-2020 Task 4 Dataset

2025-02-19

AI Summary

This study systematically evaluates large language models (LLMs) on SemEval-2020 Task 4—zero-shot commonsense validation (Task A) and explanation generation (Task B). Using zero-shot prompting, we benchmark LLaMA3-70B, Gemma2-9B, and Mixtral-8x7B under the official evaluation protocol. Results reveal that LLaMA3-70B achieves 98.40% accuracy on Task A—near human-level performance—yet attains only 93.40% on Task B, substantially underperforming fine-tuned models and exposing critical bottlenecks in causal reasoning and explanation selection. Our contributions are two-fold: (1) We introduce the first open-source, systematic zero-shot evaluation framework for commonsense reasoning in LLMs; and (2) we empirically identify a structural limitation in current LLMs’ generative explanation capabilities—particularly in causal inference—providing key evidence to guide future modeling of causal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study evaluates the performance of Large Language Models (LLMs) on SemEval-2020 Task 4 dataset, focusing on commonsense validation and explanation. Our methodology involves evaluating multiple LLMs, including LLaMA3-70B, Gemma2-9B, and Mixtral-8x7B, using zero-shot prompting techniques. The models are tested on two tasks: Task A (Commonsense Validation), where models determine whether a statement aligns with commonsense knowledge, and Task B (Commonsense Explanation), where models identify the reasoning behind implausible statements. Performance is assessed based on accuracy, and results are compared to fine-tuned transformer-based models. The results indicate that larger models outperform previous models and perform closely to human evaluation for Task A, with LLaMA3-70B achieving the highest accuracy of 98.40% in Task A whereas, lagging behind previous models with 93.40% in Task B. However, while models effectively identify implausible statements, they face challenges in selecting the most relevant explanation, highlighting limitations in causal and inferential reasoning.

Problem

• Evaluate LLMs on commonsense validation and explanation
• Compare zero-shot LLMs with fine-tuned transformer models
• Assess model performance in causal and inferential reasoning

Innovation

• Zero-shot prompting techniques
• Large Language Models evaluation
• Commonsense validation and explanation