Comprehension vs. Adoption: Evaluating a Language Workbench Through a Family of Experiments

2026-01-28

AI Summary

This study addresses a critical gap in language workbench evaluation by jointly examining metalinguistic understandability and user acceptance—factors often overlooked in existing assessments. Building upon an enhanced Model of Evaluation for Metalinguistic constructs (MEM), the authors conduct three rounds of experiments combining quantitative surveys and task-based analyses to systematically evaluate the Neverlang language workbench. Findings indicate that while users generally comprehend Neverlang’s syntax and perceive it as useful, they encounter notable challenges regarding ease of use. Crucially, the study reveals no significant correlation between metalinguistic understandability and user acceptance, thereby challenging the common assumption that “understandable implies usable.” This underscores the pivotal roles of perceived ease of use and perceived usefulness in technology adoption, suggesting that comprehensibility alone is insufficient to ensure user acceptance of language engineering tools.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: User Experience and Usability
• Machine Learning: Evaluation and Analysis

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Language workbenches are tools that enable the definition, reuse, and composition of programming languages and their ecosystems, aiming to streamline language development. To facilitate their adoption by language designers, the comprehensibility of the language used to define other languages is an important aspect to evaluate. Moreover, considering that language workbenches are relatively new tools, user acceptance emerges as a crucial factor to be accounted for during their assessment. Current literature often neglects user-centred aspects like comprehensibility and acceptance in the assessment of this breed of tools. This paper addresses this gap through a family of experiments assessing Neverlang, a modular language workbench. The study adopts a tailored version of the Method Evaluation Model (MEM) to evaluate the comprehensibility of Neverlang's meta-language and programs, as well as user acceptance in terms of perceived ease of use, perceived usefulness, and intention to use. It also investigates the relationships among these dimensions. The experiments were conducted in three iterations involving participants from academia. The results reveal that users demonstrate sufficient comprehension of Neverlang's meta-language, particularly concerning its syntax, express a favourable perception of its usefulness, and indicate their intention to use it. However, the results also indicate that Neverlang's ease of use remains a challenge. Additionally, variations in the perceived ease of use and perceived usefulness, whether low or high, influence the users'intention to use the tool. Surprisingly, no significant correlation is found between comprehensibility and user acceptance. Notably, higher comprehensibility of the meta-language does not necessarily translate into greater acceptance, underscoring the complex interplay between comprehension and adoption.

Problem

• language workbench
• comprehensibility
• user acceptance
• meta-language
• tool adoption

Innovation

• language workbench
• comprehensibility
• user acceptance
• Method Evaluation Model
• empirical study