🤖 AI Summary
This study addresses the critical issue of irreversible classification errors in manual data entry by small and medium-sized enterprises (SMEs), where semantically or morphologically similar categories often distort key performance indicators and mislead decision-making. To mitigate this, the authors propose ISEC, a novel preventive ordinal metric that integrates semantic embeddings, weighted Damerau-Levenshtein edit costs, and empirical category frequencies into a scalable, confusion-aware ranking framework. By leveraging a vector database for efficient similarity search, the method achieves substantial computational gains. Empirical validation across three heterogeneous datasets—legal records, retail inventory, and metalworking catalogs—demonstrates a 195-fold speedup over brute-force computation while maintaining high accuracy, offering SMEs a practical and efficient tool for robust data governance.
📝 Abstract
Data entry systems remain structurally vulnerable to categorical misclassifications, particularly in small and medium sized enterprises (SMEs). When nominal categories exhibit semantic or morphological proximity, human machine interaction may produce errors that are irrecoverable ex post. In the absence of automated input controls, manual data entry frequently generates irrecoverable categorical distortions that propagate into Key Performance Indicators (KPIs), thereby misleading managerial decision making. State of the art normalization tools typically evaluate semantic and morphological dimensions in isolation and rely heavily on standard dictionaries, rendering them ineffective for SME master data rich in custom SKUs, abbreviations, and domain-specific technical jargon. This paper introduces the Categorical Error Sensitivity Index (ISEC), an ordinal composite score designed to rank category pairs according to their structural susceptibility to confusion. ISEC integrates semantic distance (via word embeddings), custom weighted morphological transformation costs (through an adapted Damerau Levenshtein algorithm), and empirical frequency into a unified, mathematically robust preventive framework. By leveraging vector database architectures, ISEC reduces computational complexity, achieving approximately a 195x performance improvement over brute-force methods. Validated across three heterogeneous datasets: governmental judicial records, retail inventory, and a synthetic ISO coded metalworking catalog, ISEC provides a scalable and proactive data governance instrument that enables SMEs to detect latent structural risk embedded within their categorical data assets.