Towards Automated Identification of Violation Symptoms of Architecture Erosion

2023-06-14

AI Summary

Early detection of architectural erosion remains challenging due to the unstructured nature of code review discussions, where architectural violations are often implicitly described in natural language. Method: This study proposes an automated approach to identify symptoms of architectural violations from code review texts, leveraging a multi-model framework comprising 19 classifiers—including SVM, LSTM, and CNN—integrated with pre-trained word embeddings (word2vec, fastText, GloVe). A majority-voting ensemble strategy is introduced, and empirical analysis identifies 200-dimensional word2vec as the optimal representation. Contribution/Results: The SVM+word2vec configuration achieves an F1-score of 0.779; fastText demonstrates superior robustness across datasets; and the ensemble model further improves performance. Empirical validation via developer surveys and semi-structured interviews confirms the method’s practical interpretability and efficacy for early architectural erosion warning in real-world open-source projects (OpenStack, Qt).

Abstract

Architecture erosion has a detrimental effect on maintenance and evolution, as the implementation drifts away from the intended architecture. To prevent this, development teams need to understand early enough the symptoms of erosion, and particularly violations of the intended architecture. One way to achieve this, is through the automated identification of architecture violations from textual artifacts, and particularly code reviews. In this paper, we developed 15 machine learning-based and 4 deep learning-based classifiers with three pre-trained word embeddings to identify violation symptoms of architecture erosion from developer discussions in code reviews. Specifically, we looked at code review comments from four large open-source projects from the OpenStack (Nova and Neutron) and Qt (Qt Base and Qt Creator) communities. We then conducted a survey and semi-structured interviews to acquire feedback from the involved participants who discussed architecture violations in code reviews, to validate the usefulness of our trained classifiers. The results show that the SVM classifier based on word2vec pre-trained word embedding performs the best with an F1-score of 0.779. In most cases, classifiers with the fastText pre-trained word embedding model can achieve relatively good performance. Furthermore, 200-dimensional pre-trained word embedding models outperform classifiers that use 100 and 300-dimensional models. In addition, an ensemble classifier based on the majority voting strategy can further enhance the classifier and outperforms the individual classifiers. Finally, the findings derived from the online survey and interviews conducted with the involved developers reveal that the violation symptoms identified by our approaches have practical value and can provide early warnings for impending architecture erosion.

Problem

• Automated identification of architecture erosion symptoms
• Detecting architectural violations from code reviews
• Comparing ML/DL and LLM methods for violation detection

Innovation

• Machine learning classifiers for violation detection
• Word embeddings like word2vec and fastText
• LLM-based classifiers outperforming traditional methods