Canary in the Mine: An LLM Augmented Survey of Disciplinary Complaints to the Ordre des ingénieurs du Québec (OIQ)

2025-06-24

AI Summary

This study addresses systemic deficiencies in ethics and professional formation within engineering education by analyzing disciplinary complaint records (2010–2024) from the Quebec Order of Engineers (OIQ). It introduces the first application of pre-trained large language models (LLMs) for thematic modeling of engineering ethics incidents, augmented by interdisciplinary human validation to ensure interpretability and rigor in mining large-scale unstructured textual data. The analysis identifies prevalent violations—including negligence of duty, conflicts of interest, and ineffective communication—and uncovers root causes: curricular fragmentation, lack of structured reflective practice, and weak assessment mechanisms for professional competencies. Based on these empirically grounded insights, the study proposes three evidence-informed interventions: (1) a restructured, ethics-embedded curriculum; (2) enhanced context-rich professional training; and (3) a holistic, process-oriented framework for assessing professional competence. The work contributes both methodological innovation—bridging LLM-based NLP with engineering ethics research—and actionable pathways for reforming engineering pedagogy and accreditation practices.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Provenance, trust, security and privacy, and ethical issues in managing semantic data
• Responsible Web: Fairness, accountability, transparency and ethics of web technologies

Abstract

This study uses pre-trained LLMs to conduct thematic analysis to investigate disciplinary incidents involving engineers in Quebec, shedding light on critical gaps in engineering education. Through a comprehensive review of the disciplinary register of the Ordre des ingénieurs du Québec (OIQ)'s disciplinary register for 2010 to 2024, researchers from engineering education and human resources management in technological development laboratories conducted a thematic analysis of reported incidents to identify patterns, trends, and areas for improvement. The analysis aims to uncover the most common types of disciplinary incidents, underlying causes, and implications for the field in how engineering education addresses (or fails to address) these issues. Our findings identify recurring themes, analyze root causes, and offer recommendations for engineering educators and students to mitigate similar incidents. This research has implications for informing curriculum development, professional development, and performance evaluation, ultimately fostering a culture of professionalism and ethical responsibility in engineering. By providing empirical evidence of disciplinary incidents and their causes, this study contributes to evidence-based practices for engineering education and professional development, enhancing the engineering education community's understanding of professionalism and ethics.

Problem

• Identify common disciplinary incidents among Quebec engineers
• Analyze root causes of ethical lapses in engineering
• Improve engineering education to address professionalism gaps

Innovation

• Uses pre-trained LLMs for thematic analysis
• Analyzes disciplinary incidents from 2010-2024
• Provides recommendations for engineering education