'Show It, Don't Just Say It': The Complementary Effects of Instruction Multimodality for Software Guidance

2026-03-02

AI Summary

This study investigates how to effectively select and combine multimodal instructional strategies in software teaching to balance pedagogical efficacy with learner autonomy. Through observational analysis of ten teacher–student dyads engaged in graphic design software instruction, the research examines the impact of three modalities—verbal guidance, visual annotation, and remote screen control—on learners’ cognitive engagement and sense of digital territoriality. The work introduces “precision–autonomy trade-offs” and “digital territoriality” as novel design constraints for adaptive instructional systems, extending the principles of contiguity and learner agency. Findings indicate that visual annotation enhances spatial precision, while remote control improves spatiotemporal precision; however, both modalities may inadvertently diminish learner autonomy. Instructors dynamically adjust modality combinations to optimize both teaching efficiency and the learning experience.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Designing adaptive tutoring systems for software learning presents challenges in determining appropriate instructional modalities. To inform the design of such systems, we conducted an observational study of ten human teacher-student pairs (N=10), where experienced design software users taught novices two new graphic design software features through multi-step procedures. These lessons were limited to three communication channels (speech, visual annotations, and remote screen control) to mimic possible AI tutor modalities. We found that annotations complement speech with spatial precision and remote control complements it with spatial and temporal precision, but both cause intrusion to learner agency. Teachers adaptively select modalities to balance the need for instruction progress with students' cognitive engagement and sense of digital territory ownership. Our results provide further support to the contiguity principles and the value of agency in learning, while suggesting precision-agency trade-off and digital territoriality as new design constraints for adaptive software guidance.

Problem

• instructional modality
• adaptive tutoring systems
• learner agency
• digital territoriality
• software guidance

Innovation

• instructional multimodality
• adaptive tutoring systems
• learner agency
• digital territoriality
• precision-agency trade-off