TRAFA: Anticipating User Actions to Reduce Errors in Procedural Tasks with Predictive Feedback

2026-05-23

AI Summary

This work proposes TRAFA, a novel interactive assistance system that introduces predictive feedback into procedural task guidance—addressing the limitation of traditional systems that only provide reactive corrections after errors occur. TRAFA implements a Track-Forecast-Act framework: it continuously tracks hand and object states in real time, leverages scene context to forecast user intent, and proactively triggers interventions before task constraints are violated. By shifting from post-hoc recovery to preemptive error prevention, TRAFA enhances both accuracy and efficiency. Experimental results in an assembly task demonstrate that TRAFA significantly improves task performance while maintaining a feedback frequency comparable to conventional reactive systems.

Abstract

Interactive assistance systems typically provide feedback after an action has been completed, supporting error recovery but not preventing the error itself. We present TRAFA, a real-time predictive feedback system for procedural tasks that intervenes before errors are committed. TRAFA operationalizes predictive feedback through a Track-Forecast-Act framework that tracks hand and object state, forecasts user motion conditioned on scene context, and triggers feedback when a predicted action is likely to violate task constraints. We instantiate this pipeline in a sequential assembly setting and evaluate it through both technical benchmarking and a controlled user study against conventional reactive feedback. Our results show that predictive feedback improves task accuracy and efficiency while maintaining a comparable number of feedback events. These findings position feedback timing as a key dimension in system design and show how real-time anticipation can be integrated into interactive systems to prevent errors before they occur.

Problem

• predictive feedback
• error prevention
• procedural tasks
• interactive assistance
• real-time anticipation

Innovation

• predictive feedback
• error prevention
• real-time anticipation
• procedural tasks
• interactive assistance