Breaking the Plane: Exploring Real-Time Visualization of 3D Surfaces in Augmented Reality with Handwritten Input

2025-07-07

AI Summary

Existing 3D mathematical function learning tools suffer from abstract visualizations, limited interactivity, and lack of real-time formula input support in augmented reality (AR) environments. Method: This paper introduces the first AR-headset-based, real-time handwriting-driven 3D function visualization system. It integrates lightweight handwritten formula recognition, dynamic GPU-accelerated 3D surface rendering, and natural gesture interaction to enable a closed-loop “write-to-visualize, touch-to-tune, see-as-you-get” learning experience in AR space. Crucially, it couples semantic formula parsing with real-time GPU surface generation and optimizes a low-latency AR overlay rendering pipeline. Contribution/Results: User studies demonstrate significant improvements over conventional tools: +32% in conceptual understanding accuracy and +41% in task completion efficiency, while maintaining usability comparable to mainstream desktop software—92% of participants preferred this system as their primary tool.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We introduce Breaking the Plane, an augmented reality (AR) application built for AR headsets that enables users to visualize 3D mathematical functions using handwritten input. Researchers have demonstrated overlaying 3D visualizations of mathematical concepts through AR enhances learning motivation and comprehension, and equation parsing makes the authoring of teaching materials more time-efficient for instructors. Previous works have developed AR systems that separately employ equation parsing and 3D mathematical visualizations, but work has yet to be done to combine those features by enabling real-time interactions and dynamic visualizations that help users learn in situ. We explore this by developing an AR system featuring handwritten equation parsing, graph manipulation, and a 3D function plotter. We found that our system significantly surpassed other systems in engagement, achieved comparable ease of use to a popular visualization tool, was considered the most effective in aiding problem-solving, and was highly preferred by participants for future use.

Problem

• Real-time 3D surface visualization in AR
• Handwritten input for mathematical functions
• Combining equation parsing with dynamic interactions

Innovation

• AR system with handwritten equation parsing
• Real-time 3D function visualization
• Interactive graph manipulation in AR