Budget-Aware Keyboardless Interaction

2026-06-24

AI Summary

This work proposes a low-cost virtual keyboard system that overcomes the mobility constraints of traditional input devices and eliminates the need for complex calibration or specialized hardware found in existing solutions. By leveraging only an off-the-shelf camera and a printed sheet depicting a standard keyboard layout, the system operates under normal lighting conditions without requiring any calibration. It integrates modern image segmentation and object detection models with classical image processing techniques, and introduces a novel approach to keystroke detection by exploiting subtle color changes in the user’s fingernails upon key contact. Experimental results demonstrate the system’s effectiveness in accurately localizing the keyboard region and recognizing pressed keys. User studies further confirm its usability and practical value, significantly enhancing accessibility and ease of deployment for physical-keyboard-free interaction.

Abstract

Interacting with computers typically relies on traditional input devices such as keyboards, mice, and monitors, which can be cumbersome for users seeking greater mobility. Virtual keyboards have been explored to address these limitations, but they often involve complex setups or expensive equipment. This paper proposes a novel virtual keyboard system that leverages only a standard camera and a paper with a printed keyboard layout. Unlike previous methods requiring complex calibration or special lighting conditions, our approach can work on standard environment using modern computer vision technologies. Combining modern segmentation and detection models with traditional image processing algorithms, we efficiently identify the keyboard region. Touch detection is performed using an algorithm analyzing the color of the user's fingernail. Experiments demonstrated a promising results our proposed solution of keyboard and keystroke detection for practical applications. Participants attended our user study also found the proposed system interesting.

Problem

• virtual keyboard
• keyboardless interaction
• budget-aware
• computer vision
• touch detection

Innovation

• virtual keyboard
• camera-based interaction
• nail color analysis
• computer vision
• budget-aware HCI