Secure Development of a Hooking-Based Deception Framework Against Keylogging Techniques

2025-08-06

AI Summary

Advanced keyloggers commonly evade conventional defenses via anti-hooking techniques. To address this, we propose an active deception framework that intercepts input APIs through an enhanced API hooking layer and injects dynamically generated, realistic decoy keystrokes at runtime to mislead attackers. Our approach innovatively integrates anti-hooking behavior detection with a self-healing mechanism to ensure robust, persistent hooking under adversarial interference. Furthermore, it employs lightweight injection and stealthy obfuscation strategies to achieve minimal overhead—rendering the defense imperceptible to users—while maximizing deception success rates. Evaluated against a custom-built “super keylogger” and 50 real-world keylogger samples, our framework demonstrates strong resilience against diverse evasion attempts, significantly enhancing robustness against sophisticated anti-detection techniques.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Responsible Web: Social and technical mechanisms of refusal for web technologies and applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Keyloggers remain a serious threat in modern cybersecurity, silently capturing user keystrokes to steal credentials and sensitive information. Traditional defenses focus mainly on detection and removal, which can halt malicious activity but do little to engage or mislead adversaries. In this paper, we present a deception framework that leverages API hooking to intercept input-related API calls invoked by keyloggers at runtime and inject realistic decoy keystrokes. A core challenge, however, lies in the increasing adoption of anti-hooking techniques by advanced keyloggers. Anti-hooking strategies allow malware to bypass or detect instrumentation. To counter this, we introduce a hardened hooking layer that detects tampering and rapidly reinstates disrupted hooks, ensuring continuity of deception. We evaluate our framework against a custom-built "super keylogger" incorporating multiple evasion strategies, as well as 50 real-world malware samples spanning ten prominent keylogger families. Experimental results demonstrate that our system successfully resists sophisticated bypass attempts, maintains operational stealth, and reliably deceives attackers by feeding them decoys. The system operates with negligible performance overhead and no observable impact on user experience. Our findings show that resilient, runtime deception can play a practical and robust role in confronting advanced threats.

Problem

• Counter keyloggers with decoy keystrokes via API hooking
• Overcome anti-hooking techniques in advanced keyloggers
• Ensure deception continuity against evasion strategies

Innovation

• API hooking intercepts keylogger API calls
• Hardened hooking layer resists anti-hooking
• Decoy keystrokes deceive attackers effectively