Finding SSH Strict Key Exchange Violations by State Learning

2025-09-13

AI Summary

This work systematically evaluates the specification compliance and security of the SSH Strict Key Exchange (Strict KEX) mechanism across mainstream server implementations. We propose the first black-box state-learning framework tailored for Strict KEX–enabled deployments, integrating protocol reverse engineering and automated modeling to analyze handshake procedures across 10 SSH implementations and 5 KEX algorithm families—inferring 33 distinct state machines. Our analysis yields the first complete reconstruction of the Strict KEX handshake state machine, uncovering seven classes of specification violations and two critical vulnerabilities: one enabling rogue session injection and another permitting remote unauthenticated code execution. These findings expose severe risks inherent in optional message paths—long overlooked by conventional testing methodologies. The study establishes a novel methodological foundation for protocol implementation security verification, supported by empirical evidence and actionable insights for secure SSH deployment.

Technology Category

• Intelligent Robots: State Estimation
• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Distributed Search

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

SSH is an important protocol for secure remote shell access to servers on the Internet. At USENIX 2024, Bäumer et al. presented the Terrapin attack on SSH, which relies on the attacker injecting optional messages during the key exchange. To mitigate this attack, SSH vendors adopted an extension developed by OpenSSH called strict key exchange ("strict KEX"). With strict KEX, optional messages are forbidden during the handshake, preventing the attack. In practice, this should simplify the state machine of an SSH handshake to a linear message flow similar to that of TLS. In this work, we analyze the design, implementation, and security of strict KEX in popular SSH servers, using black-box state learning, which can uncover the hidden state machine of an implementation. In practice, it is limited by the number of learned messages and the complexity of the state machine. Thus, learning the complete state machine of SSH is infeasible. Previous research on SSH, therefore, excluded optional messages, learning only a partial state machine. However, these messages are a critical part of the Terrapin attack. We propose to instead learn the complete state machine of the handshake phase of an SSH server, but with strict KEX enabled. We investigate the security of ten SSH implementations supporting strict KEX for up to five key exchange algorithms. In total, we learn 33 state machines, revealing significant differences in the implementations. We show that seven implementations violate the strict KEX specification and find two critical security vulnerabilities. One results in a rogue session attack in the proprietary Tectia SSH implementation. Another affects the official SSH implementation of the Erlang Open Telecom Platform, and enables unauthenticated remote code execution in the security context of the SSH server.

Problem

• Analyzing strict KEX implementation security in SSH servers
• Identifying specification violations through state learning techniques
• Discovering critical vulnerabilities enabling unauthorized remote access

Innovation

• State learning analyzes strict KEX implementations
• Tests ten SSH servers across algorithms
• Reveals specification violations and security vulnerabilities