SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training

2026-06-19

AI Summary

This work addresses the challenges of heterogeneous event stream prediction, where diverse event types and complex temporal dynamics hinder effective modeling. To this end, we propose SOHET, a novel architecture that uniquely integrates event-type-specific tabular encoders with time-type embeddings and leverages either causal or bidirectional Transformers to capture sequential dynamics. We further introduce three self-supervised pretraining objectives tailored for the causal setting, which substantially enhance model generalization and convergence speed. Evaluated on a large-scale fraud detection task from Booking.com, SOHET outperforms existing methods by 5.8%, with pretraining yielding an additional 2.6% performance gain and a 2.4× acceleration in convergence. On the EBES benchmark, SOHET achieves state-of-the-art or competitive results on six out of eight tasks.

Abstract

Many machine learning applications rely on heterogeneous event streams to make predictions, either causally as events arrive or bidirectionally over complete sequences. We propose SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical architecture combining event-type-specific tabular encoders with temporal and type embeddings, processed by a causal or bidirectional transformer. We introduce three self-supervised pre-training objectives for the causal setting. On a proprietary large-scale real-world Booking.com fraud detection task with 17 event types, SOHET outperforms FlexTPP, NAPPT, and CIPPT by 5.8%. Pre-training yields an additional 2.6% gain and 2.4% faster convergence. On the EBES benchmark, bidirectional SOHET matches or exceeds the published best on 6 out of 8 tasks.

Problem

• heterogeneous event streams
• event sequence modeling
• fraud detection
• temporal prediction
• multi-type events

Innovation

• Heterogeneous Event Streams
• Transformer Architecture
• Self-Supervised Pre-Training
• Tabular Encoders
• Temporal Embeddings