Your Spending Needs Attention: Modeling Financial Habits with Transformers

2025-07-31

AI Summary

To address the challenges of high-dimensional heterogeneity and costly manual feature engineering in financial user behavior data (e.g., transaction logs, app events, customer service records), this paper proposes nuFormer—the first end-to-end user representation learning framework that jointly encodes merchant textual features (e.g., business names) and structured fields (e.g., amount, timestamp) into a Transformer architecture. Leveraging self-supervised pretraining on raw transaction sequences, nuFormer jointly optimizes user embeddings and conventional handcrafted features without requiring auxiliary data sources. Empirical evaluation on Nubank’s large-scale recommendation system demonstrates substantial improvements in core metrics—including CTR and Recall—while also exhibiting strong generalization performance in risk prediction and fraud detection tasks. These results validate nuFormer’s industrial-grade effectiveness, scalability, and methodological novelty.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Data Mining & Knowledge Management: Recommender Systems
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Predictive models play a crucial role in the financial industry, enabling risk prediction, fraud detection, and personalized recommendations, where slight changes in core model performance can result in billions of dollars in revenue or losses. While financial institutions have access to enormous amounts of user data (e.g., bank transactions, in-app events, and customer support logs), leveraging this data effectively remains challenging due to its complexity and scale. Thus, in many financial institutions, most production models follow traditional machine learning (ML) approaches by converting unstructured data into manually engineered tabular features. Conversely, other domains (e.g., natural language processing) have effectively utilized self-supervised learning (SSL) to learn rich representations from raw data, removing the need for manual feature extraction. In this paper, we investigate using transformer-based representation learning models for transaction data, hypothesizing that these models, trained on massive data, can provide a novel and powerful approach to understanding customer behavior. We propose a new method enabling the use of SSL with transaction data by adapting transformer-based models to handle both textual and structured attributes. Our approach, denoted nuFormer, includes an end-to-end fine-tuning method that integrates user embeddings with existing tabular features. Our experiments demonstrate improvements for large-scale recommendation problems at Nubank. Notably, these gains are achieved solely through enhanced representation learning rather than incorporating new data sources.

Problem

• Modeling financial habits using transformer-based representation learning
• Leveraging transaction data for self-supervised learning without manual features
• Improving financial recommendations via enhanced user behavior understanding

Innovation

• Transformer-based models for transaction data
• Self-supervised learning without manual features
• End-to-end fine-tuning with user embeddings