Behavioural Effects of Agentic Messaging: A Case Study on a Financial Service Application

2025-12-19

AI Summary

This study investigates the impact of agent-based personalized messaging on user behavior and retention in financial services, focusing on the 2025 national tax filing period. We conducted a two-month randomized controlled trial (RCT), integrating an information-retrieval–driven agent architecture, causal behavioral analysis, and temporal conversion modeling to rigorously assess effects on opt-out rates and early-filing behavior. Results demonstrate, for the first time, that agent-based personalization simultaneously reduces user churn and enhances regulatory compliance timeliness: opt-out rates decreased significantly by 21% (±0.01) relative to a rule-based baseline, while early-filing activity surged in the weeks preceding peak filing season. Our key contribution lies in empirically uncovering a positive moderating mechanism—user-level adaptive decision-making—that strengthens long-term retention. This work establishes both empirical grounding and a methodological framework for interpretable, scalable personalized interventions in financial service contexts.

Technology Category

• Multiagent Systems: Mechanism Design
• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Marketing and product personalisation provide a prominent and visible use-case for the application of Information Retrieval methods across several business domains. Recently, agentic approaches to these problems have been gaining traction. This work evaluates the behavioural and retention effects of agentic personalisation on a financial service application's customer communication system during a 2025 national tax filing period. Through a two month-long randomised controlled trial, we compare an agentic messaging approach against a business-as-usual (BAU) rule-based campaign system, focusing on two primary outcomes: unsubscribe behaviour and conversion timing. Empirical results show that agent-led messaging reduced unsubscribe events by 21% ($pm 0.01$) relative to BAU and increased early filing behaviour in the weeks preceding the national deadline. These findings demonstrate how adaptive, user-level decision-making systems can modulate engagement intensity whilst improving long-term retention indicators.

Problem

• Evaluates agentic messaging impact on customer retention
• Compares agentic vs rule-based systems for user engagement
• Measures unsubscribe reduction and early conversion timing

Innovation

• Agentic messaging reduces unsubscribe rates by 21%
• Agentic approach improves early tax filing behavior
• Adaptive user-level decision-making enhances long-term retention