ARES: A Platform for Adaptive Role-Based Evaluation of Social Engineering Risks in Human--AI Games

2026-06-16

AI Summary

This work addresses the adaptive social engineering risks posed by large language models (LLMs) in social decision-making by developing a controlled social gaming platform that supports human-human, human-AI, and AI-AI interactions. Leveraging role-conditioned LLM agents, psychology-informed participant profiling, and synchronized multimodal biosensor data—including video, eye-tracking, and smartwatch recordings—the platform integrates configurable game templates with deep learning–based feature extraction to enable, for the first time, fine-grained auditing of social engineering vulnerabilities. The project releases a 340GB open pilot dataset comprising interaction logs, psychological profiles, subjective feedback, and derived behavioral features, offering a critical empirical foundation and resource for the development of secure AI systems.

Abstract

This work introduces ARES, a platform and open pilot dataset for auditing adaptive social engineering risks in LLM-mediated social decision-making through controlled social games. ARES supports human--human, human--AI, and AI--AI settings, combining configurable game templates, role-conditioned LLM agents, psychology-informed participant profiling, structured interaction trees, and synchronised behavioural and biometric acquisition, filtering, and deep-learning-based feature extraction. The pilot dataset was collected from 15 participants interacting with a role-conditioned GPT-5.4 agent in two concatenated games: an adapted Prisoner's Dilemma and an Ultimatum Game. It comprises 340 GB of raw and processed multimodal data across six streams: interaction logs, video, screen recordings, gaze logs, smartwatch signals, and game/questionnaire metadata. These data include interaction paths, written justifications, psychological profiles, subjective feedback, perceived counterpart identity, game outcomes, and derived behavioural, facial, and gaze features. Alongside the dataset, we provide descriptive analyses to characterise the pilot release. Rigorous risk evaluation is essential for the deployment of secure AI systems, as it enables the identification and mitigation of vulnerabilities, ensures the protection of sensitive data, and supports compliance with evolving regulatory and ethical standards in society.

Problem

• social engineering risks
• human-AI interaction
• adaptive evaluation
• LLM-mediated decision-making
• behavioral auditing

Innovation

• adaptive role-based evaluation
• social engineering risks
• human-AI games
• multimodal behavioral data
• role-conditioned LLM agents