User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs

2025-06-30

AI Summary

To address bias in LLM generalization evaluation caused by data contamination, this work proposes user behavior prediction—specifically, choice modeling in domains such as movie or music recommendation—as a novel, theoretically grounded, scalable, and task-agnostic evaluation paradigm. Unlike conventional benchmarks, our framework avoids data reuse risks, enabling low-cost, large-scale measurement of true model generalization. This is the first systematic integration of user behavior modeling into LLM evaluation, ensuring both robustness and theoretical rigor. Experiments on GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct demonstrate clear performance stratification (with GPT-4o achieving highest accuracy) and consistent room for improvement across all models, validating the framework’s effectiveness, sensitivity, and practical utility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evaluation and Analysis

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Measuring the generalization ability of Large Language Models (LLMs) is challenging due to data contamination. As models grow and computation becomes cheaper, ensuring tasks and test cases are unseen during training phases will become nearly impossible. We argue that knowledge-retrieval and reasoning tasks are not ideal for measuring generalization, as LLMs are not trained for specific tasks. Instead, we propose user behavior prediction, also a key aspect of personalization, as a theoretically sound, scalable, and robust alternative. We introduce a novel framework for this approach and test it on movie and music recommendation datasets for GPT-4o, GPT-4o-mini, and Llama-3.1-8B-Instruct. Results align with our framework's predictions, showing GPT-4o outperforms GPT-4o-mini and Llama, though all models have much room for improvement, especially Llama.

Problem

• Measure LLM generalization without data contamination
• Propose user behavior prediction for robust evaluation
• Test framework on recommendation datasets for LLMs

Innovation

• Proposes user behavior prediction for LLM evaluation
• Introduces scalable framework for generalization assessment
• Tests framework on movie and music datasets