Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M
This work systematically investigates, for the first time, whether large language models (LLMs) memorize publicly available recommendation datasets (e.g., MovieLens-1M) during pretraining—and how such memorization affects recommendation performance and bias. We propose a prompt-engineering-based memory detection framework and conduct structured retrieval experiments across multiple GPT and Llama model sizes to quantify the recoverability of user profiles, item attributes, and interaction histories. Results show that all tested models exhibit non-negligible memorization; memory strength positively correlates with zero-shot recommendation accuracy but simultaneously exacerbates popularity bias. Moreover, memorization increases with model scale and exhibits architecture-dependent patterns. This study establishes a novel, trustworthy evaluation paradigm for LLMs in recommendation settings. The code is publicly released.