In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

2026-03-26

AI Summary

It remains unclear whether large language models (LLMs) genuinely perform in-context learning for molecular property prediction or merely rely on memorization from pretraining data. This work proposes a progressive information masking framework to systematically evaluate the predictive behavior of nine prominent LLMs—including GPT-4.1, GPT-5, and Gemini 2.5—across 0-shot, 60-shot, and 1000-shot settings on three MoleculeNet datasets. The experiments reveal, for the first time, that most models heavily depend on memorized knowledge rather than generalizable reasoning; their performance degrades significantly when pretrained knowledge conflicts with provided context, exposing substantial data contamination risks in current benchmarks. This study establishes a new paradigm for rigorously assessing the true in-context learning capabilities of LLMs in scientific tasks.

Abstract

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or rely primarily on memorized values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information. Complementing this, we utilize varying in-context sample sizes (0-, 60-, and 1000-shot) as an additional control for information access. This work provides a principled framework for evaluating molecular property prediction under controlled information access, addressing concerns regarding memorization and exposing conflicts between pre-trained knowledge and in-context information.

Problem

• in-context learning
• molecular property prediction
• memorization
• knowledge conflict
• large language models

Innovation

• in-context learning
• blinding study
• molecular property prediction
• memorization
• knowledge conflict