When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation

2026-04-10

AI Summary

This study addresses the challenge that large language models (LLMs) struggle to adapt to rapidly evolving software library APIs due to their static knowledge, often generating code incompatible with target environments. The authors systematically investigate the impact of API evolution—including deprecations, modifications, and additions—on LLM-generated code by constructing a benchmark comprising 270 real-world updates across eight Python libraries. Evaluating 11 prominent LLMs under scenarios where contextual information conflicts with internal knowledge, they find that models persistently favor outdated API usage even when provided with update documentation. Without documentation, only 42.55% of generated code is executable on average; this rises to 66.36% with structured documentation and larger model scales. Further gains of 11% in executability are achieved by integrating retrieval-augmented generation (RAG) and Self-Reflection reasoning strategies. The work underscores the need for evaluation benchmarks and adaptive techniques tailored to API evolution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rapid evolution of software libraries creates a significant challenge for Large Language Models (LLMs), whose static parametric knowledge often becomes stale post-training. While retrieval-augmented generation (RAG) is commonly used to provide up-to-date API specifications,"context-memory conflict"arises when external instructions contradict a model's internal parametric knowledge. This paper presents a systematic empirical study of LLM code generation under API evolution (e.g., API deprecation, API modification, and API addition), by constructing a benchmark of 270 real-world updates from eight Python libraries. We evaluate four LLM families of 11 models. Our results show that without comprehensive documentation, LLMs struggle to prioritize external context, averaging only 42.55% of generated code examples are executable in the target environment. While structured documentation and larger model scales improve LLMs'ability to update adoption, they do not fully resolve executability issues with a low 66.36% executable rate. In addition, reasoning-based strategies (e.g., Self-Reflection) significantly boost LLMs'performance with 11% improvement on executable rate. Our findings highlight the persistence of outdated patterns from LLMs, even when API update specifications are provided, and emphasize the need for evolution-aware benchmarks and techniques.

Problem

• API evolution
• knowledge conflict
• code generation
• Large Language Models
• context-memory conflict

Innovation

• context-memory conflict
• API evolution
• retrieval-augmented generation
• code executability
• Self-Reflection