CodeSCM: Causal Analysis for Multi-Modal Code Generation

2025-02-07

AI Summary

This work addresses the unclear causal effects of multimodal prompts—natural language instructions, code snippets, and input-output (I/O) examples—in multimodal code generation. We introduce structural causal modeling (SCM) and causal mediation analysis to this domain for the first time. By constructing an SCM with implicit semantic mediators, we disentangle the direct and indirect effects of each prompt modality on large language model (LLM) outputs. Quantitative analysis reveals that I/O examples exert a significant, often underestimated causal influence—frequently surpassing that of natural language instructions—and induce spurious correlations in model behavior. Our framework provides the first causally grounded, interpretable foundation for multimodal prompt design, enabling more controllable and robust code generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this paper, we propose CodeSCM, a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs). By applying interventions to CodeSCM, we measure the causal effects of different prompt modalities, such as natural language, code, and input-output examples, on the model. CodeSCM introduces latent mediator variables to separate the code and natural language semantics of a multi-modal code generation prompt. Using the principles of Causal Mediation Analysis on these mediators we quantify direct effects representing the model's spurious leanings. We find that, in addition to natural language instructions, input-output examples significantly influence code generation.

Problem

• Causal analysis of multi-modal code generation
• Impact of prompt modalities on LLMs
• Quantifying direct effects in code generation

Innovation

• Structural Causal Model (SCM)
• Causal Mediation Analysis
• Multi-modal code generation