CodeSCM: Causal Analysis for Multi-Modal Code Generation
In this paper, we propose CodeSCM, a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs). By applying interventions to CodeSCM, we measure the causal effects of different prompt modalities, such as natural language, code, and input-output examples, on the model. CodeSCM introduces latent mediator variables to separate the code and natural language semantics of a multi-modal code generation prompt. Using the principles of Causal Mediation Analysis on these mediators we quantify direct effects representing the model's spurious leanings. We find that, in addition to natural language instructions, input-output examples significantly influence code generation.
Code (1)
Tasks
Code GenerationSimilar Papers 제목 키워드 기반
Multimodal Sentiment Analysis Based on Causal Reasoning
With the rapid development of multimedia, the shift from unimodal textual sentiment analysis to multimodal image-text sentiment analysis has obtained academic and industrial attention in recent years. However, multimodal…
Causal InferencecounterfactualCounterfactual InferenceMultimodal Sentiment Analysis+2MCAM: Multimodal Causal Analysis Model for Ego-Vehicle-Level Driving Video Understanding
Accurate driving behavior recognition and reasoning are critical for autonomous driving video understanding. However, existing methods often tend to dig out the shallow causal, fail to address spurious correlations acros…
Autonomous DrivingVideo UnderstandingLyS at SemEval-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing
This paper describes our participation in SemEval 2024 Task 3, which focused on Multimodal Emotion Cause Analysis in Conversations. We developed an early prototype for an end-to-end system that uses graph-based methods f…
DecoderDependency ParsingvalidCAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation
Multimodal classifiers function as opaque black box models. While several techniques exist to interpret their predictions, very few of them are as intuitive and accessible as natural language explanations (NLEs). To buil…
Decision MakingA Causal-based Framework for Multimodal Multivariate Time Series Validation Enhanced by Unsupervised Deep Learning as an Enabler for Industry 4.0
An advanced conceptual validation framework for multimodal multivariate time series defines a multi-level contextual anomaly detection ranging from an univariate context definition, to a multimodal abstract context repre…
Anomaly DetectionCausal DiscoveryContextual Anomaly DetectionRepresentation Learning+2