Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates
Large Language Models' success on text generation has also made them better at code generation and coding tasks. While a lot of work has demonstrated their remarkable performance on tasks such as code completion and editing, it is still unclear as to why. We help bridge this gap by exploring to what degree auto-regressive models understand the logical constructs of the underlying programs. We propose Counterfactual Analysis for Programming Concept Predicates (CACP) as a counterfactual testing framework to evaluate whether Large Code Models understand programming concepts. With only black-box access to the model, we use CACP to evaluate ten popular Large Code Models for four different programming concepts. Our findings suggest that current models lack understanding of concepts such as data flow and control flow.
Code (0)
등록된 구현이 없습니다.
Tasks
Code CompletionCode GenerationcounterfactualText GenerationSimilar Papers 제목 키워드 기반
Enhancing Programming eTextbooks with ChatGPT Generated Counterfactual-Thinking-Inspired Questions
Digital textbooks have become an integral part of everyday learning tasks. In this work, we consider the use of digital textbooks for programming classes. Generally, students struggle with utilizing textbooks on programm…
counterfactualCounterfactual Fairness Is Not Demographic Parity, and Other Observations
Blanket statements of equivalence between causal concepts and purely probabilistic concepts should be approached with care. In this short note, I examine a recent claim that counterfactual fairness is equivalent to demog…
counterfactualFairnessCrowd Sourced Data Analysis: Mapping of Programming Concepts to Syntactical Patterns
Since programming concepts do not match their syntactic representations, code search is a very tedious task. For instance in Java or C, array doesn't match [], so using "array" as a query, one cannot find what they are l…
Code SearchTAGLLM-Aided Customizable Profiling of Code Data Based On Programming Language Concepts
Data profiling is critical in machine learning for generating descriptive statistics, supporting both deeper understanding and downstream tasks like data valuation and curation. This work addresses profiling specifically…
Code GenerationData ValuationDescriptiveDiscovering Concept Directions from Diffusion-based Counterfactuals via Latent Clustering
Concept-based explanations have emerged as an effective approach within Explainable Artificial Intelligence, enabling interpretable insights by aligning model decisions with human-understandable concepts. However, existi…
ClusteringcounterfactualExplainable artificial intelligence