On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations
With the prevalence of publicly available source code repositories to train deep neural network models, neural program models can do well in source code analysis tasks such as predicting method names in given programs that cannot be easily done by traditional program analysis techniques. Although such neural program models have been tested on various existing datasets, the extent to which they generalize to unforeseen source code is largely unknown. Since it is very challenging to test neural program models on all unforeseen programs, in this paper, we propose to evaluate the generalizability of neural program models with respect to semantic-preserving transformations: a generalizable neural program model should perform equally well on programs that are of the same semantics but of different lexical appearances and syntactical structures. We compare the results of various neural program models for the method name prediction task on programs before and after automated semantic-preserving transformations. We use three Java datasets of different sizes and three state-of-the-art neural network models for code, namely code2vec, code2seq, and GGNN, to build nine such neural program models for evaluation. Our results show that even with small semantically preserving changes to the programs, these neural program models often fail to generalize their performance. Our results also suggest that neural program models based on data and control dependencies in programs generalize better than neural program models based only on abstract syntax trees. On the positive side, we observe that as the size of the training dataset grows and diversifies the generalizability of correct predictions produced by the neural program models can be improved too. Our results on the generalizability of neural program models provide insights to measure their limitations and provide a stepping stone for their improvement.
Code (1)
Tasks
Method name predictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations
The abundance of publicly available source code repositories, in conjunction with the advances in neural networks, has enabled data-driven approaches to program analysis. These approaches, called neural program analyzers…
Method name predictionEvaluating the Generalizability of LLMs in Automated Program Repair
LLM-based automated program repair methods have attracted significant attention for their state-of-the-art performance. However, they were primarily evaluated on a few well known datasets like Defects4J, raising question…
Program RepairPrompt EngineeringGALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis
Despite achieving superior performance in human-level control problems, unlike humans, deep reinforcement learning (DRL) lacks high-order intelligence (e.g., logic deduction and reuse), thus it behaves ineffectively than…
Decision MakingDeep Reinforcement LearningProgram Synthesisreinforcement-learning+2Making Neural Programming Architectures Generalize via Recursion
Empirically, neural networks that attempt to learn programs from data have exhibited poor generalizability. Moreover, it has traditionally been difficult to reason about the behavior of these models beyond a certain leve…
On the Relation between Weak Completion Semantics and Answer Set Semantics
The Weak Completion Semantics (WCS) is a computational cognitive theory that has shown to be successful in modeling episodes of human reasoning. As the WCS is a recently developed logic programming approach, this paper i…
RelationTranslation