How could Neural Networks understand Programs?
Semantic understanding of programs is a fundamental problem for programming language processing (PLP). Recent works that learn representations of code based on pre-training techniques in NLP have pushed the frontiers in this direction. However, the semantics of PL and NL have essential differences. These being ignored, we believe it is difficult to build a model to better understand programs, by either directly applying off-the-shelf NLP pre-training techniques to the source code, or adding features to the model by the heuristic. In fact, the semantics of a program can be rigorously defined by formal semantics in PL theory. For example, the operational semantics, describes the meaning of a valid program as updating the environment (i.e., the memory address-value function) through fundamental operations, such as memory I/O and conditional branching. Inspired by this, we propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition, which is indispensable for program understanding. To validate our proposal, we present a hierarchical Transformer-based pre-training model called OSCAR to better facilitate the understanding of programs. OSCAR learns from intermediate representation (IR) and an encoded representation derived from static analysis, which are used for representing the fundamental operations and approximating the environment transitions respectively. OSCAR empirically shows the outstanding capability of program semantics understanding on many practical software engineering tasks.
Code (1)
Tasks
validMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL
Can non-programmers annotate natural language utterances with complex programs that represent their meaning? We introduce APEL, a framework in which non-programmers select among candidate programs generated by a seed sem…
Bayesian InferenceText to SQLText-To-SQLSyntax-Guided Program Reduction for Understanding Neural Code Intelligence Models
Neural code intelligence (CI) models are opaque black-boxes and offer little insight on the features they use in making predictions. This opacity may lead to distrust in their prediction and hamper their wider adoption i…
Method name predictionProbabilistic Neural-symbolic Models for Interpretable Visual Question Answering
We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formula…
counterfactualQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Position Paper: Towards Transparent Machine Learning
Transparent machine learning is introduced as an alternative form of machine learning, where both the model and the learning system are represented in source code form. The goal of this project is to enable direct human …
BIG-bench Machine LearningFormPositionCan Neural Networks Understand Programs like Humans?
Program understanding is a fundamental task in program language processing. Despite the success, existing works fail to take human minds as reference in understanding programs. In this paper, we incorporate human minds a…
Clone DetectionCode SummarizationGraph Neural Network