A Survey on Pretrained Language Models for Neural Code Intelligence

2022-12-20 · Yichen Xu, Yanqiao Zhu

As the complexity of modern software continues to escalate, software engineering has become an increasingly daunting and error-prone endeavor. In recent years, the field of Neural Code Intelligence (NCI) has emerged as a promising solution, leveraging the power of deep learning techniques to tackle analytical tasks on source code with the goal of improving programming efficiency and minimizing human errors within the software industry. Pretrained language models have become a dominant force in NCI research, consistently delivering state-of-the-art results across a wide range of tasks, including code summarization, generation, and translation. In this paper, we present a comprehensive survey of the NCI domain, including a thorough review of pretraining techniques, tasks, datasets, and model architectures. We hope this paper will serve as a bridge between the natural language and programming language communities, offering insights for future research in this rapidly evolving field.

📄 PDF Abstract BibTeX arXiv:2212.10079

Code (0)

등록된 구현이 없습니다.

Tasks

Code SummarizationSurvey

A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

2022-02-10 · Erfan Al-Hossami, Samira Shaikh

In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI…

Survey

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

2023-02-18 · Ce Zhou, Qian Li, Chen Li, Jun Yu 외

Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A PFM (e.g., BERT, ChatGPT, and GPT-4) is trained on large-scale data which provides a reaso…

Graph Learning Language Modelling Logical Reasoning

Large Language Models for Multilingual Code Intelligence: A Survey

2026-04-27 · Chao Jiang, Dugang Liu, Cheng Wen, Zhiwu Xu 외 arxiv

Large language models have transformed AI-assisted software engineering, but current research remains biased toward high-resource languages such as Python, with weaker performance in languages like Rust and OCaml. Since …

Code Translation Code Generation

AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models

2021-04-16 · Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha

Transformer-based pretrained language models (PLMs) have started a new era in modern natural language processing (NLP). These models combine the power of transformers, transfer learning, and self-supervised learning (SSL…

Self-Supervised Learning Survey Transfer Learning

A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond

2024-03-21 · Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng 외

Neural Code Intelligence -- leveraging deep learning to understand, generate, and optimize code -- holds immense potential for transformative impacts on the whole society. Bridging the gap between Natural Language and Pr…