CommitBART: A Large Pre-trained Model for GitHub Commits

2022-08-17 · Shangqing Liu, Yanzhou Li, Xiaofei Xie, Yang Liu

GitHub commits, which record the code changes with natural language messages for description, play a critical role for software developers to comprehend the software evolution. To promote the development of the open-source software community, we collect a commit benchmark including over 7.99 million commits across 7 programming languages. Based on this benchmark, we present CommitBART, a large pre-trained encoder-decoder Transformer model for GitHub commits. The model is pre-trained by three categories (i.e., denoising objectives, cross-modal generation and contrastive learning) for six pre-training tasks to learn commit fragment representations. Furthermore, we unify a ``commit intelligence'' framework with one understanding task and three generation tasks for commits. The comprehensive experiments on these tasks demonstrate that CommitBARTsignificantly outperforms previous pre-trained works for code. Further analysis also reveals each pre-training task enhances the model performance.

📄 PDF Abstract BibTeX arXiv:2208.08100

Code (0)

등록된 구현이 없습니다.

Tasks

Contrastive Learning Decoder Denoising

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Adam 설명 없음

AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans

2025-09-26 · Yangtian Zi, Zixuan Wu, Aleksander Boruch-Gruszecki, Jonathan Bell 외 arxiv

Fine-tuning large language models for code editing has typically relied on mining commits and pull requests. The working hypothesis has been that commit messages describe human intent in natural language, and patches to …

KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation

2024-01-16 · Wei Tao, Yucheng Zhou, Yanlin Wang, Hongyu Zhang 외

Commit messages are natural language descriptions of code changes, which are important for software evolution such as code understanding and maintenance. However, previous methods are trained on the entire dataset withou…

Denoising

MoreFixes: A Large-Scale Dataset of CVE Fix Commits Mined through Enhanced Repository Discovery

2024-07-10 · Promise 2024 7 · Jafar Akhoundali, Sajad Rahim Nouri, Kristian Rietveld, and Olga Gadyatskaya

Vulnerability datasets have become an important instrument in software security research, being used to develop automated, machine learning-based vulnerability detection and patching approaches. Yet, any limitations of t…

Vulnerability Detection

Detecting Continuous Integration Skip : A Reinforcement Learning-based Approach

2024-05-15 · Hajer Mhalla, Mohamed Aymen Saied

The software industry is experiencing a surge in the adoption of Continuous Integration (CI) practices, both in commercial and open-source environments. CI practices facilitate the seamless integration of code changes by…

Binary Classification Deep Reinforcement Learning reinforcement-learning Reinforcement Learning

VulCurator: A Vulnerability-Fixing Commit Detector

2022-09-07 · Truong Giang Nguyen, Thanh Le-Cong, Hong Jin Kang, Xuan-Bach D. Le 외

Open-source software (OSS) vulnerability management process is important nowadays, as the number of discovered OSS vulnerabilities is increasing over time. Monitoring vulnerability-fixing commits is a part of the standar…

Deep Learning Management