CodeS: Towards Code Model Generalization Under Distribution Shift
Distribution shift has been a longstanding challenge for the reliable deployment of deep learning (DL) models due to unexpected accuracy degradation. Although DL has been becoming a driving force for large-scale source code analysis in the big code era, limited progress has been made on distribution shift analysis and benchmarking for source code tasks. To fill this gap, this paper initiates to propose CodeS, a distribution shift benchmark dataset, for source code learning. Specifically, CodeS supports two programming languages (Java and Python) and five shift types (task, programmer, time-stamp, token, and concrete syntax tree). Extensive experiments based on CodeS reveal that 1) out-of-distribution detectors from other domains (e.g., computer vision) do not generalize to source code, 2) all code classification models suffer from distribution shifts, 3) representation-based shifts have a higher impact on the model than others, and 4) pre-trained bimodal models are relatively more resistant to distribution shifts.
Code (2)
Tasks
BenchmarkingCode ClassificationmodelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Graph Out-of-Distribution Generalization via Causal Intervention
Out-of-distribution (OOD) generalization has gained increasing attentions for learning on graphs, as graph neural networks (GNNs) often exhibit performance degradation with distribution shifts. The challenge is that dist…
Causal InferenceOut-of-Distribution GeneralizationCyclically Equivariant Neural Decoders for Cyclic Codes
Neural decoders were introduced as a generalization of the classic Belief Propagation (BP) decoding algorithms, where the Trellis graph in the BP algorithm is viewed as a neural network, and the weights in the Trellis gr…
DecoderTackling Long-Tailed Category Distribution Under Domain Shifts
Machine learning models fail to perform well on real-world applications when 1) the category distribution P(Y) of the training dataset suffers from long-tailed distribution and 2) the test data is drawn from different co…
Domain GeneralizationMeta-LearningSemantic SimilaritySemantic Textual SimilarityDiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shifts
Soft snake robots offer remarkable flexibility and adaptability in complex environments, yet their control remains challenging due to highly nonlinear dynamics. Existing model-based and bio-inspired controllers rely on s…
Reinforcement LearningOffline RLCharacterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning
Deep Metric Learning (DML) aims to find representations suitable for zero-shot transfer to a priori unknown test distributions. However, common evaluation protocols only test a single, fixed data split in which train and…
Metric Learning