SCELMo: Source Code Embeddings from Language Models

2020-04-28 · Rafael - Michael Karampatsis, Charles Sutton

Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Contextual embeddings are common in natural language processing but have not been previously applied in software engineering. We introduce a new set of deep contextualized word representations for computer programs based on language models. We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018). We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection. We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.

📄 PDF Abstract BibTeX arXiv:2004.13214

Code (0)

등록된 구현이 없습니다.

Tasks

Code Search Program Repair

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

BiLSTM A Bidirectional LSTM, or biLSTM, is a sequence processing model that consists of two LSTMs: one taking the input in a forward direction, and the other in a backwards…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

ELMo Embeddings from Language Models, or ELMo, is a type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and…

Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora

2020-10-27 · EMNLP (MRL) 2021 11 · Takashi Wada, Tomoharu Iwata, Yuji Matsumoto, Timothy Baldwin 외

We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus (e.g. a few hundred sentence pairs). Our method obtains word embeddings via an LSTM encoder-decoder mod…

Bilingual Lexicon Induction Cross-Lingual Word Embeddings Decoder Sentence+3

GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge

2024-09-26 · Daniil Gurgurov, Rishu Kumar, Simon Ostermann

Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Using LLMs for such languages is often difficult due…

Natural Language Inference Sentiment Analysis Topic Classification Word Embeddings+1

A Literature Study of Embeddings on Source Code

2019-04-05 · Zimin Chen, Martin Monperrus

Natural language processing has improved tremendously after the success of word embedding techniques such as word2vec. Recently, the same idea has been applied on source code with encouraging results. In this survey, we …

Articles Survey

Word-level Language Identification Using Subword Embeddings for Code-mixed Bangla-English Social Media Data

2022-06-01 · DCLRL (LREC) 2022 6 · Aparna Dutta

This paper reports work on building a word-level language identification (LID) model for code-mixed Bangla-English social media data using subword embeddings, with an ultimate goal of using this LID module as the first s…

Language Identification POS

Code-switching Language Modeling With Bilingual Word Embeddings: A Case Study for Egyptian Arabic-English

2019-09-24 · Injy Hamed, Moritz Zhu, Mohamed Elmahdy, Slim Abdennadher 외

Code-switching (CS) is a widespread phenomenon among bilingual and multilingual societies. The lack of CS resources hinders the performance of many NLP tasks. In this work, we explore the potential use of bilingual word …

Language Modeling Language Modelling Word Embeddings