Which *BERT? A Survey Organizing Contextualized Encoders
Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSurveySimilar Papers 제목 키워드 기반
Topic Modeling with Contextualized Word Representation Clusters
Clustering token-level contextualized word representations produces output that shares many similarities with topic models for English text collections. Unlike clusterings of vocabulary-level word embeddings, the resulti…
ClusteringTopic ModelsWord EmbeddingsBERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate these pre-trained models into neural mach…
de-enLanguage ModelingLanguage ModellingMachine Translation+2InvBERT: Reconstructing Text from Contextualized Word Embeddings by inverting the BERT pipeline
Digital Humanities and Computational Literary Studies apply text mining methods to investigate literature. Such automated approaches enable quantitative studies on large corpora which would not be feasible by manual insp…
DecoderWord EmbeddingsImproving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance
Recent advances in automatic evaluation metrics for text have shown that deep contextualized word representations, such as those generated by BERT encoders, are helpful for designing metrics that correlate well with huma…
Text GenerationUtilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges
Recent years have witnessed a substantial increase in the use of deep learning to solve various natural language processing (NLP) problems. Early deep learning models were constrained by their sequential or unidirectiona…
Information RetrievalRetrievalSurvey