Papers Masked Language Modeling
“Masked Language Modeling” 태그가 달린 논문 475편 · 필터 해제
STT: Soft Template Tuning for Few-Shot Adaptation
Prompt tuning has been an extremely effective tool to adapt a pre-trained model to downstream tasks. However, standard prompt-based methods mainly consider the case of sufficient data of downstream tasks. It is still unc…
Few-Shot LearningLanguage ModelingLanguage ModellingMasked Language Modeling+2Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages
We leverage pre-trained language models to solve the task of complex NER for two low-resource languages: Chinese and Spanish. We use the technique of Whole Word Masking(WWM) to boost the performance of masked language mo…
Language ModelingLanguage ModellingMasked Language ModelingNER+1GPTs at Factify 2022: Prompt Aided Fact-Verification
One of the most pressing societal issues is the fight against false news. The false claims, as difficult as they are to expose, create a lot of damage. To tackle the problem, fact verification becomes crucial and thus ha…
Fact VerificationLanguage ModelingLanguage ModellingMasked Language ModelingSemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (M…
Language ModelingLanguage ModellingMasked Language ModelingSemantic SegmentationGeneral Framework for Reversible Data Hiding in Texts Based on Masked Language Modeling
With the fast development of natural language processing, recent advances in information hiding focus on covertly embedding secret information into texts. These algorithms either modify a given cover text or directly gen…
Language ModelingLanguage ModellingMasked Language ModelingSSM-DTA: Breaking the Barriers of Data Scarcity in Drug-Target Affinity Prediction
Accurate prediction of Drug-Target Affinity (DTA) is of vital importance in early-stage drug discovery, facilitating the identification of drugs that can effectively interact with specific targets and regulate their acti…
Drug DiscoveryLanguage ModelingLanguage ModellingMasked Language Modeling+2Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle thi…
Fill MaskLanguage ModelingLanguage ModellingMasked Language Modeling+8LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Unified vision-language frameworks have greatly advanced in recent years, most of which adopt an encoder-decoder architecture to unify image-text tasks as sequence-to-sequence generation. However, existing video-language…
DecoderLanguage ModelingLanguage ModellingMasked Language Modeling+6GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2…
2D Object DetectionContrastive LearningImage CaptioningInstance Segmentation+10VL-BEiT: Generative Vision-Language Pretraining
We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and …
image-classificationImage ClassificationImage-text RetrievalLanguage Modeling+9Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is mo…
Contrastive LearningCross-Lingual TransferImage-text RetrievalLanguage Modeling+10Developing Language Resources and NLP Tools for the North Korean Language
Since the division of Korea, the two Korean languages have diverged significantly over the last 70 years. However, due to the lack of linguistic source of the North Korean language, there is no DPRK-based language model.…
Language ModelingLanguage ModellingMasked Language ModelingSentiment AnalysisDiscovering Financial Hypernyms by Prompting Masked Language Models
With the rising popularity of Transformer-based language models, several studies have tried to exploit their masked language modeling capabilities to automatically extract relational linguistic knowledge, although this k…
Domain AdaptationLanguage ModelingLanguage ModellingMasked Language ModelingTraining and Inference on Any-Order Autoregressive Models the Right Way
Conditional inference on arbitrary subsets of variables is a core problem in probabilistic inference with important applications such as masked language modeling and image inpainting. In recent years, the family of Any-O…
Image InpaintingLanguage ModelingLanguage ModellingMasked Language ModelingTranscormer: Transformer for Sentence Scoring with Sliding Language Modeling
Sentence scoring aims at measuring the likelihood score of a sentence and is widely used in many natural language processing scenarios, like reranking, which is to select the best sentence from multiple candidates. Previ…
Causal Language ModelingLanguage ModelingLanguage ModellingMasked Language Modeling+2Enhancing Continual Learning with Global Prototypes: Counteracting Negative Representation Drift
Continual learning (CL) aims to learn a sequence of tasks over time, with data distributions shifting from one task to another. When training on new task data, data representations from old tasks may drift. Some negative…
Continual LearningLanguage ModelingLanguage ModellingMasked Language ModelingRetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
Despite pre-training's progress in many important NLP tasks, it remains to explore effective pre-training strategies for dense retrieval. In this paper, we propose RetroMAE, a new retrieval oriented pre-training paradigm…
DecoderInformation RetrievalLanguage ModelingLanguage Modelling+7MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification
In text summarization and simplification, system outputs must be evaluated along multiple dimensions such as relevance, factual consistency, fluency, and grammaticality, and a wide range of possible outputs could be of h…
Language ModelingLanguage ModellingMasked Language ModelingText Simplification+1Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in causal and masked language modeling, across …
Language ModelingLanguage ModellingMasked Language ModelingMemorizationMultilingual Normalization of Temporal Expressions with Masked Language Models
The detection and normalization of temporal expressions is an important task and preprocessing step for many applications. However, prior work on normalization is rule-based, which severely limits the applicability in re…
Language ModelingLanguage ModellingMasked Language Modeling