SCRIPT: Self-Critic PreTraining of Transformers
We introduce Self-CRItic Pretraining Transformers (SCRIPT) for representation learning of text. The popular masked language modeling (MLM) pretraining methods like BERT replace some tokens with [MASK] and an encoder is trained to recover them, while ELECTRA trains a discriminator to detect replaced tokens proposed by a generator. In contrast, we train a language model as in MLM and further derive a discriminator or critic on top of the encoder without using any additional parameters. That is, the model itself is a critic. SCRIPT combines MLM training and discriminative training for learning rich representations and compute- and sample-efficiency. We demonstrate improved sample-efficiency in pretraining and enhanced representations evidenced by improved downstream task performance on GLUE and SQuAD over strong baselines. Also, the self-critic scores can be directly used as pseudo-log-likelihood for efficient scoring.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMasked Language ModelingRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models
Pathology foundation models learn morphological representations through self-supervised pretraining on large-scale whole-slide images, yet they do not explicitly capture the underlying molecular state of the tissue. Spat…
Pay Attention to MLPs
Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLP…
Image ClassificationNatural Language InferenceQuestion AnsweringSentiment AnalysisSelf-supervised pretraining of vision transformers for animal behavioral analysis and neural encoding
The brain can only be fully understood through the lens of the behavior it generates -- a guiding principle in modern neuroscience research that nevertheless presents significant technical challenges. Many studies captur…
Action SegmentationContrastive LearningPose EstimationLocation-Aware Self-Supervised Transformers for Semantic Segmentation
Pixel-level labels are particularly expensive to acquire. Hence, pretraining is a critical step to improve models on a task like semantic segmentation. However, prominent algorithms for pretraining neural networks use im…
Contrastive Learningimage-classificationImage ClassificationSegmentation+2FactoFormer: Factorized Hyperspectral Transformers with Self-Supervised Pretraining
Hyperspectral images (HSIs) contain rich spectral and spatial information. Motivated by the success of transformers in the field of natural language processing and computer vision where they have shown the ability to lea…