Headless Language Models: Learning without Predicting with Contrastive Weight Tying

2023-09-15 · Nathan Godey, Éric de la Clergerie, Benoît Sagot

Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative method that shifts away from probability prediction and instead focuses on reconstructing input embeddings in a contrastive fashion via Constrastive Weight Tying (CWT). We apply this approach to pretrain Headless Language Models in both monolingual and multilingual contexts. Our method offers practical advantages, substantially reducing training computational requirements by up to 20 times, while simultaneously enhancing downstream performance and data efficiency. We observe a significant +1.6 GLUE score increase and a notable +2.7 LAMBADA accuracy improvement compared to classical LMs within similar compute budgets.

📄 PDF Abstract BibTeX arXiv:2309.08351

Code (0)

등록된 구현이 없습니다.

Tasks

LAMBADA

Methods 이 논문이 사용한 방법론

Weight Tying Weight Tying improves the performance of language models by tying (sharing) the weights of the embedding and softmax layers. This…

SSH: Single Stage Headless Face Detector

2017-08-14 · ICCV 2017 10 · Mahyar Najibi, Pouya Samangouei, Rama Chellappa, Larry Davis

We introduce the Single Stage Headless (SSH) face detector. Unlike two stage proposal-classification detectors, SSH detects faces in a single stage directly from the early convolutional layers in a classification network…

General Classification GPU

Headless Horseman: Adversarial Attacks on Transfer Learning Models

2020-04-20 · Ahmed Abdelkader, Michael J. Curry, Liam Fowl, Tom Goldstein 외

Transfer learning facilitates the training of task-specific classifiers using pre-trained models as feature extractors. We present a family of transferable adversarial attacks against such classifiers, generated without …

Adversarial Attack Transfer Learning

Extracting Headless MWEs from Dependency Parse Trees: Parsing, Tagging, and Joint Modeling Approaches

2020-05-06 · ACL 2020 6 · Tianze Shi, Lillian Lee

An interesting and frequent type of multi-word expression (MWE) is the headless MWE, for which there are no true internal syntactic dominance relations; examples include many named entities ("Wells Fargo") and dates ("Ju…

Decoder

In-Context Reinforcement Learning for Variable Action Spaces

2023-12-20 · Viacheslav Sinii, Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman 외

Recently, it has been shown that transformers pre-trained on diverse datasets with multi-episode contexts can generalize to new reinforcement learning tasks in-context. A key limitation of previously proposed models is t…

In-Context Reinforcement Learning Multi-Armed Bandits reinforcement-learning Reinforcement Learning

S23DR 2026: End-to-End 3D Wireframe Prediction via DETR-Style Set Prediction with Contrastive Denoising

2026-06-12 · Nitiz Khanal arxiv

We present WireframeDETR, our submission to the Structured Semantic 3D Reconstruction (S23DR) 2026 Challenge, which requires predicting a 3D building wireframe from multi-view COLMAP point clouds. Our method applies DETR…

3D Reconstruction Point Clouds