Continuous Active Learning Using Pretrained Transformers

2022-08-15 · Nima Sadri, Gordon V. Cormack

Pre-trained and fine-tuned transformer models like BERT and T5 have improved the state of the art in ad-hoc retrieval and question-answering, but not as yet in high-recall information retrieval, where the objective is to retrieve substantially all relevant documents. We investigate whether the use of transformer-based models for reranking and/or featurization can improve the Baseline Model Implementation of the TREC Total Recall Track, which represents the current state of the art for high-recall information retrieval. We also introduce CALBERT, a model that can be used to continuously fine-tune a BERT-based model based on relevance feedback.

📄 PDF Abstract BibTeX arXiv:2208.06955

Code (0)

등록된 구현이 없습니다.

Tasks

Active Learning Information Retrieval Question AnsweringRerankingRetrieval

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Gated Linear Unit A Gated Linear Unit, or GLU computes: $$ \mathrm{GLU}(a, b) = a \otimes \sigma(b) $$ It is used in natural language processing architectures, for example the Gated CNN,…

s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning

2021-10-26 · Hangbo Bao, Li Dong, Wenhui Wang, Nan Yang 외

Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language g…

Abstractive Text Summarization Question Generation Question-Generation Text Generation

Interactive Image Segmentation with Cross-Modality Vision Transformers

2023-07-05 · Kun Li, George Vosselman, Michael Ying Yang

Interactive image segmentation aims to segment the target from the background with the manual guidance, which takes as input multimodal data such as images, clicks, scribbles, and bounding boxes. Recently, vision transfo…

Image Segmentation Interactive Segmentation Segmentation Semantic Segmentation

Text Summarization with Pretrained Encoders

2019-08-22 · IJCNLP 2019 11 · Yang Liu, Mirella Lapata

Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper…

Abstractive Text Summarization Decoder Document Summarization Extractive Document Summarization+3

ActiveMimic: Egocentric Video Pretraining with Active Perception

2026-06-04 · Xingyao Lin, Guojin Zhong, Tianyi Lu, Ziyi Ye 외 arxiv

Egocentric human video offers a scalable alternative to robot data for pretraining, yet models pretrained on such video consistently underperform those pretrained on robot data. We attribute this gap to a missing signal,…

Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers

2025-11-24 · Rowan Bradbury, Aniket Srinivasan Ashok, Sai Ram Kasanagottu, Gunmay Jhingran 외 arxiv

Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We …