Conformer-Kernel with Query Term Independence for Document Retrieval
The Transformer-Kernel (TK) model has demonstrated strong reranking performance on the TREC Deep Learning benchmark---and can be considered to be an efficient (but slightly less effective) alternative to BERT-based ranking models. In this work, we extend the TK architecture to the full retrieval setting by incorporating the query term independence assumption. Furthermore, to reduce the memory complexity of the Transformer layers with respect to the input sequence length, we propose a new Conformer layer. We show that the Conformer's GPU memory requirement scales linearly with input sequence length, making it a more viable option when ranking long documents. Finally, we demonstrate that incorporating explicit term matching signal into the model can be particularly useful in the full retrieval setting. We present preliminary results from our work in this paper.
Code (1)
Tasks
GPURerankingRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Conformer-Kernel with Query Term Independence at TREC 2020 Deep Learning Track
We benchmark Conformer-Kernel models under the strict blind evaluation setting of the TREC 2020 Deep Learning track. In particular, we study the impact of incorporating: (i) Explicit term matching to complement matching …
RetrievalImproving Transformer-Kernel Ranking Model Using Conformer and Query Term Independence
The Transformer-Kernel (TK) model has demonstrated strong reranking performance on the TREC Deep Learning benchmark -- and can be considered to be an efficient (but slightly less effective) alternative to other Transform…
Document RankingRerankingRetrievalIncorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
Classical information retrieval (IR) methods, such as query likelihood and BM25, score documents independently w.r.t. each query term, and then accumulate the scores. Assuming query term independence allows precomputing …
Information RetrievalPassage RankingRe-RankingRetrievalMulti-Convformer: Extending Conformer with Multiple Convolution Kernels
Convolutions have become essential in state-of-the-art end-to-end Automatic Speech Recognition~(ASR) systems due to their efficient modelling of local context. Notably, its use in Conformers has led to superior performan…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech RecognitionAudio-Visual Efficient Conformer for Robust Speech Recognition
End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made i…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Lip ReadingRobust Speech Recognition+2