Multi-objective Representation Learning for Scientific Document Retrieval
Existing dense retrieval models for scientific documents have been optimized for either retrieval by short queries, or for document similarity, but usually not for both. In this paper, we explore the space of combining multiple objectives to achieve a single representation model that presents a good balance between both modes of dense retrieval, combining the relevance judgements from MS MARCO with the citation similarity of SPECTER, and the self-supervised objective of independent cropping. We also consider the addition of training data from document co-citation in a sentence context and domain-specific synthetic data. We show that combining multiple objectives yields models that generalize well across different benchmark tasks, improving up to 73% over models trained on a single objective.
Code (1)
Tasks
Representation LearningRetrievalSentenceSimilar Papers 제목 키워드 기반
Document-as-Image Representations Fall Short for Scientific Retrieval
Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing benchmarks for scientific document retriev…
LLM-Based Compact Reranking with Document Features for Scientific Retrieval
Scientific retrieval is essential for advancing academic discovery. Within this process, document reranking plays a critical role by refining first-stage retrieval results. However, large language model (LLM) listwise re…
Large Language ModelRerankingRetrievalSparse Meets Dense: A Hybrid Approach to Enhance Scientific Document Retrieval
Traditional information retrieval is based on sparse bag-of-words vector representations of documents and queries. More recent deep-learning approaches have used dense embeddings learned using a transformer-based large l…
Deep LearningInformation RetrievalLanguage ModelingLanguage Modelling+2Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
Scientific document classification is a critical task for a wide range of applications, but the cost of obtaining massive amounts of human-labeled data can be prohibitive. To address this challenge, we propose a weakly-s…
Document ClassificationRetrievalColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval
Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed…
Representation LearningContrastive Learning