Generative Dense Retrieval: Memory Can Be a Burden

2024-01-19 · Peiwen Yuan, Xinglin Wang, Shaoxiong Feng, Boyuan Pan, Yiwei Li, HeDa Wang, Xupeng Miao, Kan Li

Generative Retrieval (GR), autoregressively decoding relevant document identifiers given a query, has been shown to perform well under the setting of small-scale corpora. By memorizing the document corpus with model parameters, GR implicitly achieves deep interaction between query and document. However, such a memorizing mechanism faces three drawbacks: (1) Poor memory accuracy for fine-grained features of documents; (2) Memory confusion gets worse as the corpus size increases; (3) Huge memory update costs for new documents. To alleviate these problems, we propose the Generative Dense Retrieval (GDR) paradigm. Specifically, GDR first uses the limited memory volume to achieve inter-cluster matching from query to relevant document clusters. Memorizing-free matching mechanism from Dense Retrieval (DR) is then introduced to conduct fine-grained intra-cluster matching from clusters to relevant documents. The coarse-to-fine process maximizes the advantages of GR's deep interaction and DR's scalability. Besides, we design a cluster identifier constructing strategy to facilitate corpus memory and a cluster-adaptive negative sampling strategy to enhance the intra-cluster mapping ability. Empirical results show that GDR obtains an average of 3.0 R@100 improvement on NQ dataset under multiple settings and has better scalability.

📄 PDF Abstract BibTeX arXiv:2401.10487

Code (1)

ypw0102/gdr 공식 구현 jax

Tasks

Retrieval

Learning to Rank in Generative Retrieval

2023-06-27 · Yongqi Li, Nan Yang, Liang Wang, Furu Wei 외

Generative retrieval stands out as a promising new paradigm in text retrieval that aims to generate identifier strings of relevant passages as the retrieval target. This generative paradigm taps into powerful generative …

Learning-To-Rank Passage Ranking Retrieval Text Generation+1

Unifying Generative and Dense Retrieval for Sequential Recommendation

2024-11-27 · Liu Yang, Fabian Paischer, Kaveh Hassani, Jiacheng Li 외

Sequential dense retrieval models utilize advanced sequence learning techniques to compute item and user representations, which are then used to rank relevant items for a user through inner product computation between th…

Recommendation Systems Retrieval Sequential Recommendation

Distillation Enhanced Generative Retrieval

2024-02-16 · Yongqi Li, Zhen Zhang, Wenjie Wang, Liqiang Nie 외

Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generative language models, distinct fro…

Retrieval Text Retrieval

To Know is to Construct: Schema-Constrained Generation for Agent Memory

2026-04-22 · Lei Zheng, Weinan Song, Daili Li, Yanming Yang arxiv

Constructivist epistemology argues that knowledge is actively constructed rather than passively copied. Despite the generative nature of Large Language Models (LLMs), most existing agent memory systems are still based on…

Dense Associative Memory with Epanechnikov Energy

2025-06-12 · Benjamin Hoover, Zhaoyang Shi, Krishnakumar Balasubramanian, Dmitry Krotov 외

We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is base…

Density Estimation