Suffix Retrieval-Augmented Language Modeling

2022-11-06 · Zecheng Wang, Yik-Cheung Tam

Causal language modeling (LM) uses word history to predict the next word. BERT, on the other hand, makes use of bi-directional word information in a sentence to predict words at masked positions. While BERT is effective in sequence encoding, it is non-causal by nature and is not designed for sequence generation. In this paper, we propose a novel language model, SUffix REtrieval-Augmented LM (SUREALM), that simulates a bi-directional contextual effect in an autoregressive manner. SUREALM employs an embedding retriever to search for training sentences in a data store that share similar word history during sequence generation. In particular, the suffix portions of the retrieved sentences mimick the "future" context. We evaluated our proposed model on the DSTC9 spoken dialogue corpus and showed promising word perplexity reduction on the validation and test set compared to competitive baselines.

📄 PDF Abstract BibTeX arXiv:2211.03053

Code (1)

victor-wang-902/surealm 공식 구현 pytorch

Tasks

Causal Language ModelingLanguage ModelingLanguage Modelling RetrievalSentence

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Test 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection

2025-07-20 · Jerry Wang, Fang Yu arxiv

Adversarial prompt attacks can significantly alter the reliability of Retrieval-Augmented Generation (RAG) systems by re-ranking them to produce incorrect outputs. In this paper, we present a novel method that applies Di…

Question Answering

Bypassing Prompt Injection Detectors through Evasive Injections

2026-01-31 · Md Jahedur Rahman, Ihsen Alouani arxiv

Large language models (LLMs) are increasingly used in interactive and retrieval-augmented systems, but they remain vulnerable to prompt injection attacks, where injected secondary prompts force the model to deviate from …

Suffix Trees as Language Models

2012-05-01 · LREC 2012 5 · Casey Redd Kennington, Martin Kay, Annemarie Friedrich

Suffix trees are data structures that can be used to index a corpus. In this paper, we explore how some properties of suffix trees naturally provide the functionality of an n-gram language model with variable n. We expla…

Information Retrieval Language Modeling Language Modelling Machine Translation+2

Accelerating Diffusion Language Models via Structured Suffix Modeling

2026-08-24 · Zifeng Cheng, Keda Li, Zhiwei Jiang, Cong Wang 외 arxiv

Diffusion Language Models (DLMs) exhibit strong parallel decoding capabilities by denoising multiple tokens in a single generation step. However, this parallelism comes with substantial computational overhead, as each st…

Utilizing Metadata for Better Retrieval-Augmented Generation

2026-01-17 · Raquib Bin Yousuf, Shengzhe Xu, Mandar Sharma, Andrew Neeser 외 arxiv

Retrieval-Augmented Generation systems depend on retrieving semantically relevant document chunks to support accurate, grounded outputs from large language models. In structured and repetitive corpora such as regulatory …