The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More

2024-06-07 · Ouail Kitouni, Niklas Nolte, Diane Bouchacourt, Adina Williams, Mike Rabbat, Mark Ibrahim

Today's best language models still struggle with hallucinations: factually incorrect generations, which impede their ability to reliably retrieve information seen during training. The reversal curse, where models cannot recall information when probed in a different order than was encountered during training, exemplifies this in information retrieval. We reframe the reversal curse as a factorization curse - a failure of models to learn the same joint distribution under different factorizations. Through a series of controlled experiments with increasing levels of realism including WikiReversal, a setting we introduce to closely simulate a knowledge intensive finetuning task, we find that the factorization curse is an inherent failure of the next-token prediction objective used in popular large language models. Moreover, we demonstrate reliable information retrieval cannot be solved with scale, reversed tokens, or even naive bidirectional-attention training. Consequently, various approaches to finetuning on specialized data would necessarily provide mixed results on downstream tasks, unless the model has already seen the right sequence of tokens. Across five tasks of varying levels of complexity, our results uncover a promising path forward: factorization-agnostic objectives can significantly mitigate the reversal curse and hint at improved knowledge storage and planning capabilities.

📄 PDF Abstract BibTeX arXiv:2406.05183

Code (0)

등록된 구현이 없습니다.

Tasks

Information Retrieval Retrieval

Methods 이 논문이 사용한 방법론

HINT An unsupervised approach for identifying Hierarchical Information Threads by analysing the network of related articles in a collection. In particular, HINT leverages article…

RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

2025-01-23 · Munachiso Nwadike, Zangir Iklassov, Toluwani Aremu, Tatsuya Hiraoka 외

We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomen…

Infinite Mask Diffusion for Few-Step Distillation

2026-05-11 · Jaehoon Yoo, Wonjung Kim, Chanhyuk Lee, Seunghoon Hong arxiv

Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling, offering the advantages of parallel decoding and bidirectional context processing within a simple yet …

Overcoming the Curse of Dimensionality in Reinforcement Learning Through Approximate Factorization

2024-11-12 · Chenbei Lu, Laixi Shi, Zaiwei Chen, Chenye Wu 외

Reinforcement Learning (RL) algorithms are known to suffer from the curse of dimensionality, which refers to the fact that large-scale problems often lead to exponentially high sample complexity. A common solution is to …

Q-Learning Reinforcement Learning (RL)

LEDOM: Reverse Language Model

2025-07-02 · Xunjian Yin, Sitao Cheng, Yuxi Xie, Xinyu Hu 외 arxiv

Autoregressive language models are trained exclusively left-to-right. We explore the complementary factorization, training right-to-left at scale, and ask what reasoning patterns emerge when a model conditions on future …

UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation

2026-08-17 · Rongcheng Lin, Yan Sun, Jamey Zhang, Guanglei Xiong 외 arxiv

Industrial recommenders rely on two model families that have evolved largely independently: feature-interaction models over multi-field user/item features, and sequential models over user-behavior histories. Production s…

Collaborative Filtering