Contextual Position Encoding: Learning to Count What's Important
The attention mechanism is a critical component of Large Language Models (LLMs) that allows tokens in a sequence to interact with each other, but is order-invariant. Incorporating position encoding (PE) makes it possible to address by position, such as attending to the i-th token. However, current PE methods use token counts to derive position, and thus cannot generalize to higher levels of abstraction, such as attending to the i-th sentence. In this paper, we propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the $i$-th particular word, noun, or sentence. We show that CoPE can solve the selective copy, counting and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingPositionSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Position Information in Transformers: An Overview
Transformers are arguably the main workhorse in recent Natural Language Processing research. By definition a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential an…
ClusteringPositionMeasuring What AI Systems Might Do: Towards A Measurement Science in AI
Scientists, policy-makers, business leaders, and members of the public care about what modern artificial intelligence systems are disposed to do. Yet terms such as capabilities, propensities, skills, values, and abilitie…
Positional encoding is not the same as context: A study on positional encoding for sequential recommendation
The rapid growth of streaming media and e-commerce has driven advancements in recommendation systems, particularly Sequential Recommendation Systems (SRS). These systems employ users' interaction histories to predict fut…
Recommendation SystemsSequential RecommendationEnergy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
Standard transformer attention computes pairwise token similarity but treats all tokens as equally salient and all positions as equally local, regardless of the informational structure of the input. We identify two compl…
What Are the Invariant Occlusive Components of Image Patches? A Probabilistic Generative Approach
We study optimal image encoding based on a generative approach with non-linear feature combinations and explicit position encoding. By far most approaches to unsupervised learning learning of visual features, such as spa…
Position