Not All Memories are Created Equal: Learning to Expire
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work has investigated mechanisms to reduce the computational cost of preserving and storing the memories. However, not all content in the past is equally important to remember. We propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant information. This enables Transformers to scale to attend to tens of thousands of previous timesteps efficiently, as not all hidden states from previous timesteps are preserved. We demonstrate that Expire-Span can help models identify and retain critical information and show it can achieve state of the art results on long-context language modeling, reinforcement learning, and algorithmic tasks. Finally, we show that Expire-Span can scale to memories that are tens of thousands in size, which is helpful on incredibly long context tasks such as character-level PG-19 and a frame-by-frame moving objects task.
Code (1)
Tasks
AllLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Not All Memories are Created Equal: Learning to Forget by Expiring
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of preserving and storing memories. However…
AllLanguage ModelingLanguage ModellingComparative Companies' Stock Valuation through Financial Metrics and its Social Implications
Out of the companies, Dolby is the company with the best overall financial and operation health. According to the table that accounted its financial statements for the past three years, Dolby has stable profit margins th…
MarketingMemory-QA: Answering Recall Questions Based on Multimodal Memories
We introduce Memory-QA, a novel real-world task that involves answering recall questions about visual content from previously stored multimodal memories. This task poses unique challenges, including the creation of task-…
The design of a proto-animal brain based upon spike timing
A basal animal model is described as an organism similar to a Limpet that is attached to the sea floor living in a reproductive community. Its brain model uses logic cells (gates) to create a high frequency spike generat…
Short-Term Memory Through Persistent Activity: Evolution of Self-Stopping and Self-Sustaining Activity in Spiking Neural Networks
Memories in the brain are separated in two categories: short-term and long-term memories. Long-term memories remain for a lifetime, while short-term ones exist from a few milliseconds to a few minutes. Within short-term …