Papers ListOps
“ListOps” 태그가 달린 논문 22편 · 필터 해제
Context Is Not Comprehension
The dominant way of judging Large Language Models (LLMs) has been to ask how well they can recall explicit facts from very long inputs. While today's best models achieve near perfect recall, this masks a harder skill: pe…
ListOpsSmall Models, Smarter Learning: The Power of Joint Task Training
The ability of a model to learn a task depends strongly on both the task difficulty and the model size. We aim to understand how task difficulty relates to the minimum number of parameters required for learning specific …
ListOpsInvestigating Recurrent Transformers with Dynamic Halt
In this paper, we comprehensively study the inductive biases of two major approaches to augmenting Transformers with a recurrent mechanism: (1) the approach of incorporating a depth-wise recurrence similar to Universal T…
DiagnosticLanguage ModelingLanguage ModellingListOpsCached Transformers: Improving Transformers with Differentiable Memory Cache
This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens. GRC attention…
image-classificationImage ClassificationInstance SegmentationLanguage Modeling+6Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI
Pre-trained language models have recently emerged as a powerful tool for fine-tuning a variety of language tasks. Ideally, when models are pre-trained on large amount of data, they are expected to gain implicit knowledge…
General KnowledgeListOpsOpening the Black Box: Analyzing Attention Weights and Hidden States in Pre-trained Language Models for Non-language Tasks
Investigating deep learning language models has always been a significant research area due to the ``black box" nature of most advanced models. With the recent advancements in pre-trained language models based on transfo…
Language ModellingListOpsBeam Tree Recursive Cells
We propose Beam Tree Recursive Cell (BT-Cell) - a backpropagation-friendly framework to extend Recursive Neural Networks (RvNNs) with beam search for latent structure induction. We further extend this framework by propos…
ListOpsSequence Modeling with Multiresolution Convolutional Memory
Efficiently capturing the long-range patterns in sequential data sources salient to a given task -- such as classification and generative modeling -- poses a fundamental challenge. Popular approaches in the space tradeof…
Density EstimationListOpsSequential Image ClassificationDARTFormer: Finding The Best Type Of Attention
Given the wide and ever growing range of different efficient Transformer attention mechanisms, it is important to identify which attention is most effective when given a task. In this work, we are also interested in comb…
ListOpsNeural Architecture Searchtext-classificationText Classification+1Mega: Moving Average Equipped Gated Attention
The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega,…
Image ClassificationInductive BiasLanguage ModelingLanguage Modelling+5Simplified State Space Layers for Sequence Modeling
Models using structured state space sequence (S4) layers have achieved state-of-the-art performance on long-range sequence modeling tasks. An S4 layer combines linear state space models (SSMs), the HiPPO framework, and d…
Computational EfficiencyListOpsLong-range modelingPathfinder+3Training Discrete Deep Generative Models via Gapped Straight-Through Estimator
While deep generative models have succeeded in image processing, natural language processing, and reinforcement learning, training that involves discrete random variables remains challenging due to the high variance of i…
ListOpsreinforcement-learningReinforcement Learning (RL)Dynamic Token Normalization Improves Vision Transformers
Vision Transformer (ViT) and its variants (e.g., Swin, PVT) have achieved great success in various computer vision tasks, owing to their capability to learn long-range contextual information. Layer Normalization (LN) is …
Inductive BiasListOpsobject-detectionObject DetectionORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning
The ability to reason with multiple hierarchical structures is an attractive and desirable property of sequential inductive biases for natural language processing. Do the state-of-the-art Transformers and LSTM architectu…
DiagnosticListOpsRelational ReasoningSystematic GeneralizationEfficiently Modeling Long Sequences with Structured State Spaces
A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly on long-range dependencies. Although conventional models incl…
Data AugmentationLanguage ModelingLanguage ModellingListOps+3The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite progress across a broad range of applications, Transformers have limited success in systematic generalization. The situation is especially frustrating in the case of algorithmic tasks, where they often fail to fi…
ListOpsSystematic GeneralizationAdaptive Control Flow in Transformers Improves Systematic Generalization
Despite successes across a broad range of applications, Transformers have limited capability in systematic generalization. The situation is especially frustrating in the case of algorithmic tasks, where they often fail t…
ListOpsSystematic GeneralizationGoing Beyond Linear Transformers with Recurrent Fast Weight Programmers
Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s. However, the original FW…
Atari GamesListOpsModeling Hierarchical Structures with Continuous Recursive Neural Networks
Recursive Neural Networks (RvNNs), which compose sequences according to their underlying hierarchical syntactic structure, have performed well in several natural language processing tasks compared to similar models witho…
ListOpsNatural Language InferenceSentiment AnalysisLong Range Arena: A Benchmark for Efficient Transformers
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this…
16kBenchmarkingListOpsLong-range modeling+1