Papers Hard Attention
“Hard Attention” 태그가 달린 논문 100편 · 필터 해제
Comparison of different Unique hard attention transformer models by the formal languages they can recognize
This note is a survey of various results on the capabilities of unique hard attention transformers encoders (UHATs) to recognize formal languages. We distinguish between masked vs. non-masked, finite vs. infinite image a…
Hard AttentionSurveyCharacterizing the Expressivity of Transformer Language Models
Transformer-based language models (LMs) have achieved widespread empirical success, but their theoretical expressive power remains only partially understood. Prior work often relies on idealized models with assumptions -…
Hard AttentionExact Expressive Power of Transformers with Padding
Chain of thought is a natural inference-time method for increasing the computational power of transformer-based large language models (LLMs), but comes at the cost of sequential decoding. Are there more efficient alterna…
Hard AttentionEmergence of Fixational and Saccadic Movements in a Multi-Level Recurrent Attention Model for Vision
Inspired by foveal vision, hard attention models promise interpretability and parameter economy. However, existing models like the Recurrent Model of Visual Attention (RAM) and Deep Recurrent Attention Model (DRAM) faile…
Hard Attentionimage-classificationImage ClassificationNoPE: The Counting Power of Transformers with No Positional Encodings
Positional Encodings (PEs) seem to be indispensable for ensuring expressiveness of transformers; without them attention transformers reduce to a bag-of-word model. NoPE-transformers (i.e. with No PEs) with unique hard at…
Hard AttentionNeuroevolution of Self-Attention Over Proto-Objects
Proto-objects - image regions that share common visual properties - offer a promising alternative to traditional attention mechanisms based on rectangular-shaped image patches in neural networks. Although previous work d…
Hard AttentionImage SegmentationSemantic SegmentationAttentionDrop: A Novel Regularization Method for Transformer Models
Transformer-based architectures achieve state-of-the-art performance across a wide range of tasks in natural language processing, computer vision, and speech. However, their immense capacity often leads to overfitting, e…
Hard AttentionCenter-guided Classifier for Semantic Segmentation of Remote Sensing Images
Compared with natural images, remote sensing images (RSIs) have the unique characteristic. i.e., larger intraclass variance, which makes semantic segmentation for remote sensing images more challenging. Moreover, existin…
Hard AttentionSegmentationSemantic SegmentationUnique Hard Attention: A Tale of Two Sides
Understanding the expressive power of transformers has recently attracted attention, as it offers insights into their abilities and limitations. Many studies analyze unique hard attention transformers, where attention se…
Hard AttentionLower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers
Chain-of-thought reasoning and scratchpads have emerged as critical tools for enhancing the computational capabilities of transformers. While theoretical results show that polynomial-length scratchpads can extend transfo…
Hard AttentionEhrenfeucht-Haussler Rank and Chain of Thought
The notion of rank of a Boolean function has been a cornerstone in the theory of PAC learning, enabling quasipolynomial-time learning algorithms for polynomial-size decision trees. We present a novel characterization of …
DecoderHard AttentionPAC learningPositionSimulating Hard Attention Using Soft Attention
We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine several subclasses of languages recogniz…
Hard AttentionTransformers in Uniform TC$^0$
Previous work has shown that the languages recognized by average-hard attention transformers (AHATs) and softmax-attention transformers (SMATs) are within the circuit complexity class TC$^0$. However, these results assum…
Hard AttentionSoft-Hard Attention U-Net Model and Benchmark Dataset for Multiscale Image Shadow Removal
Effective shadow removal is pivotal in enhancing the visual quality of images in various applications, ranging from computer vision to digital photography. During the last decades physics and machine learning -based meth…
BenchmarkingHard AttentionImage Shadow RemovalPrivacy Preserving+1Hard-Attention Gates with Gradient Routing for Endoscopic Image Computing
To address overfitting and enhance model generalization in gastroenterological polyp size assessment, our study introduces Feature-Selection Gates (FSG) or Hard-Attention Gates (HAG) alongside Gradient Routing (GR) for d…
Binary Classificationfeature selectionHard AttentionTRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision
Neuromorphic processors are well-suited for efficiently handling sparse events from event-based cameras. However, they face significant challenges in the growth of computing demand and hardware costs as the input resolut…
Event-based visionHard AttentionTransformers as Transducers
We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of transductions. We do so using variants of RASP, a pr…
Hard AttentionPOSRecurrent Alignment with Hard Attention for Hierarchical Text Rating
While large language models (LLMs) excel at understanding and generating plain text, they are not tailored to handle hierarchical text structures or directly predict task-specific properties such as text rating. In fact,…
Hard AttentionGQHAN: A Grover-inspired Quantum Hard Attention Network
Numerous current Quantum Machine Learning (QML) models exhibit an inadequacy in discerning the significance of quantum data, resulting in diminished efficacy when handling extensive quantum datasets. Hard Attention Mecha…
Binary ClassificationHard AttentionQuantum Machine LearningMutual Distillation Learning For Person Re-Identification
With the rapid advancements in deep learning technologies, person re-identification (ReID) has witnessed remarkable performance improvements. However, the majority of prior works have traditionally focused on solving the…
Hard AttentionPerson Re-Identification