Unsupervised Attention Mechanism across Neural Network Layers

2019-02-27 · Baihan Lin

Inspired by the adaptation phenomenon of neuronal firing, we propose an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, UAM constrained the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrated the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in computer vision and reinforcement learning tasks. Lastly, UAM tracks dependency and critical learning stages across layers and recurrent time steps of deep networks.

📄 PDF Abstract BibTeX arXiv:1902.10658

Code (1)

doerlbh/UnsupervisedAttentionMechanism 공식 구현 pytorch

Tasks

Few-Shot Learning Image ClassificationModel Selectionreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Attention-based Cross-Layer Domain Alignment for Unsupervised Domain Adaptation

2022-02-27 · Xu Ma, Junkun Yuan, Yen-Wei Chen, Ruofeng Tong 외

Unsupervised domain adaptation (UDA) aims to learn transferable knowledge from a labeled source domain and adapts a trained model to an unlabeled target domain. To bridge the gap between source and target domains, one pr…

Domain Adaptation Semantic Similarity Semantic Textual Similarity Unsupervised Domain Adaptation

MUFASA: A Multi-Layer Framework for Slot Attention

2026-02-07 · Sebastian Bock, Leonie Schüßler, Krishnakant Singh, Simone Schaub-Meyer 외 arxiv

Unsupervised object-centric learning (OCL) decomposes visual scenes into distinct entities. Slot attention is a popular approach that represents individual objects as latent vectors, called slots. Current methods obtain …

Unsupervised Object Segmentation

LEA-Net: Layer-wise External Attention Network for Efficient Color Anomaly Detection

2021-09-12 · Ryoya Katafuchi, Terumasa Tokunaga

The utilization of prior knowledge about anomalies is an essential issue for anomaly detections. Recently, the visual attention mechanism has become a promising way to improve the performance of CNNs for some computer vi…

Anomaly Detection

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

2026-02-01 · Zizhuo Fu, Wenxuan Zeng, Runsheng Wang, Meng Li arxiv

Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS …

Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models

2020-09-03 · Joseph F DeRose, Jiayao Wang, Matthew Berger

Advances in language modeling have led to the development of deep attention-based models that are performant across a wide variety of natural language processing (NLP) problems. These language models are typified by a pr…

Deep Attention Language Modeling Language Modelling Sentence