Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding
Hierarchical structures are popular in recent vision transformers, however, they require sophisticated designs and massive datasets to work well. In this paper, we explore the idea of nesting basic local transformers on non-overlapping image blocks and aggregating them in a hierarchical way. We find that the block aggregation function plays a critical role in enabling cross-block non-local information communication. This observation leads us to design a simplified architecture that requires minor code changes upon the original vision transformer. The benefits of the proposed judiciously-selected design are threefold: (1) NesT converges faster and requires much less training data to achieve good generalization on both ImageNet and small datasets like CIFAR; (2) when extending our key ideas to image generation, NesT leads to a strong decoder that is 8$\times$ faster than previous transformer-based generators; and (3) we show that decoupling the feature learning and abstraction processes via this nested hierarchy in our design enables constructing a novel method (named GradCAT) for visually interpreting the learned model. Source code is available https://github.com/google-research/nested-transformer.
Code (6)
Tasks
DecoderImage ClassificationImage GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
HiTRANS: A Hierarchical Transformer Network for Nested Named Entity Recognition
Nested Named Entity Recognition (NNER) has been extensively studied, aiming to identify all nested entities from potential spans (i.e., one or more continuous tokens). However, recent studies for NNER either focus on ted…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Nested Named Entity Recognition+2Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing
Transformer has been applied in the field of computer vision due to its excellent performance in natural language processing, surpassing traditional convolutional neural networks and achieving new state-of-the-art. ViT d…
image-classificationImage ClassificationSentenceHierarchical Reinforcement Learning with Deep Nested Agents
Deep hierarchical reinforcement learning has gained a lot of attention in recent years due to its ability to produce state-of-the-art results in challenging environments where non-hierarchical frameworks fail to learn us…
Hierarchical Reinforcement LearningMinecraftreinforcement-learningReinforcement Learning+1Sparse Hierarchical Tucker Factorization and its Application to Healthcare
We propose a new tensor factorization method, called the Sparse Hierarchical-Tucker (Sparse H-Tucker), for sparse and high-order data tensors. Sparse H-Tucker is inspired by its namesake, the classical Hierarchical Tucke…
Forming Trees with Treeformers
Human language is known to exhibit a nested, hierarchical structure, allowing us to form complex sentences out of smaller pieces. However, many state-of-the-art neural networks models such as Transformers have no explici…
Abstractive Text SummarizationInductive BiasMachine TranslationNatural Language Understanding+1