Multi-class Token Transformer for Weakly Supervised Semantic Segmentation

2022-03-06 · CVPR 2022 1 · Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, Dan Xu

This paper proposes a new transformer-based framework to learn class-specific object localization maps as pseudo labels for weakly supervised semantic segmentation (WSSS). Inspired by the fact that the attended regions of the one-class token in the standard vision transformer can be leveraged to form a class-agnostic localization map, we investigate if the transformer model can also effectively capture class-specific attention for more discriminative object localization by learning multiple class tokens within the transformer. To this end, we propose a Multi-class Token Transformer, termed as MCTformer, which uses multiple class tokens to learn interactions between the class tokens and the patch tokens. The proposed MCTformer can successfully produce class-discriminative object localization maps from class-to-patch attentions corresponding to different class tokens. We also propose to use a patch-level pairwise affinity, which is extracted from the patch-to-patch transformer attention, to further refine the localization maps. Moreover, the proposed framework is shown to fully complement the Class Activation Mapping (CAM) method, leading to remarkably superior WSSS results on the PASCAL VOC and MS COCO datasets. These results underline the importance of the class token for WSSS.

📄 PDF Abstract BibTeX arXiv:2203.02891

Code (1)

xulianuwa/mctformer 공식 구현 pytorch

Tasks

ObjectObject Localization Semantic Segmentation Weakly supervised Semantic Segmentation Weakly-Supervised Semantic Segmentation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation

2023-08-06 · Lian Xu, Mohammed Bennamoun, Farid Boussaid, Hamid Laga 외

This paper proposes a novel transformer-based framework that aims to enhance weakly supervised semantic segmentation (WSSS) by generating accurate class-specific object localization maps as pseudo labels. Building upon t…

Object Localization Semantic Segmentation Weakly supervised Semantic Segmentation Weakly-Supervised Semantic Segmentation

Class Tokens Infusion for Weakly Supervised Semantic Segmentation

2024-01-01 · CVPR 2024 1 · Sung-Hoon Yoon, Hoyong Kwon, Hyeonseong Kim, Kuk-Jin Yoon

Weakly Supervised Semantic Segmentation (WSSS) relies on Class Activation Maps (CAMs) to extract spatial information from image-level labels. With the success of Vision Transformer (ViT) the migration of ViT is activ…

Semantic Segmentation Weakly supervised Semantic Segmentation Weakly-Supervised Semantic Segmentation

TimeMIL: Advancing Multivariate Time Series Classification via a Time-aware Multiple Instance Learning

2024-05-06 · Xiwen Chen, Peijie Qiu, Wenhui Zhu, Huayu Li 외

Deep neural networks, including transformers and convolutional neural networks, have significantly improved multivariate time series classification (MTSC). However, these methods often rely on supervised learning, which …

Multiple Instance Learning Time Series Time Series Classification

Learning Multi-Modal Class-Specific Tokens for Weakly Supervised Dense Object Localization

2023-01-01 · CVPR 2023 1 · Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid 외

Weakly supervised dense object localization (WSDOL) relies generally on Class Activation Mapping (CAM), which exploits the correlation between the class weights of the image classifier and the pixel-level features. D…

Object Localization Representation Learning Semantic Segmentation Weakly supervised Semantic Segmentation+1

CLARiTy: A Vision Transformer for Multi-Label Classification and Weakly-Supervised Localization of Chest X-ray Pathologies

2025-12-18 · John M. Statheros, Hairong Wang, Richard Klein arxiv

The interpretation of chest X-rays (CXRs) poses significant challenges, particularly in achieving accurate multi-label pathology classification and spatial localization. These tasks demand different levels of annotation …

Multi-Label Classification