PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification
Multi-label image classification is a prediction task that aims to identify more than one label from a given image. This paper considers the semantic consistency of the latent space between the visual patch and linguistic label domains and introduces the conditional transport (CT) theory to bridge the acknowledged gap. While recent cross-modal attention-based studies have attempted to align such two representations and achieved impressive performance, they required carefully-designed alignment modules and extra complex operations in the attention computation. We find that by formulating the multi-label classification as a CT problem, we can exploit the interactions between the image and label efficiently by minimizing the bidirectional CT cost. Specifically, after feeding the images and textual labels into the modality-specific encoders, we view each image as a mixture of patch embeddings and a mixture of label embeddings, which capture the local region features and the class prototypes, respectively. CT is then employed to learn and align those two semantic sets by defining the forward and backward navigators. Importantly, the defined navigators in CT distance model the similarities between patches and labels, which provides an interpretable tool to visualize the learned prototypes. Extensive experiments on three public image benchmarks show that the proposed model consistently outperforms the previous methods.
Code (1)
Tasks
image-classificationImage ClassificationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONMulti-Label Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PatchCTG: Patch Cardiotocography Transformer for Antepartum Fetal Health Monitoring
Antepartum Cardiotocography (CTG) is vital for fetal health monitoring, but traditional methods like the Dawes-Redman system are often limited by high inter-observer variability, leading to inconsistent interpretations a…
Decision MakingSensitivitySpecificityPatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels
Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models' predictions needs to be addressed before deploying them. We introduce a novel approach,…
Fairnessimage-classificationImage ClassificationWasserstein Geodesic Generator for Conditional Distributions
Generating samples given a specific label requires estimating conditional distributions. We derive a tractable upper bound of the Wasserstein distance between conditional distributions to lay the theoretical groundwork t…
Model alignment using inter-modal bridges
Foundation models have demonstrated remarkable performance across modalities such as language and vision. However, model reuse across distinct modalities (e.g., text and vision) remains limited due to the difficulty of a…
Image GenerationmodelObject RecognitionSemantic-Aware Representation Learning via Conditional Transport for Multi-Label Image Classification
Multi-label image classification is a critical task in machine learning that aims to accurately assign multiple labels to a single image. While existing methods often utilize attention mechanisms or graph convolutional n…
Multi-Label Image ClassificationRepresentation Learning