TLDR: Text Based Last-layer Retraining for Debiasing Image Classifiers
An image classifier may depend on incidental features stemming from a strong correlation between the feature and the classification target in the training dataset. Recently, Last Layer Retraining (LLR) with group-balanced datasets is shown to be efficient in mitigating the spurious correlation of classifiers. However, the acquisition of image-based group-balanced datasets is costly, which hinders the general applicability of the LLR method. In this work, we propose to perform LLR based on text datasets built with large language models to debias a general image classifier. To that end, we demonstrate that text can generally be a proxy for its corresponding image beyond the image-text joint embedding space, which is achieved with a linear projector that ensures orthogonality between its weight and the modality gap of the joint embedding space. In addition, we propose a systematic validation procedure that checks whether the generated words are compatible with the embedding space of CLIP and the image classifier, which is shown to be effective for improving debiasing performance. We dub these procedures as TLDR (Text-based Last layer retraining for Debiasing image classifieRs) and show our method achieves the performance that is competitive with the LLR methods that require group-balanced image dataset for retraining. Furthermore, TLDR outperforms other baselines that involve training the last layer without any group annotated dataset. Codes: https://github.com/beotborry/TLDR
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Effect of Pretraining on Extractive Summarization for Scientific Documents
Large pretrained models have seen enormous success in extractive summarization tasks. In this work, we investigate the influence of pretraining on a BERT-based extractive summarization system for scientific documents. We…
Extractive SummarizationWord EmbeddingsTLDR at SemEval-2022 Task 1: Using Transformers to Learn Dictionaries and Representations
We propose a pair of deep learning models, which employ unsupervised pretraining, attention mechanisms and contrastive learning for representation learning from dictionary definitions, and definition modeling from such r…
Contrastive LearningRepresentation LearningReverse DictionaryWord EmbeddingsTLDR: Token-Level Detective Reward Model for Large Vision Language Models
Although reward models have been successful in improving multimodal large language models, the reward models themselves remain brutal and contain minimal information. Notably, existing reward models only mimic human anno…
HallucinationHallucination EvaluationTLDR: Extreme Summarization of Scientific Documents
We introduce TLDR generation, a new form of extreme summarization, for scientific papers. TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-spec…
Abstractive Text SummarizationExtreme SummarizationTexture Learning Domain Randomization for Domain Generalized Segmentation
Deep Neural Networks (DNNs)-based semantic segmentation models trained on a source domain often struggle to generalize to unseen target domains, i.e., a domain gap problem. Texture often contributes to the domain gap, ma…
Domain GeneralizationSegmentationSemantic Segmentation