Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision

2024-07-01 · Hao Dong, Eleni Chatzi, Olga Fink

The task of open-set domain generalization (OSDG) involves recognizing novel classes within unseen domains, which becomes more challenging with multiple modalities as input. Existing works have only addressed unimodal OSDG within the meta-learning framework, without considering multimodal scenarios. In this work, we introduce a novel approach to address Multimodal Open-Set Domain Generalization (MM-OSDG) for the first time, utilizing self-supervision. To this end, we introduce two innovative multimodal self-supervised pretext tasks: Masked Cross-modal Translation and Multimodal Jigsaw Puzzles. These tasks facilitate the learning of multimodal representative features, thereby enhancing generalization and open-class detection capabilities. Additionally, we propose a novel entropy weighting mechanism to balance the loss across different modalities. Furthermore, we extend our approach to tackle also the Multimodal Open-Set Domain Adaptation (MM-OSDA) problem, especially in scenarios where unlabeled data from the target domain is available. Extensive experiments conducted under MM-OSDG, MM-OSDA, and Multimodal Closed-Set DG settings on the EPIC-Kitchens and HAC datasets demonstrate the efficacy and versatility of the proposed approach. Our source code is available at https://github.com/donghao51/MOOSA.

📄 PDF Abstract BibTeX arXiv:2407.01518

Code (1)

donghao51/moosa 공식 구현 pytorch

Tasks

Domain Adaptation Domain Generalization Meta-Learning

Methods 이 논문이 사용한 방법론

Jigsaw Jigsaw is a self-supervision approach that relies on jigsaw-like puzzles as the pretext task in order to learn image representations.

Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models

2025-01-30 · Hao Dong, Moru Liu, Kaiyang Zhou, Eleni Chatzi 외

In real-world scenarios, achieving domain adaptation and generalization poses significant challenges, as models must adapt to or generalize across unknown target distributions. Extending these capabilities to unseen mult…

Action Recognition Domain Adaptation Domain Generalization Semantic Segmentation+1

Multimodal Prompt Retrieval for Generative Visual Question Answering

2023-06-30 · Timothy Ossowski, Junjie Hu

Recent years have witnessed impressive results of pre-trained vision-language models on knowledge-intensive tasks such as visual question answering (VQA). Despite the recent advances in VQA, existing methods mainly adopt…

Domain Adaptation Generative Visual Question Answering Question Answering Retrieval+2

On the Out-Of-Distribution Generalization of Large Multimodal Models

2025-01-01 · CVPR 2025 1 · Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai 외

We investigate the generalization boundaries of current Large Multimodal Models (LMMs) via comprehensive evaluation under out-of-distribution scenarios and domain-specific tasks. We evaluate their zero-shot generaliz…

In-Context Learning Out-of-Distribution Generalization Zero-shot Generalization

On the Out-Of-Distribution Generalization of Multimodal Large Language Models

2024-02-09 · Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai 외

We investigate the generalization boundaries of current Multimodal Large Language Models (MLLMs) via comprehensive evaluation under out-of-distribution scenarios and domain-specific tasks. We evaluate their zero-shot gen…

In-Context Learning Out-of-Distribution Generalization Zero-shot Generalization

Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

2026-03-20 · Jiajie Li, Chenhui Xu, Meihuan Liu, Jinjun Xiong arxiv

Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation…

Reinforcement Learning