SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
Vision-language models (VLMs) have made significant strides in cross-modal understanding through large-scale paired datasets. However, in fashion domain, datasets often exhibit a disparity between the information conveyed in image and text. This issue stems from datasets containing multiple images of a single fashion item all paired with one text, leading to cases where some textual details are not visible in individual images. This mismatch, particularly when non-co-occurring elements are masked, undermines the training of conventional VLM objectives like Masked Language Modeling and Masked Image Modeling, thereby hindering the model's ability to accurately align fine-grained visual and textual features. Addressing this problem, we propose Synchronized attentional Masking (SyncMask), which generate masks that pinpoint the image patches and word tokens where the information co-occur in both image and text. This synchronization is accomplished by harnessing cross-attentional features obtained from a momentum model, ensuring a precise alignment between the two modalities. Additionally, we enhance grouped batch sampling with semi-hard negatives, effectively mitigating false negative issues in Image-Text Matching and Image-Text Contrastive learning objectives within fashion datasets. Our experiments demonstrate the effectiveness of the proposed approach, outperforming existing methods in three downstream tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningImage-text matchingLanguage ModelingLanguage ModellingMasked Language ModelingText MatchingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MV-Fashion: Towards Enabling Virtual Try-On and Size Estimation with Multi-View Paired Data
Existing 4D human datasets fall short for fashion-specific research, lacking either realistic garment dynamics or task-specific annotations. Synthetic datasets suffer from a realism gap, whereas real-world captures lack …
Novel View SynthesisVirtual Try-onPoint CloudsWhere to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network
Fashion products typically feature in compositions of a variety of styles at different clothing parts. In order to distinguish images of different fashion products, we need to extract both appearance (i.e., "how to descr…
Image RetrievalRetrievalEnhancing Egocentric 3D Pose Estimation with Third Person Views
In this paper, we propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The key idea is to leverage high-level features linking first- an…
3D Pose EstimationDomain AdaptationPose EstimationRetailSMV: Exocentric vs. Egocentric Adaptation of Foundation Video World Models in Retail
Foundation video diffusion models are increasingly viewed as world simulators for embodied agents, yet their pretraining on internet-scale generic video leaves them poorly aligned with real-world deployment domains. We s…
EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos
Egocentric memory is widely used in embodied intelligence, but it may be insufficient for comprehensive spatial-temporal reasoning. Inspired by human recall from both field and observer perspectives, we introduce EgoExoM…