Step-Wise Hierarchical Alignment Network for Image-Text Matching
Image-text matching plays a central role in bridging the semantic gap between vision and language. The key point to achieve precise visual-semantic alignment lies in capturing the fine-grained cross-modal correspondence between image and text. Most previous methods rely on single-step reasoning to discover the visual-semantic interactions, which lacks the ability of exploiting the multi-level information to locate the hierarchical fine-grained relevance. Different from them, in this work, we propose a step-wise hierarchical alignment network (SHAN) that decomposes image-text matching into multi-step cross-modal reasoning process. Specifically, we first achieve local-to-local alignment at fragment level, following by performing global-to-local and global-to-global alignment at context level sequentially. This progressive alignment strategy supplies our model with more complementary and sufficient semantic clues to understand the hierarchical correlations between image and text. The experimental results on two benchmark datasets demonstrate the superiority of our proposed method.
Code (0)
등록된 구현이 없습니다.
Tasks
Image-text matchingText MatchingSimilar Papers 제목 키워드 기반
Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers
Text-to-image (T2I) generation models often struggle with multi-instance synthesis (MIS), where they must accurately depict multiple distinct instances in a single image based on complex prompts detailing individual feat…
AttributeLayout GenerationHAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation
Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We…
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language m…
cross-modal alignmentHallucinationMambaHyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on …
Long-Context UnderstandingCross-Modal RetrievalStructure-Preserving Zero-Shot Image Editing via Stage-Wise Latent Injection in Diffusion Models
We propose a diffusion-based framework for zero-shot image editing that unifies text-guided and reference-guided approaches without requiring fine-tuning. Our method leverages diffusion inversion and timestep-specific nu…
Attribute