Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation
Automatic radiology report generation has attracted enormous research interest due to its practical value in reducing the workload of radiologists. However, simultaneously establishing global correspondences between the image (e.g., Chest X-ray) and its related report and local alignments between image patches and keywords remains challenging. To this end, we propose an Unify, Align and then Refine (UAR) approach to learn multi-level cross-modal alignments and introduce three novel modules: Latent Space Unifier (LSU), Cross-modal Representation Aligner (CRA) and Text-to-Image Refiner (TIR). Specifically, LSU unifies multimodal data into discrete tokens, making it flexible to learn common knowledge among modalities with a shared network. The modality-agnostic CRA learns discriminative features via a set of orthonormal basis and a dual-gate mechanism first and then globally aligns visual and textual representations under a triplet contrastive loss. TIR boosts token-level local alignment via calibrating text-to-image attention with a learnable mask. Additionally, we design a two-stage training procedure to make UAR gradually grasp cross-modal alignments at different levels, which imitates radiologists' workflow: writing sentence by sentence first and then checking word by word. Extensive experiments and analyses on IU-Xray and MIMIC-CXR benchmark datasets demonstrate the superiority of our UAR against varied state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceTripletMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Semantic-Aligned Learning with Collaborative Refinement for Unsupervised VI-ReID
Unsupervised visible-infrared person re-identification (USL-VI-ReID) seeks to match pedestrian images of the same individual across different modalities without human annotations for model learning. Previous methods unif…
Contrastive LearningPerson Re-IdentificationPseudo LabelDeepAlignment: Unsupervised Ontology Matching with Refined Word Vectors
Ontologies compartmentalize types and relations in a target domain and provide the semantic backbone needed for a plethora of practical applications. Very often different ontologies are developed independently for the sa…
Entity AlignmentFeature EngineeringOntology MatchingRepresentation LearningCP3: Unifying Point Cloud Completion by Pretrain-Prompt-Predict Paradigm
Point cloud completion aims to predict complete shape from its partial observation. Current approaches mainly consist of generation and refinement stages in a coarse-to-fine style. However, the generation stage often lac…
Point Cloud CompletionPoint Cloud GenerationContextual Feedback Loops: Amplifying Deep Reasoning with Iterative Top-Down Feedback
Conventional deep networks rely on one-way backpropagation that overlooks reconciling high-level predictions with lower-level representations. We propose \emph{Contextual Feedback Loops} (CFLs), a lightweight mechanism t…
SST-2u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approac…
Instruction FollowingLanguage ModelingLanguage ModellingLarge Language Model