Papers cross-modal alignment
“cross-modal alignment” 태그가 달린 논문 342편 · 필터 해제
Transformer-based Spatial Grounding: A Comprehensive Survey
Spatial grounding, the process of associating natural language expressions with corresponding image regions, has rapidly advanced due to the introduction of transformer-based models, significantly enhancing multimodal re…
cross-modal alignmentSurveySystematic Literature ReviewCATVis: Context-Aware Thought Visualization
EEG-based brain-computer interfaces (BCIs) have shown promise in various applications, such as motor imagery and cognitive state monitoring. However, decoding visual representations from EEG signals remains a significant…
cross-modal alignmentEEGImage GenerationMotor Imagery+1Bridge Feature Matching and Cross-Modal Alignment with Mutual-filtering for Zero-shot Anomaly Detection
With the advent of vision-language models (e.g., CLIP) in zero- and few-shot settings, CLIP has been widely applied to zero-shot anomaly detection (ZSAD) in recent research, where the rare classes are essential and expec…
Anomaly ClassificationAnomaly Detectioncross-modal alignmentzero-shot anomaly detectionEvaluating Attribute Confusion in Fashion Text-to-Image Generation
Despite the rapid advances in Text-to-Image (T2I) generation models, their evaluation remains challenging in domains like fashion, involving complex compositional generation. Recent automated T2I evaluation methods lever…
Attributecross-modal alignmentImage GenerationQuestion Answering+5Skywork-R1V3 Technical Report
We introduce Skywork-R1V3, an advanced, open-source vision-language model (VLM) that pioneers a new approach to visual reasoning. Its key innovation lies in effectively transferring reasoning skills from text-only Large …
cross-modal alignmentMathematical ReasoningMultimodal ReasoningVisual ReasoningRSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models
Referring Remote Sensing Image Segmentation provides a flexible and fine-grained framework for remote sensing scene analysis via vision-language collaborative interpretation. Current approaches predominantly utilize a th…
cross-modal alignmentImage SegmentationSegmentationSemantic SegmentationDeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following, without requiring task-specific audio instruction-tuning. Recent LALMs t…
cross-modal alignmentInstruction FollowingLanguage ModelingLanguage Modelling+1Flash-VStream: Efficient Real-Time Understanding for Long Video Streams
Benefiting from the advances in large language models and cross-modal alignment, existing multimodal large language models have achieved prominent performance in image and short video understanding. However, the understa…
cross-modal alignmentEgoSchemaMMEMVBench+2TSDASeg: A Two-Stage Model with Direct Alignment for Interactive Point Cloud Segmentation
The rapid advancement of 3D vision-language models (VLMs) has spurred significant interest in interactive point cloud processing tasks, particularly for real-world applications. However, existing methods often underperfo…
cross-modal alignmentInteractive SegmentationPoint Cloud SegmentationSegmentation+1DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
Previous multimodal sentence representation learning methods have achieved impressive performance. However, most approaches focus on aligning images and text at a coarse level, facing two critical challenges:cross-modal …
cross-modal alignmentRepresentation LearningSemantic SimilaritySemantic Textual Similarity+2HyperPath: Knowledge-Guided Hyperbolic Semantic Hierarchy Modeling for WSI Analysis
Pathology is essential for cancer diagnosis, with multiple instance learning (MIL) widely used for whole slide image (WSI) analysis. WSIs exhibit a natural hierarchy -- patches, regions, and slides -- with distinct seman…
cross-modal alignmentMultiple Instance LearningSpeech-Language Models with Decoupled Tokenizers and Multi-Token Prediction
Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation. However, challenges remain in achieving effective cross-modal alignment and high-quality speech generatio…
cross-modal alignmentTAViS: Text-bridged Audio-Visual Segmentation with Foundation Models
Audio-Visual Segmentation (AVS) faces a fundamental challenge of effectively aligning audio and visual modalities. While recent approaches leverage foundation models to address data scarcity, they often rely on single-mo…
cross-modal alignmentSegmentationImproving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration
Learning medical visual representations from image-report pairs through joint learning has garnered increasing research attention due to its potential to alleviate the data scarcity problem in the medical domain. The pri…
cross-modal alignmentImage to textImage-to-Text Retrievalobject-detection+4ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model
In real-word scenarios, person re-identification (ReID) expects to identify a person-of-interest via the descriptive query, regardless of whether the query is a single modality or a combination of multiple modalities. Ho…
cross-modal alignmentDescriptivePerson Re-IdentificationOmniDRCA: Parallel Speech-Text Foundation Model via Dual-Resolution Speech Representations and Contrastive Alignment
Recent studies on end-to-end speech generation with large language models (LLMs) have attracted significant community attention, with multiple works extending text-based LLMs to generate discrete speech tokens. Existing …
cross-modal alignmentQuestion AnsweringSpeech SynthesisText GenerationFusing Cross-modal and Uni-modal Representations: A Kronecker Product Approach
Cross-modal embeddings, such as CLIP, BLIP and their variants, have achieved promising results in aligning representations across modalities. However, these embeddings could underperform compared to state-of-the-art sing…
cross-modal alignmentGenerating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations
Vision-Language Navigation (VLN) enables intelligent agents to navigate environments by integrating visual perception and natural language instructions, yet faces significant challenges due to the scarcity of fine-graine…
cross-modal alignmentNavigateVision-Language NavigationWhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction
Mean Opinion Score (MOS) prediction for text to music systems requires evaluating both overall musical quality and text prompt alignment. This paper introduces WhisQ, a multimodal architecture that addresses this dual-as…
cross-modal alignmentLanguage ModelingLanguage ModellingRepresentation Learning+1Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
The rapid progress of Multimodal Large Language Models(MLLMs) has transformed the AI landscape. These models combine pre-trained LLMs with various modality encoders. This integration requires a systematic understanding o…
cross-modal alignmentLarge Language ModelMultimodal Large Language ModelRepresentation Learning