Multi-modal Deep Analysis for Multimedia
With the rapid development of Internet and multimedia services in the past decade, a huge amount of user-generated and service provider-generated multimedia data become available. These data are heterogeneous and multi-modal in nature, imposing great challenges for processing and analyzing them. Multi-modal data consist of a mixture of various types of data from different modalities such as texts, images, videos, audios etc. In this article, we present a deep and comprehensive overview for multi-modal analysis in multimedia. We introduce two scientific research problems, data-driven correlational representation and knowledge-guided fusion for multimedia analysis. To address the two scientific problems, we investigate them from the following aspects: 1) multi-modal correlational representation: multi-modal fusion of data across different modalities, and 2) multi-modal data and knowledge fusion: multi-modal fusion of data with domain knowledge. More specifically, on data-driven correlational representation, we highlight three important categories of methods, such as multi-modal deep representation, multi-modal transfer learning, and multi-modal hashing. On knowledge-guided fusion, we discuss the approaches for fusing knowledge with data and four exemplar applications that require various kinds of domain knowledge, including multi-modal visual question answering, multi-modal video summarization, multi-modal visual pattern mining and multi-modal recommendation. Finally, we bring forward our insights and future research directions.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-modal RecommendationQuestion AnsweringTransfer LearningVideo SummarizationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Formalizing Multimedia Recommendation through Multimodal Deep Learning
Recommender systems (RSs) offer personalized navigation experiences on online platforms, but recommendation remains a challenging task, particularly in specific scenarios and domains. Multimodality can help tap into rich…
BenchmarkingDeep LearningMultimedia recommendationMultimodal Deep Learning+1Research on the Brain-inspired Cross-modal Neural Cognitive Computing Framework
To address modeling problems of brain-inspired intelligence, this thesis is focused on researching in the semantic-oriented framework design for multimedia and multimodal information. The Multimedia Neural Cognitive Comp…
Multimedia Verification Through Multi-Agent Deep Research Multimodal Large Language Models
This paper presents our submission to the ACMMM25 - Grand Challenge on Multimedia Verification. We developed a multi-agent verification system that combines Multimodal Large Language Models (MLLMs) with specialized verif…
Information ExtractionComputational Linguistics Applications for Multimedia Services
We present Computational Linguistics Applications for Multimedia Services (CLAMS), a platform that provides access to computational content analysis tools for archival multimedia material that appear in different media, …
Evaluation Pitfalls and Challenges in Multimedia Event Extraction
Multimedia event extraction aims to jointly identify events and their arguments across multiple modalities, such as text and images, to support more comprehensive event understanding. While recent work reports steady and…
Event Extraction