Papers multimodal interaction
“multimodal interaction” 태그가 달린 논문 106편 · 필터 해제
A multi-stage augmented multimodal interaction network for fish feeding intensity quantification
In recirculating aquaculture systems, accurate and effective assessment of fish feeding intensity is crucial for reducing feed costs and calculating optimal feeding times. However, current studies have limitations in mod…
Decision Makingmultimodal interactionStream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
The emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal interaction. Existing LMMs typically concatena…
Large Language Modelmultimodal interactionInterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
As multimodal large models (MLLMs) continue to advance across challenging tasks, a key question emerges: What essential capabilities are still missing? A critical aspect of human learning is continuous interaction with t…
multimodal interactionChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understanding poses significant challenges to existing multimodal larg…
Chart UnderstandingLogical Reasoningmultimodal interactionVisual ReasoningI2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions …
Mixture-of-Expertsmultimodal interactionDeepSORT-Driven Visual Tracking Approach for Gesture Recognition in Interactive Systems
Based on the DeepSORT algorithm, this study explores the application of visual tracking technology in intelligent human-computer interaction, especially in the field of gesture recognition and tracking. With the rapid de…
Gesture Recognitionmultimodal interactionVisual TrackingMing-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction
We introduce Ming-Lite-Uni, an open-source multimodal framework featuring a newly designed unified visual generator and a native multimodal autoregressive model tailored for unifying vision and language. Specifically, th…
Image Generationmultimodal interactionText to Image GenerationText-to-Image GenerationA Survey of Interactive Generative Video
Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology th…
Autonomous Drivingmultimodal interactionSurveyImmersive Multimedia Communication: State-of-the-Art on eXtended Reality Streaming
Extended reality (XR) is rapidly advancing, and poised to revolutionize content creation and consumption. In XR, users integrate various sensory inputs to form a cohesive perception of the virtual environment. This surve…
multimodal interactionOVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer
Open-vocabulary multiple object tracking aims to generalize trackers to unseen categories during training, enabling their application across a variety of real-world scenarios. However, the existing open-vocabulary tracke…
Decodermultimodal interactionMultiple Object TrackingMultiple Object Tracking with Transformer+1ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting
Efficient and privacy-preserving multimodal interaction is essential as AR, VR, and modern smartphones with powerful cameras become primary interfaces for human-computer communication. Existing powerful large vision-lang…
Image Captioningmultimodal interactionPrivacy PreservingSemantic ParsingInteractive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving
Humans have long relied on visual aids like sketches and diagrams to support reasoning and problem-solving. Visual tools, like auxiliary lines in geometry or graphs in calculus, are essential for understanding complex id…
Mathmultimodal interactionTowards Explainable Multimodal Depression Recognition for Clinical Interviews
Recently, multimodal depression recognition for clinical interviews (MDRC) has recently attracted considerable attention. Existing MDRC studies mainly focus on improving task performance and have achieved significant dev…
Decision MakingDepression DetectionExplainable artificial intelligenceMedical Diagnosis+3FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection
Multimodal 3D object detection has garnered considerable interest in autonomous driving. However, multimodal detectors suffer from dimension mismatches that derive from fusing 3D points with 2D pixels coarsely, which lea…
3D Object DetectionAutonomous Drivingmultimodal interactionobject-detection+1MODfinity: Unsupervised Domain Adaptation with Multimodal Information Flow Intertwining
Multimodal unsupervised domain adaptation leverages unlabeled data in the target domain to enhance multimodal systems continuously. While current state-of-the-art methods encourage interaction between sub-models of d…
Domain AdaptationModel Selectionmultimodal interactionUnsupervised Domain AdaptationComputer Vision-Driven Gesture Recognition: Toward Natural and Intuitive Human-Computer
This study mainly explores the application of natural gesture recognition based on computer vision in human-computer interaction, aiming to improve the fluency and naturalness of human-computer interaction through gestur…
Gesture Recognitionmultimodal interactionGenerative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability
As the boundaries of human computer interaction expand, Generative AI emerges as a key driver in reshaping user interfaces, introducing new possibilities for personalized, multimodal and cross-platform interactions. This…
multimodal interactionCMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation
Multimodal emotion recognition in conversation (MER) aims to accurately identify emotions in conversational utterances by integrating multimodal information. Previous methods usually treat multimodal information as equal…
Emotion RecognitionEmotion Recognition in ConversationMultimodal Emotion Recognitionmultimodal interactionSpider: Any-to-Many Multimodal LLM
Multimodal LLMs (MLLMs) have emerged as an extension of Large Language Models (LLMs), enabling the integration of various modalities. However, Any-to-Any MLLMs are limited to generating pairwise modalities 'Text + X' wit…
multimodal interactionMIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval
Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-re…
Image ComprehensionInformation Retrievalmultimodal interactionRetrieval