Papers multimodal generation
“multimodal generation” 태그가 달린 논문 188편 · 필터 해제
LongLive-Plug: Once-for-All Distillation for Video Generation
Video diffusion models are increasingly developed into specialized models for diverse downstream tasks, and this development often includes a distillation stage, for example to accelerate sampling or to improve long-vide…
multimodal generationVideo GenerationFOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding
Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically conditio…
multimodal generationVisual GroundingG-CARL: Grounded Checklist-Aligned Reward Learning for Patient-Oriented Medical Report Interpretation
Personalized interpretation of medical reports has emerged as an increasingly important need among patients. Addressing this need requires both evidence-grounded medical factuality and context-dependent patient communica…
Reinforcement Learningmultimodal generationSPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation
Spatial perception and reasoning from visual observations require recovering geometric structure, establishing correspondences, and understanding spatial relations. Existing approaches typically address these capabilitie…
multimodal generationSpatial Reasoning3D ReconstructionM$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data
Integrating heterogeneous biomedical data, including clinical metadata, histopathology images, and molecular profiles, is crucial for comprehensive disease understanding. However, gene expression data acquisition remains…
multimodal generationContrastive LearningVQ-Touch: A Data-Efficient Tactile Generation Framework Across Sensors and Scenarios
Tactile image generation significantly reduces the dependency on expensive and wear-prone sensors by synthesizing high-fidelity tactile data, offering an efficient solution for tactile information acquisition in robotic …
multimodal generationImage GenerationConcurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes
Human cognition does not separate understanding and generation. A teacher at a whiteboard speaks and draws together, each modality reshapes the other. In this paper, we bring this coupled loop to artificial systems. Mask…
multimodal generationVisual ReasoningVision as Unified Multimodal Generation
We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectu…
Camera Pose Estimationmultimodal generationDepth EstimationImage GenerationUnified Audio Intelligence Without Regressing on Text Intelligence
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A…
multimodal generationSpeech RecognitionAudio GenerationAttending to Multimodal Generation One Token at a Time
Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circ…
multimodal generationWorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling
World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling. Meanwhile, unified vi…
multimodal generationAn Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens…
multimodal generationCan MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): it has no single correct answer, and most aesthetic evaluation measures models against numeric scores rather than the written cri…
multimodal generationVPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks
Recent advancements in Image-to-Video (I2V) generation have transformed input images from simple appearance references into interactive control interfaces where visual cues such as arrows, sketches, and emojis orchestrat…
multimodal generationVideo GenerationNavigating User Behavior toward Personalized Multimodal Generation
Modern AIGC pipelines deliver high-fidelity images and videos but presuppose a well-formed creation instruction, while end users rarely articulate visual details, leaving generators misaligned with user demand. We study …
multimodal generationVideo GenerationTailorMind: Towards Preference-Aligned Multimodal Content Generation
Personalized content systems depend on available UGC and struggle when suitable content is absent, delayed, or costly to create. Although multimodal generators can synthesize content on demand, how to translate behaviora…
Collaborative Filteringmultimodal generationPareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration
Unified multimodal models (UMMs) have recently emerged as a promising paradigm for integrating multimodal understanding and generation within a single autoregressive transformer. However, during multimodal instruction tu…
parameter-efficient fine-tuningmultimodal generationImage GenerationSoul Computing: A Theoretical Framework and Technical Architecture for Intelligent Agents with Independent Consciousness
Breakthroughs in large language models and multimodal generation technologies have propelled the digital reconstruction of human mental traits, emotional patterns, and long-term memory from science fiction toward enginee…
multimodal generationResonant Minds: Closed-Loop Social Avatars with Theory of Mind
Creating lifelike digital humans with genuine social intelligence requires unifying cognitive reasoning and multimodal generation within a coherent framework. Current approaches treat these as separate tasks: Large Langu…
multimodal generationVideo GenerationUniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation
Recent years have seen remarkable progress in unified vision-language models handling both multimodal understanding and generation within a single architecture. While autoregressive VLMs can reason across modalities, the…
multimodal generationImage GenerationText Generation