홈 › Papers

DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling

2025-12-02 · Kairun Wen, Yuzhi Huang, Runyu Chen, Hui Zheng, Yunlong Lin, Panwang Pan, Chenxin Li, Wenyan Cong, Jian Zhang, Junbin Lu, Chenguo Lin, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Yue Huang, Xinghao Ding, Rakesh Ranjan, Zhiwen Fan arxiv

Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.

📄 PDF Abstract BibTeX arXiv:2512.03000

Code (0)

등록된 구현이 없습니다.

Tasks

Camera Pose Estimation Depth Estimation

Spectral-Aware Text-to-Time Series Generation with Billion-Scale Multimodal Meteorological Data

2026-03-28 · Shijie Zhang arxiv

Text-to-time-series generation is particularly important in meteorology, where natural language offers intuitive control over complex, multi-scale atmospheric dynamics. Existing approaches are constrained by the lack of …

Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

2026-04-14 · Yingying Zhao, Chengyin Hu, Qike Zhang, Xin Li 외 arxiv

Vision-Language Models (VLMs) have shown remarkable performance, yet their security remains insufficiently understood. Existing adversarial studies focus almost exclusively on the digital setting, leaving physical-world …

Visual Question Answering Multimodal Reasoning Adversarial Attack Image Captioning

RynnBrain: Open Embodied Foundation Models

2026-02-13 · Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng 외 arxiv

Despite rapid progress in multimodal foundation models, embodied intelligence community still lacks a unified, physically grounded foundation model that integrates perception, reasoning, and planning within real-world sp…

Spatial Reasoning

Weakly supervised multimodal segmentation of acoustic borehole images with depth-aware cross-attention

2026-03-21 · Jose Luis Lima de Jesus Silva arxiv

Acoustic borehole images provide high-resolution borehole-wall structure, but large-scale interpretation remains difficult because dense expert annotations are rarely available and subsurface information is intrinsically…

MambaFusion: Adaptive State-Space Fusion for Multimodal 3D Object Detection

2026-02-08 · Venkatraman Narayanan, Bala Sai, Rahul Ahuja, Pratik Likhar 외 arxiv

Reliable 3D object detection is fundamental to autonomous driving, and multimodal fusion algorithms using cameras and LiDAR remain a persistent challenge. Cameras provide dense visual cues but ill posed depth; LiDAR prov…

3D Object Detection Autonomous Driving