Replay: Multi-modal Multi-view Acted Videos for Casual Holography
We introduce Replay, a collection of multi-view, multi-modal videos of humans interacting socially. Each scene is filmed in high production quality, from different viewpoints with several static cameras, as well as wearable action cameras, and recorded with a large array of microphones at different positions in the room. Overall, the dataset contains over 4000 minutes of footage and over 7 million timestamped high-resolution frames annotated with camera poses and partially with foreground masks. The Replay dataset has many potential applications, such as novel-view synthesis, 3D reconstruction, novel-view acoustic synthesis, human body and face analysis, and training generative models. We provide a benchmark for training and evaluating novel-view synthesis, with two scenarios of different difficulty. Finally, we evaluate several baseline state-of-the-art methods on the new benchmark.
Code (1)
Tasks
3D ReconstructionNovel View SynthesisSimilar Papers 제목 키워드 기반
Towards Universal Soccer Video Understanding
As a globally celebrated sport, soccer has attracted widespread interest from fans all over the world. This paper aims to develop a comprehensive multi-modal framework for soccer video understanding. Specifically, we mak…
Action ClassificationSports UnderstandingVideo UnderstandingOnline Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks
Online Continual Learning (OCL) is a critical area in machine learning, focusing on enabling models to adapt to evolving data streams in real-time while addressing challenges such as catastrophic forgetting and the stabi…
Continual Learningimage-classificationImage Classificationobject-detection+3A Review Paper of the Effects of Distinct Modalities and ML Techniques to Distracted Driving Detection
Distracted driving remains a significant global challenge with severe human and economic repercussions, demanding improved detection and intervention strategies. While previous studies have extensively explored single-mo…
Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning
Food analysis has become increasingly critical for health-related tasks such as personalized nutrition and chronic disease prevention. However, existing large multimodal models (LMMs) in food analysis suffer from catastr…
Semantic SimilarityContinual LearningAttention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions, yet continual fine-tuning often causes severe forgetting of …
Continual Learning