FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures
Forecasting motion and spatial positions of objects is of fundamental importance, especially in safety-critical settings such as autonomous driving. In this work, we address the issue by forecasting two different modalities that carry complementary information, namely optical flow and depth. To this end we propose FLODCAST a flow and depth forecasting model that leverages a multitask recurrent architecture, trained to jointly forecast both modalities at once. We stress the importance of training using flows and depth maps together, demonstrating that both tasks improve when the model is informed of the other modality. We train the proposed model to also perform predictions for several timesteps in the future. This provides better supervision and leads to more precise predictions, retaining the capability of the model to yield outputs autoregressively for any future time horizon. We test our model on the challenging Cityscapes dataset, obtaining state of the art results for both flow and depth forecasting. Thanks to the high quality of the generated flows, we also report benefits on the downstream task of segmentation forecasting, injecting our predictions in a flow-based mask-warping framework.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingOptical Flow EstimationSimilar Papers 제목 키워드 기반
A Hybrid Method for Traffic Flow Forecasting Using Multimodal Deep Learning
Traffic flow forecasting has been regarded as a key problem of intelligent transport systems. In this work, we propose a hybrid multimodal deep learning method for short-term traffic flow forecasting, which can jointly a…
Deep LearningMultimodal Deep LearningShort-term Streamflow and Flood Forecasting based on Graph Convolutional Recurrent Neural Network and Residual Error Learning
Accurate short-term streamflow and flood forecasting are critical for mitigating river flood impacts, especially given the increasing climate variability. Machine learning-based streamflow forecasting relies on large str…
Inference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer
Real-time video motion transfer applications such as immersive gaming and vision-based anomaly detection require accurate yet diverse future predictions to support realistic synthesis and robust downstream decision makin…
Time Series ForecastingAnomaly DetectionDecision MakingStateFlow: Dual-State Recurrent Modeling for Long-Horizon Time Series Forecasting
Long-horizon multivariate time series forecasting (LTSF) remains challenging due to non-stationarity, regime shifts, and error accumulation. The Variability-Aware Recursive Neural Network (VARNN) is designed to track suc…
Multivariate Time Series ForecastingMulti-Graph Convolutional-Recurrent Neural Network (MGC-RNN) for Short-Term Forecasting of Transit Passenger Flow
Short-term forecasting of passenger flow is critical for transit management and crowd regulation. Spatial dependencies, temporal dependencies, inter-station correlations driven by other latent factors, and exogenous fact…
DecoderManagement