Predicting Future Instance Segmentation by Forecasting Convolutional Features

2018-03-30 · ECCV 2018 9 · Pauline Luc, Camille Couprie, Yann Lecun, Jakob Verbeek

Anticipating future events is an important prerequisite towards intelligent behavior. Video forecasting has been studied as a proxy task towards this goal. Recent work has shown that to predict semantic segmentation of future frames, forecasting at the semantic level is more effective than forecasting RGB frames and then segmenting these. In this paper we consider the more challenging problem of future instance segmentation, which additionally segments out individual objects. To deal with a varying number of output labels per image, we develop a predictive model in the space of fixed-sized convolutional features of the Mask R-CNN instance segmentation model. We apply the "detection head'" of Mask R-CNN on the predicted features to produce the instance segmentation of future frames. Experiments show that this approach significantly improves over strong baselines based on optical flow and repurposed instance segmentation architectures.

📄 PDF Abstract BibTeX arXiv:1803.11496

Code (1)

facebookresearch/instpred pytorch

Tasks

Instance Segmentation Optical Flow Estimation Segmentation Semantic SegmentationVideo ForecastingVideo Prediction

Methods 이 논문이 사용한 방법론

RPN A Region Proposal Network, or RPN, is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

RoIAlign Region of Interest Align, or RoIAlign, is an operation for extracting a small feature map from each RoI in detection and segmentation based tasks. It removes the harsh…

Mask R-CNN Mask R-CNN extends Faster R-CNN to solve instance segmentation tasks. It achieves this by adding a branch for predicting an…

Forecasting Future Instance Segmentation with Learned Optical Flow and Warping

2022-11-15 · Andrea Ciamarra, Federico Becattini, Lorenzo Seidenari, Alberto del Bimbo

For an autonomous vehicle it is essential to observe the ongoing dynamics of a scene and consequently predict imminent future scenarios to ensure safety to itself and others. This can be done using different sensors and …

Instance Segmentation Optical Flow Estimation Semantic Segmentation

Finding Islands of Predictability in Action Forecasting

2022-10-13 · Daniel Scarafoni, Irfan Essa, Thomas Ploetz

We address dense action forecasting: the problem of predicting future action sequence over long durations based on partial observation. Our key insight is that future action sequences are more accurately modeled with var…

Predicting Deeper into the Future of Semantic Segmentation

2017-03-22 · ICCV 2017 10 · Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek 외

The ability to predict and therefore to anticipate the future is an important attribute of intelligence. It is also of utmost importance in real-time systems, e.g. in robotics or autonomous driving, which depend on visua…

Attribute Autonomous Driving Decision Making Optical Flow Estimation+3

Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion

2021-01-26 · Josip Šarić, Sacha Vražić, Siniša Šegvić

Dense semantic forecasting anticipates future events in video by inferring pixel-level semantics of an unobserved future image. We present a novel approach that is applicable to various single-frame architectures and tas…

Future prediction Panoptic Segmentation regression Segmentation+1

CASNet: Common Attribute Support Network for image instance and panoptic segmentation

2020-07-17 · Xiaolong Liu, Yuqing Hou, Anbang Yao, Yurong Chen 외

Instance segmentation and panoptic segmentation is being paid more and more attention in recent years. In comparison with bounding box based object detection and semantic segmentation, instance segmentation can provide m…

Attribute Clustering Instance Segmentation object-detection+4