CompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices
We present CompactFlowNet, the first real-time mobile neural network for optical flow prediction, which involves determining the displacement of each pixel in an initial frame relative to the corresponding pixel in a subsequent frame. Optical flow serves as a fundamental building block for various video-related tasks, such as video restoration, motion estimation, video stabilization, object tracking, action recognition, and video generation. While current state-of-the-art methods prioritize accuracy, they often overlook constraints regarding speed and memory usage. Existing light models typically focus on reducing size but still exhibit high latency, compromise significantly on quality, or are optimized for high-performance GPUs, resulting in sub-optimal performance on mobile devices. This study aims to develop a mobile-optimized optical flow model by proposing a novel mobile device-compatible architecture, as well as enhancements to the training pipeline, which optimize the model for reduced weight, low memory utilization, and increased speed while maintaining minimal error. Our approach demonstrates superior or comparable performance to the state-of-the-art lightweight models on the challenging KITTI and Sintel benchmarks. Furthermore, it attains a significantly accelerated inference speed, thereby yielding real-time operational efficiency on the iPhone 8, while surpassing real-time performance levels on more advanced mobile devices.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionMotion EstimationObject TrackingOptical Flow EstimationVideo GenerationVideo RestorationVideo StabilizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MemFlow: Optical Flow Estimation and Prediction with Memory
Optical flow is a classical task that is important to the vision community. Classical optical flow estimation uses two frames as input, whilst some recent methods consider multiple frames to explicitly model long-range i…
Future predictionOptical Flow EstimationPredictionFlowDA: Unsupervised Domain Adaptive Framework for Optical Flow Estimation
Collecting real-world optical flow datasets is a formidable challenge due to the high cost of labeling. A shortage of datasets significantly constrains the real-world performance of optical flow models. Building virtual …
Dataset GenerationOptical Flow EstimationSciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations
Optical flow estimation is crucial to a variety of vision tasks. Despite substantial recent advancements, achieving real-time on-device optical flow estimation remains a complex challenge. First, an optical flow model mu…
Optical Flow EstimationregressionA Framework for Real-Time Face and Facial Feature Tracking using Optical Flow Pre-estimation and Template Tracking
This work presents a framework for tracking head movements and capturing the movements of the mouth and both the eyebrows in real-time. We present a head tracker which is a combination of a optical flow and a template ba…
Optical Flow EstimationDeGraF-Flow: Extending DeGraF Features for accurate and efficient sparse-to-dense optical flow estimation
Modern optical flow methods make use of salient scene feature points detected and matched within the scene as a basis for sparse-to-dense optical flow estimation. Current feature detectors however either give sparse, non…
Computational EfficiencyGPUOptical Flow Estimation