DIFT: Dynamic Iterative Field Transforms for Memory Efficient Optical Flow
Recent advancements in neural network-based optical flow estimation often come with prohibitively high computational and memory requirements, presenting challenges in their model adaptation for mobile and low-power use cases. In this paper, we introduce a lightweight low-latency and memory-efficient model, Dynamic Iterative Field Transforms (DIFT), for optical flow estimation feasible for edge applications such as mobile, XR, micro UAVs, robotics and cameras. DIFT follows an iterative refinement framework leveraging variable resolution of cost volumes for correspondence estimation. We propose a memory efficient solution for cost volume processing to reduce peak memory. Also, we present a novel dynamic coarse-to-fine cost volume processing during various stages of refinement to avoid multiple levels of cost volumes. We demonstrate first real-time cost-volume based optical flow DL architecture on Snapdragon 8 Gen 1 HTP efficient mobile AI accelerator with 32 inf/sec and 5.89 EPE (endpoint error) on KITTI with manageable accuracy-performance tradeoffs.
Code (0)
등록된 구현이 없습니다.
Tasks
Optical Flow EstimationSimilar Papers 제목 키워드 기반
Generating Full-field Evolution of Physical Dynamics from Irregular Sparse Observations
Modeling and reconstructing multidimensional physical dynamics from sparse and off-grid observations presents a fundamental challenge in scientific research. Recently, diffusion-based generative modeling shows promising …
Computational EfficiencyDenoisingMedDIFT: Multi-Scale Diffusion-Based Correspondence in 3D Medical Imaging
Accurate spatial correspondence between medical images is essential for longitudinal analysis, lesion tracking, and image-guided interventions. Medical image registration methods rely on local intensity-based similarity …
Medical Image RegistrationFinetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion
Traditional knowledge graph (KG) completion models learn embeddings to predict missing facts. Recent works attempt to complete KGs in a text-generation manner with large language models (LLMs). However, they need to grou…
Knowledge Graph CompletionText GenerationEmergent Correspondence from Image Diffusion
Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strat…
Semantic correspondenceRobot-DIFT: Correspondence-Sensitive Diffusion Features for Contact-Rich Robot Manipulation
Robot manipulation often fails in the final millimeters: a policy may recognize the right object yet miss the pose offsets, boundaries, or pre-contact alignments needed for action. We argue that such failures arise when …
Robot Manipulation