StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation
We present StableMotion, a novel framework leverages knowledge (geometry and content priors) from pretrained large-scale image diffusion models to perform motion estimation, solving single-image-based image rectification tasks such as Stitched Image Rectangling (SIR) and Rolling Shutter Correction (RSC). Specifically, StableMotion framework takes text-to-image Stable Diffusion (SD) models as backbone and repurposes it into an image-to-motion estimator. To mitigate inconsistent output produced by diffusion models, we propose Adaptive Ensemble Strategy (AES) that consolidates multiple outputs into a cohesive, high-fidelity result. Additionally, we present the concept of Sampling Steps Disaster (SSD), the counterintuitive scenario where increasing the number of sampling steps can lead to poorer outcomes, which enables our framework to achieve one-step inference. StableMotion is verified on two image rectification tasks and delivers state-of-the-art performance in both, as well as showing strong generalizability. Supported by SSD, StableMotion offers a speedup of 200 times compared to previous diffusion model-based methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Motion EstimationRolling Shutter CorrectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
StableMotion: Training Motion Cleanup Models with Unpaired Corrupted Data
Motion capture (mocap) data often exhibits visually jarring artifacts due to inaccurate sensors and post-processing. Cleaning this corrupted data can require substantial manual effort from human experts, which can be a c…
Motion GenerationText2Stereo: Repurposing Stable Diffusion for Stereo Generation with Consistency Rewards
In this paper, we propose a novel diffusion-based approach to generate stereo images given a text prompt. Since stereo image datasets with large baselines are scarce, training a diffusion model from scratch is not feasib…
TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking
Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging a…
Scene Understanding3D ReconstructionFreqPhys: Repurposing Implicit Physiological Frequency Prior for Robust Remote Photoplethysmography
Remote photoplethysmography (rPPG) enables contactless physiological monitoring by capturing subtle skin-color variations from facial videos. However, most existing methods predominantly rely on time-domain modeling, mak…
Representation LearningDVD: Deterministic Video Depth Estimation with Generative Priors
Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve sem…
Depth Estimation