From Pairs to Sequences: Track-Aware Policy Gradients for Keypoint Detection
Keypoint-based matching is a fundamental component of modern 3D vision systems, such as Structure-from-Motion (SfM) and SLAM. Most existing learning-based methods are trained on image pairs, a paradigm that fails to explicitly optimize for the long-term trackability of keypoints across sequences under challenging viewpoint and illumination changes. In this paper, we reframe keypoint detection as a sequential decision-making problem. We introduce TraqPoint, a novel, end-to-end Reinforcement Learning (RL) framework designed to optimize the \textbf{Tra}ck-\textbf{q}uality (Traq) of keypoints directly on image sequences. Our core innovation is a track-aware reward mechanism that jointly encourages the consistency and distinctiveness of keypoints across multiple views, guided by a policy gradient method. Extensive evaluations on sparse matching benchmarks, including relative pose estimation and 3D reconstruction, demonstrate that TraqPoint significantly outperforms some state-of-the-art (SOTA) keypoint detection and description methods.The code will be available at https://github.com/xiaomi-research/traqpoint.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningKeypoint Detection3D ReconstructionPose EstimationSimilar Papers 제목 키워드 기반
Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
Reinforcement Learning, particularly through policy gradient methods, has played a central role in enabling reasoning capabilities of Large Language Models. However, the optimization stability of policy gradients in this…
Stochastic OptimizationReinforcement LearningNot All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models
Preference-based alignment is pivotal for training large reasoning models; however, standard methods like Direct Preference Optimization (DPO) typically treat all preference pairs uniformly, overlooking the evolving util…
Mathematical ReasoningDiffusion Policy with Bayesian Expert Selection for Active Multi-Target Tracking
Active multi-target tracking requires a mobile robot to balance exploration for undetected targets with exploitation of uncertain tracked ones. Diffusion policies have emerged as a powerful approach for capturing diverse…
Estimating Dynamic Flow Features in Groups of Tracked Objects
Interpreting motion captured in image sequences is crucial for a wide range of computer vision applications. Typical estimation approaches include optical flow (OF), which approximates the apparent motion instantaneously…
Multiple Object TrackingObject TrackingOptical Flow EstimationSoftTreeMax: Policy Gradient with Tree Search
Policy-gradient methods are widely used for learning control policies. They can be easily distributed to multiple workers and reach state-of-the-art results in many domains. Unfortunately, they exhibit large variance and…
Policy Gradient Methods