Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
Pixel-level 2D object semantic understanding is an important topic in computer vision and could help machine deeply understand objects (e.g. functionality and affordance) in our daily life. However, most previous methods directly train on correspondences in 2D images, which is end-to-end but loses plenty of information in 3D spaces. In this paper, we propose a new method on predicting image corresponding semantics in 3D domain and then projecting them back onto 2D images to achieve pixel-level understanding. In order to obtain reliable 3D semantic labels that are absent in current image datasets, we build a large scale keypoint knowledge engine called KeypointNet, which contains 103,450 keypoints and 8,234 3D models from 16 object categories. Our method leverages the advantages in 3D vision and can explicitly reason about objects self-occlusion and visibility. We show that our method gives comparative and even superior results on standard semantic benchmarks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities in image understanding. However, these models often struggle with grasping pixel-level semantic details, e.g., the key…
Keypoint DetectionLanguage ModelingLanguage ModellingLarge Language ModelKptLLM++: Towards Generic Keypoint Comprehension with Large Language Model
The emergence of Multimodal Large Language Models (MLLMs) has revolutionized image understanding by bridging textual and visual modalities. However, these models often struggle with capturing fine-grained semantic inform…
Keypoint DetectionLanguage ModelingLanguage ModellingLarge Language Model+1Talking Points: Describing and Localizing Pixels
Vision-language models have achieved remarkable success in cross-modal understanding. Yet, these models remain limited to object-level or region-level grounding, lacking the capability for pixel-precise keypoint comprehe…
Unsupervised Learning of Object Structure and Dynamics from Videos
Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stoc…
Action Recognitioncontinuous-controlContinuous ControlObject+2Keypoint Autoencoders: Learning Interest Points of Semantics
Understanding point clouds is of great importance. Many previous methods focus on detecting salient keypoints to identity structures of point clouds. However, existing methods neglect the semantics of points selected, le…