paper-with-me

홈 › Papers

KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension

2024-11-04 · Jie Yang, Wang Zeng, Sheng Jin, Lumin Xu, Wentao Liu, Chen Qian, Ruimao Zhang

Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities in image understanding. However, these models often struggle with grasping pixel-level semantic details, e.g., the keypoints of an object. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics in keypoints, followed by the precise determination of their positions through a chain-of-thought process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints.

📄 PDF Abstract BibTeX arXiv:2411.01846

Code (0)

등록된 구현이 없습니다.

Tasks

Keypoint DetectionLanguage ModelingLanguage ModellingLarge Language Model

Similar Papers 제목 키워드 기반

KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model

2025-07-15 · Jie Yang, Wang Zeng, Sheng Jin, Lumin Xu 외

The emergence of Multimodal Large Language Models (MLLMs) has revolutionized image understanding by bridging textual and visual modalities. However, these models often struggle with capturing fine-grained semantic inform…

Keypoint DetectionLanguage ModelingLanguage ModellingLarge Language Model+1

GKDT: General Keypoint Detection Transformer

2026-07-01 · Changsheng Lu, Yuxin Chen, Haokun Gui, Rong Wang 외 arxiv

With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) mo…

Keypoint Detection

ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models

2024-12-09 · Bingchen Gong, Diego Gomez, Abdullah Hamdi, Abdelrahman Eldesokey 외

We propose a novel zero-shot approach for keypoint detection on 3D shapes. Point-level reasoning on visual data is challenging as it requires precise localization capability, posing problems even for powerful models like…

Keypoint Detection

PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment

2025-07-12 · Dewen Zhang, Tahir Hussain, Wangpeng An, Hayaru Shouno

Human pose estimation traditionally relies on architectures that encode keypoint priors, limiting their generalization to novel poses or unseen keypoints. Recent language-guided approaches like LocLLM reformulate keypoin…

Large Language ModelPose EstimationZero-shot Generalization

2.5D U-Net with Depth Reduction for 3D CryoET Object Identification

2025-02-19 · Yusuke Uchida, Takaaki Fukui

Cryo-electron tomography (cryoET) is a crucial technique for unveiling the structure of protein complexes. Automatically analyzing tomograms captured by cryoET is an essential step toward understanding cellular structure…

Electron TomographyKeypoint Detection