Adapt PointFormer: 3D Point Cloud Analysis via Adapting 2D Visual Transformers

2024-07-18 · Mengke Li, Da Li, Guoqing Yang, Yiu-ming Cheung, Hui Huang

Pre-trained large-scale models have exhibited remarkable efficacy in computer vision, particularly for 2D image analysis. However, when it comes to 3D point clouds, the constrained accessibility of data, in contrast to the vast repositories of images, poses a challenge for the development of 3D pre-trained models. This paper therefore attempts to directly leverage pre-trained models with 2D prior knowledge to accomplish the tasks for 3D point cloud analysis. Accordingly, we propose the Adaptive PointFormer (APF), which fine-tunes pre-trained 2D models with only a modest number of parameters to directly process point clouds, obviating the need for mapping to images. Specifically, we convert raw point clouds into point embeddings for aligning dimensions with image tokens. Given the inherent disorder in point clouds, in contrast to the structured nature of images, we then sequence the point embeddings to optimize the utilization of 2D attention priors. To calibrate attention across 3D and 2D domains and reduce computational overhead, a trainable PointFormer with a limited number of parameters is subsequently concatenated to a frozen pre-trained image model. Extensive experiments on various benchmarks demonstrate the effectiveness of the proposed APF. The source code and more details are available at https://vcc.tech/research/2024/PointFormer.

📄 PDF Abstract BibTeX arXiv:2407.13200

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

3D Object Detection with Pointformer

2020-12-21 · CVPR 2021 1 · Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li 외

Feature learning for 3D object detection from point clouds is very challenging due to the irregularity of 3D point cloud data. In this paper, we propose Pointformer, a Transformer backbone designed for 3D point clouds to…

3D Object Detection Object object-detection Object Detection+1

6D-ViT: Category-Level 6D Object Pose Estimation via Transformer-based Instance Representation Learning

2021-10-10 · Lu Zou, Zhangjin Huang, Naijie Gu, Guoping Wang

This paper presents 6D-ViT, a transformer-based instance representation learning network, which is suitable for highly accurate category-level object pose estimation on RGB-D images. Specifically, a novel two-stream enco…

6D Pose Estimation using RGB Decoder Pose Estimation Representation Learning

HyperPointFormer: Multimodal Fusion in 3D Space with Dual-Branch Cross-Attention Transformers

2025-05-29 · Aldino Rizaldy, Richard Gloaguen, Fabian Ewald Fassnacht, Pedram Ghamisi

Multimodal remote sensing data, including spectral and lidar or photogrammetry, is crucial for achieving satisfactory land-use / land-cover classification results in urban scenes. So far, most studies have been conducted…

Land Cover Classification

CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection

2022-04-01 · CVPR 2022 1 · Yanan Zhang, Jiaxin Chen, Di Huang

In autonomous driving, LiDAR point-clouds and RGB images are two major data modalities with complementary cues for 3D object detection. However, it is quite difficult to sufficiently use them, due to large inter-modal di…

3D Object Detection Autonomous Driving Contrastive Learning Data Augmentation+3

Test-Time Adaptation in Point Clouds: Leveraging Sampling Variation with Weight Averaging

2024-11-02 · Ali Bahri, Moslem Yazdanpanah, Mehrdad Noori, Sahar Dastani 외

Test-Time Adaptation (TTA) addresses distribution shifts during testing by adapting a pretrained model without access to source data. In this work, we propose a novel TTA approach for 3D point cloud classification, combi…

3D Point Cloud Classification Point Cloud Classification Test-time Adaptation