FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models

2023-03-22 · ICCV 2023 1 · Jianglong Ye, Naiyan Wang, Xiaolong Wang

Recent works on generalizable NeRFs have shown promising results on novel view synthesis from single or few images. However, such models have rarely been applied on other downstream tasks beyond synthesis such as semantic understanding and parsing. In this paper, we propose a novel framework named FeatureNeRF to learn generalizable NeRFs by distilling pre-trained vision foundation models (e.g., DINO, Latent Diffusion). FeatureNeRF leverages 2D pre-trained foundation models to 3D space via neural rendering, and then extract deep features for 3D query points from NeRF MLPs. Consequently, it allows to map 2D images to continuous 3D semantic feature volumes, which can be used for various downstream tasks. We evaluate FeatureNeRF on tasks of 2D/3D semantic keypoint transfer and 2D/3D object part segmentation. Our extensive experiments demonstrate the effectiveness of FeatureNeRF as a generalizable 3D semantic feature extractor. Our project page is available at https://jianglongye.com/featurenerf/ .

📄 PDF Abstract BibTeX arXiv:2303.12786

Code (1)

jianglongye/featurenerf 공식 구현 pytorch

Tasks

NeRFNeural RenderingNovel View Synthesis

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

2024-06-17 · Letian Wang, Seung Wook Kim, Jiawei Yang, Cunjun Yu 외

We propose DistillNeRF, a self-supervised learning framework addressing the challenge of understanding 3D environments from limited 2D observations in outdoor autonomous driving scenes. Our method is a generalizable feed…

3D geometry 3D Semantic Occupancy Prediction Autonomous Driving Depth Estimation+3

Advancing Adversarial Robustness in GNeRFs: The IL2-NeRF Attack

2025-01-01 · CVPR 2025 1 · Nicole Meng, Caleb Manicke, Ronak Sahu, Caiwen Ding 외

Generalizable Neural Radiance Fields (GNeRF) are recognized as one of the most promising techniques for novel view synthesis and 3D model generation in real-world applications. However, like other generative models i…

3D Reconstruction Adversarial Attack Adversarial Robustness NeRF+1

BoostMVSNeRFs: Boosting MVS-based NeRFs to Generalizable View Synthesis in Large-scale Scenes

2024-07-22 · Chih-Hai Su, Chih-Yao Hu, Shr-Ruei Tsai, Jie-Ying Lee 외

While Neural Radiance Fields (NeRFs) have demonstrated exceptional quality, their protracted training duration remains a limitation. Generalizable and MVS-based NeRFs, although capable of mitigating training time, often …

NeRF

DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields

2023-03-25 · CVPR 2023 1 · Yu Chen, Gim Hee Lee

Recent works such as BARF and GARF can bundle adjust camera poses with neural radiance fields (NeRF) which is based on coordinate-MLPs. Despite the impressive results, these methods cannot be applied to Generalizable NeR…