TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields

2023-09-29 · Tianyu Huang, Yihan Zeng, Bowen Dong, Hang Xu, Songcen Xu, Rynson W. H. Lau, WangMeng Zuo

Recent works learn 3D representation explicitly under text-3D guidance. However, limited text-3D data restricts the vocabulary scale and text control of generations. Generators may easily fall into a stereotype concept for certain text prompts, thus losing open-vocabulary generation ability. To tackle this issue, we introduce a conditional 3D generative model, namely TextField3D. Specifically, rather than using the text prompts as input directly, we suggest to inject dynamic noise into the latent space of given text prompts, i.e., Noisy Text Fields (NTFs). In this way, limited 3D data can be mapped to the appropriate range of textual latent space that is expanded by NTFs. To this end, an NTFGen module is proposed to model general text latent code in noisy fields. Meanwhile, an NTFBind module is proposed to align view-invariant image latent code to noisy fields, further supporting image-conditional 3D generation. To guide the conditional generation in both geometry and texture, multi-modal discrimination is constructed with a text-3D discriminator and a text-2.5D discriminator. Compared to previous methods, TextField3D includes three merits: 1) large vocabulary, 2) text consistency, and 3) low latency. Extensive experiments demonstrate that our method achieves a potential open-vocabulary 3D generation capability.

📄 PDF Abstract BibTeX arXiv:2309.17175

Code (0)

등록된 구현이 없습니다.

Tasks

3D Generation

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

TextField: Learning A Deep Direction Field for Irregular Scene Text Detection

2018-12-04 · Yongchao Xu, Yukang Wang, Wei Zhou, Yongpan Wang 외

Scene text detection is an important step of scene text reading system. The main challenges lie on significantly varied sizes and aspect ratios, arbitrary orientations and shapes. Driven by recent progress in deep learni…

Scene Text Detection Text Detection

Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

2023-01-02 · ICCV 2023 1 · Jianzong Wu, Xiangtai Li, Henghui Ding, Xia Li 외

In this work, we focus on open vocabulary instance segmentation to expand a segmentation model to classify and segment instance-level novel categories. Previous approaches have relied on massive caption datasets and comp…

Caption Generation Instance Segmentation Panoptic Segmentation Segmentation+1

From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models

2024-04-01 · CVPR 2024 1 · Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen 외

Scene graph generation (SGG) aims to parse a visual scene into an intermediate graph representation for downstream reasoning tasks. Despite recent advancements, existing methods struggle to generate scene graphs with nov…

Graph Generation Image to text Relation Scene Graph Generation+1

Structure-Aware Feature Rectification with Region Adjacency Graphs for Training-Free Open-Vocabulary Semantic Segmentation

2025-12-08 · Qiming Huang, Hao Ai, Jianbo Jiao arxiv

Benefiting from the inductive biases learned from large-scale datasets, open-vocabulary semantic segmentation (OVSS) leverages the power of vision-language models, such as CLIP, to achieve remarkable progress without req…

Semantic Segmentation

ReLaGS: Relational Language Gaussian Splatting

2026-03-18 · Yaxu Xie, Abdalla Arafa, Alireza Javanmardi, Christen Millerdurai 외 arxiv

Achieving unified 3D perception and reasoning across tasks such as segmentation, retrieval, and relation understanding remains challenging, as existing methods are either object-centric or rely on costly training for int…

Scene Graph Generation Relational Reasoning Graph Neural Network