Beyond Flatland: Pre-training with a Strong 3D Inductive Bias

2021-11-30 · Shubhaankar Gupta, Thomas P. O'Connell, Bernhard Egger

Pre-training on large-scale databases consisting of natural images and then fine-tuning them to fit the application at hand, or transfer-learning, is a popular strategy in computer vision. However, Kataoka et al., 2020 introduced a technique to eliminate the need for natural images in supervised deep learning by proposing a novel synthetic, formula-based method to generate 2D fractals as training corpus. Using one synthetically generated fractal for each class, they achieved transfer learning results comparable to models pre-trained on natural images. In this project, we take inspiration from their work and build on this idea -- using 3D procedural object renders. Since the image formation process in the natural world is based on its 3D structure, we expect pre-training with 3D mesh renders to provide an implicit bias leading to better generalization capabilities in a transfer learning setting and that invariances to 3D rotation and illumination are easier to be learned based on 3D data. Similar to the previous work, our training corpus will be fully synthetic and derived from simple procedural strategies; we will go beyond classic data augmentation and also vary illumination and pose which are controllable in our setting and study their effect on transfer learning capabilities in context to prior work. In addition, we will compare the 2D fractal and 3D procedural object networks to human and non-human primate brain data to learn more about the 2D vs. 3D nature of biological vision.

📄 PDF Abstract BibTeX arXiv:2112.00113

Code (0)

등록된 구현이 없습니다.

Tasks

Data AugmentationInductive BiasTransfer Learning

SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation

2023-10-01 · Matthias Lindemann, Alexander Koller, Ivan Titov

Strong inductive biases enable learning from little data and help generalization outside of the training distribution. Popular neural architectures such as Transformers lack strong structural inductive biases for seq2seq…

Few-Shot Learning Inductive Bias Systematic Generalization

On the Inductive Bias of Stacking Towards Improving Reasoning

2024-09-27 · Nikunj Saunshi, Stefani Karp, Shankar Krishnan, Sobhan Miryoosefi 외

Given the increasing scale of model sizes, novel training strategies like gradual stacking [Gong et al., 2019, Reddi et al., 2023] have garnered interest. Stacking enables efficient training by gradually growing the dept…

Inductive Bias Math Reading Comprehension

Generalization through variance: how noise shapes inductive biases in diffusion models

2025-04-16 · John J. Vastola

How diffusion models generalize beyond their training set is not known, and is somewhat mysterious given two facts: the optimum of the denoising score matching (DSM) objective usually used to train diffusion models is th…

Denoising Inductive Bias

Learning Language Representations with Logical Inductive Bias

2023-02-19 · Jianshu Chen

Transformer architectures have achieved great success in solving natural language tasks, which learn strong language representations from large-scale unlabeled texts. In this paper, we seek to go further beyond and explo…

Inductive Bias Representation Learning

Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting

2026-03-17 · Yu-Chen Den, Kuan-Yu Chen, Kendro Vincent, Darby Tien-Hao Chang arxiv

Transformer-based models have been widely adopted for time-series forecasting due to their high representational capacity and architectural flexibility. However, many Transformer variants implicitly assume stationarity a…

Time Series Forecasting Knowledge Distillation