Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations

2024-07-05 · Matthias Lindemann, Alexander Koller, Ivan Titov

Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

📄 PDF Abstract BibTeX arXiv:2407.04543

Code (1)

namednil/step 공식 구현 pytorch

Tasks

Chunking Few-Shot LearningInductive BiasSemantic Parsing

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Sigmoid Activation 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Tanh Activation 설명 없음

Multi-Head Attention 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

On Inductive Biases That Enable Generalization of Diffusion Transformers

2024-10-28 · Jie An, De Wang, Pengsheng Guo, Jiebo Luo 외

Recent work studying the generalization of diffusion models with UNet-based denoisers reveals inductive biases that can be expressed via geometry-adaptive harmonic bases. However, in practice, more recent denoising netwo…

Denoising Inductive Bias

SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation

2023-10-01 · Matthias Lindemann, Alexander Koller, Ivan Titov

Strong inductive biases enable learning from little data and help generalization outside of the training distribution. Popular neural architectures such as Transformers lack strong structural inductive biases for seq2seq…

Few-Shot Learning Inductive Bias Systematic Generalization

Injecting structural hints: Using language models to study inductive biases in language learning

2023-04-25 · Isabel Papadimitriou, Dan Jurafsky

Both humans and large language models are able to learn language without explicit structural supervision. What inductive biases make this learning possible? We address this fundamental cognitive question by leveraging tr…

Inductive Bias Transfer Learning

Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling

2026-05-18 · Afiq Abdillah Effiezal Aswadi, Oliver Britton, Ross Baker, Matthew Farrugia-Roberts arxiv

Modern deep learning science often assumes that neural networks learn from a fixed data distribution. However, many practically important learning problems involve data distributions that change throughout training. How …

Inductive biases in deep learning models for weather prediction

2023-04-06 · Jannik Thuemmel, Matthias Karlbauer, Sebastian Otte, Christiane Zarfl 외

Deep learning has gained immense popularity in the Earth sciences as it enables us to formulate purely data-driven models of complex Earth system processes. Deep learning-based weather prediction (DLWP) models have made …

Deep Learning Prediction Self-Supervised Learning