Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments

2023-11-07 · Danial Kamali, Parisa Kordjamshidi

Compositional generalization, the ability of intelligent models to extrapolate understanding of components to novel compositions, is a fundamental yet challenging facet in AI research, especially within multimodal environments. In this work, we address this challenge by exploiting the syntactic structure of language to boost compositional generalization. This paper elevates the importance of syntactic grounding, particularly through attention masking techniques derived from text input parsing. We introduce and evaluate the merits of using syntactic information in the multimodal grounding problem. Our results on grounded compositional generalization underscore the positive impact of dependency parsing across diverse tasks when utilized with Weight Sharing across the Transformer encoder. The results push the state-of-the-art in multimodal grounding and parameter-efficient modeling and provide insights for future research.

📄 PDF Abstract BibTeX arXiv:2311.04364

Code (0)

등록된 구현이 없습니다.

Tasks

Compositional Generalization (AVG)Dependency Parsing

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Residual Connection 설명 없음

Multi-Head Attention 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Compositional Generalization Requires Compositional Parsers

2022-02-24 · Pia Weißenhorn, Yuekun Yao, Lucia Donatelli, Alexander Koller

A rapidly growing body of research on compositional generalization investigates the ability of a semantic parser to dynamically recombine linguistic elements seen in training into unseen sequences. We present a systemati…

Representational Homomorphism Predicts and Improves Compositional Generalization In Transformer Language Model

2026-01-26 · Zhiyu An, Wan Du arxiv

Compositional generalization-the ability to interpret novel combinations of familiar components-remains a persistent challenge for neural networks. Behavioral evaluations reveal \emph{when} models fail but offer limited …

Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings

2023-01-21 · Rohan Pandey

Past work probing compositionality in sentence embedding models faces issues determining the causal impact of implicit syntax representations. Given a sentence, we construct a neural module net based on its syntax parse …

Semantic Composition Sentence Sentence Embedding Sentence-Embedding+1

Grounded Graph Decoding Improves Compositional Generalization in Question Answering

2021-11-05 · Findings (EMNLP) 2021 11 · Yu Gai, Paras Jain, Wendi Zhang, Joseph E. Gonzalez 외

Question answering models struggle to generalize to novel compositions of training patterns, such to longer sequences or more complex test structures. Current end-to-end models learn a flat input embedding which can lose…

Question Answering

When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks

2022-10-23 · Ankur Sikarwar, Arkil Patel, Navin Goyal

Humans can reason compositionally whilst grounding language utterances to the real world. Recent benchmarks like ReaSCAN use navigation tasks grounded in a grid world to assess whether neural models exhibit similar capab…

Compositional Generalization (AVG)