SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics
Recently, deep neural networks (DNNs) have achieved great success in semantically challenging NLP tasks, yet it remains unclear whether DNN models can capture compositional meanings, those aspects of meaning that have been long studied in formal semantics. To investigate this issue, we propose a Systematic Generalization testbed based on Natural language Semantics (SyGNS), whose challenge is to map natural language sentences to multiple forms of scoped meaning representations, designed to account for various semantic phenomena. Using SyGNS, we test whether neural networks can systematically parse sentences involving novel combinations of logical expressions such as quantifiers and negation. Experiments show that Transformer and GRU models can generalize to unseen combinations of quantifiers, negations, and modifiers that are similar to given training instances in form, but not to the others. We also find that the generalization performance to unseen combinations is better when the form of meaning representations is simpler. The data and code for SyGNS are publicly available at https://github.com/verypluming/SyGNS.
Code (1)
Tasks
NegationSystematic GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis
Analyzing model performance in various unseen environments is a critical research problem in the machine learning community. To study this problem, it is important to construct a testbed with out-of-distribution test set…
BenchmarkingDomain GeneralizationImage RetrievalSystematic Generalization on gSCAN with Language Conditioned Embedding
Systematic Generalization refers to a learning algorithm's ability to extrapolate learned behavior to unseen situations that are distinct but semantically similar to its training data. As shown in recent work, state-of-t…
Systematic GeneralizationUnlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning
Systematic, compositional generalization beyond the training distribution remains a core challenge in machine learning -- and a critical bottleneck for the emergent reasoning abilities of modern language models. This wor…
Logical forms complement probability in understanding language model (and human) performance
With the increasing interest in using large language models (LLMs) for planning in natural language, understanding their behaviors becomes an important research question. This work conducts a systematic investigation of …
Language ModelingLanguage ModellingLogical ReasoningNatural Language UnderstandingFacts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
Language models are pretrained on sequences that blend statistical regularities (making text fluent) with factual associations between specific tokens (knowledge of facts). While recent work suggests that the variability…