Benchmarking Compositionality with Formal Languages
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability. Whether large neural models in NLP can acquire this ability while learning from data is an open question. In this paper, we investigate this problem from the perspective of formal languages. We use deterministic finite-state transducers to make an unbounded number of datasets with controllable properties governing compositionality. By randomly sampling over many transducers, we explore which of their properties contribute to learnability of a compositional relation by a neural network. We find that the models either learn the relations completely or not at all. The key is transition coverage, setting a soft learnability limit at 400 examples per transition.
Code (1)
Tasks
BenchmarkingOpen-Ended Question AnsweringSimilar Papers 제목 키워드 기반
The Combinatorics of \textit{Salva Veritate} Principles
Various concepts of grammatical compositionality arise in many theories of both natural and artificial languages, and often play a key role in accounts of the syntax-semantics interface. We propose that many instances of…
On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation
Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language…
BenchmarkingMachine TranslationNMTMeasuring Compositionality in Representation Learning
Many machine learning algorithms represent input data with vector embeddings or discrete codes. When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to…
BIG-bench Machine LearningRepresentation LearningInternal and external pressures on language emergence: least effort, object constancy and frequency
In previous work, artificial agents were shown to achieve almost perfect accuracy in referential games where they have to communicate to identify images. Nevertheless, the resulting communication protocols rarely display…
DiagnosticPositionInternal and External Pressures on Language Emergence: Least Effort, Object Constancy and Frequency
In previous work, artificial agents were shown to achieve almost perfect accuracy in referential games where they have to communicate to identify images. Nevertheless, the resulting communication protocols rarely display…
DiagnosticPosition