*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task
We present *-CFQ ("star-CFQ"): a suite of large-scale datasets of varying scope based on the CFQ semantic parsing benchmark, designed for principled investigation of the scalability of machine learning systems in a realistic compositional task setting. Using this suite, we conduct a series of experiments investigating the ability of Transformers to benefit from increased training size under conditions of fixed computational cost. We show that compositional generalization remains a challenge at all training sizes, and we show that increasing the scope of natural language leads to consistently higher error rates, which are only partially offset by increased training data. We further show that while additional training data from a related domain improves the accuracy in data-starved situations, this improvement is limited and diminishes as the distance from the related domain to the target domain increases.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningSemantic ParsingSimilar Papers 제목 키워드 기반
Compositional and Equilibrium-Free Conditions for Power System Stability -- Part I: Theory
Traditional centralized stability analysis struggles with scalability in large complex modern power grids. This two-part paper proposes a compositional and equilibrium-free approach to analyzing power system stability. I…
Evaluating Neural Word Representations in Tensor-Based Compositional Settings
We provide a comparative study between neural word representations and traditional vector spaces based on co-occurrence counts, in a number of compositional tasks. We use three different semantic spaces and implement sev…
SentenceSentence SimilarityWord EmbeddingsScalable Evaluation and Neural Models for Compositional Generalization
Compositional generalization-a key open challenge in modern machine learning-requires models to predict unknown combinations of known concepts. However, assessing compositional generalization remains a fundamental challe…
Do LSTMs Learn Compositionally?
LSTM-based language models exhibit compositionality in their representations, but how this behavior emerges over the course of training has not been explored. Analyzing synthetic data experiments with contextual decompos…
MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation
Compositionality is critical for 3D object and scene generation, but existing part-aware 3D generation methods suffer from poor scalability due to quadratic global attention costs when increasing the number of components…
Scene Generation3D Generation