Papers Grounded language learning
“Grounded language learning” 태그가 달린 논문 56편 · 필터 해제
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling
Today's most accurate language models are trained on orders of magnitude more language data than human language learners receive - but with no supervision from other sensory modalities that play a crucial role in human l…
Grounded language learningLanguage AcquisitionLanguage ModelingLanguage Modelling+2Visually Grounded Language Learning: a review of language games, datasets, tasks, and models
In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many …
Grounded language learningLanguage ModellingNatural Language UnderstandingSystematic Literature ReviewExpand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment
Language models have been supervised with both language-only objective and visual grounding in existing studies of visual-grounded language learning. However, due to differences in the distribution and scale of visual-gr…
Grounded language learningLanguage ModelingLanguage ModellingRepresentation Learning+2Improved Compositional Generalization by Generating Demonstrations for Meta-Learning
Meta-learning and few-shot prompting are viable methods to induce certain types of compositional behaviour. However, these methods can be very sensitive to the choice of support examples used. Choosing good supports from…
Grounded language learningMeta-LearningVisual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences
Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them. We improve visual story generation by producing a new image-grounded…
Coherence EvaluationGrounded language learningNatural Language Visual GroundingSentence+3Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
People rely heavily on context to enrich meaning beyond what is literally said, enabling concise but effective communication. To interact successfully and naturally with people, user-facing artificial intelligence system…
Grounded language learningCompositional Generalization in Grounded Language Learning via Induced Model Sparsity
We provide a study of how induced model sparsity can help achieve compositional generalization and better sample efficiency in grounded language learning problems. We consider simple language-conditioned navigation probl…
Grounded language learningPretraining on Interactions for Learning Grounded Affordance Representations
Lexical semantics and cognitive science point to affordances (i.e. the actions that objects support) as critical for understanding and representing nouns and verbs. However, study of these semantic features has not yet b…
Grounded language learningNot Cheating on the Turing Test: Towards Grounded Language Learning in Artificial Intelligence
Recent hype surrounding the increasing sophistication of language processing models has renewed optimism regarding machines achieving a human-like command of natural language. Research in the area of natural language und…
Grounded language learningNatural Language UnderstandingPhilosophyImproving Systematic Generalization Through Modularity and Augmentation
Systematic generalization is the ability to combine known parts into novel meaning; an important aspect of efficient human learning, but a weakness of neural network learning. In this work, we investigate how two well-kn…
Data AugmentationGrounded language learningSystematic GeneralizationSeeing the advantage: visually grounding word embeddings to better capture human semantic knowledge
Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are p…
Grounded language learningImage RetrievalLearning Semantic RepresentationsVisual Grounding+2SILG: The Multi-domain Symbolic Interactive Language Grounding Benchmark
Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding be…
Grounded language learningNetHackSILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding be…
Grounded language learningNetHackGrounding Language Representation with Visual Object Information via Cross Modal Pretraining
Previous studies of visual grounded language learning use a convolutional neural network (CNN) to extract features from the whole image for grounding with the sentence description. However, this approach has two main dra…
Grounded language learningObjectSentenceNeural Abstructions: Abstractions that Support Construction for Grounded Language Learning
Although virtual agents are increasingly situated in environments where natural language is the most effective mode of interaction with humans, these exchanges are rarely used as an opportunity for learning. Leveraging l…
Grounded language learningMinecraftAlign before Fuse: Vision and Language Representation Learning with Momentum Distillation
Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual token…
Cross-Modal RetrievalGrounded language learningImage-text matchingImage-text Retrieval+8Semantic sentence similarity: size does not always matter
This study addresses the question whether visually grounded speech recognition (VGS) models learn to capture sentence semantics without access to any prior linguistic knowledge. We produce synthetic and natural spoken ve…
Grounded language learningImage RetrievalLearning Semantic RepresentationsSemantic Similarity+7Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction
We investigate grounded language learning through real-world data, by modelling a teacher-learner dynamics through the natural interactions occurring between users and search engines; in particular, we explore the emerge…
Grounded language learningInteractive Learning from Activity Description
We present a novel interactive learning protocol that enables training request-fulfilling agents by verbally describing their activities. Unlike imitation learning (IL), our protocol allows the teaching agent to provide …
General Reinforcement LearningGrounded language learningImitation LearningReinforcement Learning (RL)A Visuospatial Dataset for Naturalistic Verb Learning
We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality environment and is designed to emulate the quality of language data to which a pre-verbal ch…
Grounded language learningLanguage Acquisition