CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings
This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where language and images are projected in the same semantic space (in this case CLIP by OpenAI), CoLLIE learns a transformation function that adjusts the language embeddings when needed to accommodate new language use. This is done by predicting the difference vector that needs to be applied, as well as a scaling factor for this vector, so that the adjustment is only applied when needed. Unlike traditional few-shot learning, the model does not just learn new classes and labels, but can also generalize to similar language use and leverage semantic compositionality. We verify the model's performance on two different tasks of identifying the targets of referring expressions, where it has to learn new language use. The results show that the model can efficiently learn and generalize from only a few examples, with little interference with the model's original zero-shot performance.
Code (1)
Tasks
Continual LearningFew-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way
Large language models (LLMs) are increasingly pivotal in a wide range of natural language processing tasks. Access to pre-trained models, courtesy of the open-source community, has made it possible to adapt these models …
GPUparameter-efficient fine-tuningCOLLIE: Systematic Construction of Constrained Text Generation Tasks
Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained g…
Logical ReasoningSentenceText GenerationScalable Annotation of Fine-Grained Categories Without Experts
We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a coll…
A Broad-Coverage Deep Semantic Lexicon for Verbs
Progress on deep language understanding is inhibited by the lack of a broad coverage lexicon that connects linguistic behavior to ontological concepts and axioms. We have developed COLLIE-V, a deep lexical resource for v…
SiRi: A Simple Selective Retraining Mechanism for Transformer-based Visual Grounding
In this paper, we investigate how to achieve better visual grounding with modern vision-language transformers, and propose a simple yet powerful Selective Retraining (SiRi) mechanism for this challenging task. Particular…
Visual Grounding