Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language
We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs). Our system uses a language model to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever. We open-source our code at https://github.com/ContextualAI/lens and provide an interactive demo.
Code (1)
Tasks
DescriptiveLanguage ModelingLanguage ModellingObject RecognitionSimilar Papers 제목 키워드 기반
Image Quality Is Not All You Want: Task-Driven Lens Design for Image Classification
In computer vision, it has long been taken for granted that high-quality images obtained through well-designed camera lenses would lead to superior results. However, we find that this common perception is not a "one-size…
Allimage-classificationImage ClassificationViT-Explainer: An Interactive Walkthrough of the Vision Transformer Pipeline
Transformer-based architectures have become the shared backbone of natural language processing and computer vision. However, understanding how these models operate remains challenging, particularly in vision settings, wh…
A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
Large language models such as BERT and the GPT series started a paradigm shift that calls for building general-purpose models via pre-training on large datasets, followed by fine-tuning on task-specific datasets. There i…
Image CaptioningMachine TranslationMultimodal Machine TranslationQuestion Answering+2Adaptive Camera Sensor for Vision Models
Domain shift remains a persistent challenge in deep-learning-based computer vision, often requiring extensive model modifications or large labeled datasets to address. Inspired by human visual perception, which adjusts i…
Exploring Data Pipelines through the Process Lens: a Reference Model forComputer Vision
Researchers have identified datasets used for training computer vision (CV) models as an important source of hazardous outcomes, and continue to examine popular CV datasets to expose their harms. These works tend to trea…