Contextual Media Retrieval Using Natural Language Queries
The widespread integration of cameras in hand-held and head-worn devices as well as the ability to share content online enables a large and diverse visual capture of the world that millions of users build up collectively every day. We envision these images as well as associated meta information, such as GPS coordinates and timestamps, to form a collective visual memory that can be queried while automatically taking the ever-changing context of mobile users into account. As a first step towards this vision, in this work we present Xplore-M-Ego: a novel media retrieval system that allows users to query a dynamic database of images and videos using spatio-temporal natural language queries. We evaluate our system using a new dataset of real user queries as well as through a usability study. One key finding is that there is a considerable amount of inter-user variability, for example in the resolution of spatial relations in natural language utterances. We show that our retrieval system can cope with this variability using personalisation through an online learning-based retrieval formulation.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language QueriesRetrievalSimilar Papers 제목 키워드 기반
Evaluating Contextually Mediated Factual Recall in Multilingual Large Language Models
Large language models (LLMs) can recall a wide range of factual knowledge across languages. However, existing factual recall evaluations primarily assess fact retrieval in isolation, where the queried entity is explicitl…
RAPID: Retrieval-Augmented Parallel Inference Drafting for Text-Based Video Event Retrieval
Retrieving events from videos using text queries has become increasingly challenging due to the rapid growth of multimedia content. Existing methods for text-based video event retrieval often focus heavily on object-leve…
RetrievalSemantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incu…
Question AnsweringRetrievalRetrieval-augmented GenerationTriviaQALeveraging Lightweight Entity Extraction for Scalable Event-Based Image Retrieval
Retrieving images from natural language descriptions is a core task at the intersection of computer vision and natural language processing, with wide-ranging applications in search engines, media archiving, and digital c…
Image RetrievalText RetrievalSoccerRAG: Multimodal Soccer Information Retrieval via Natural Queries
The rapid evolution of digital sports media necessitates sophisticated information retrieval systems that can efficiently parse extensive multimodal datasets. This paper introduces SoccerRAG, an innovative framework desi…
Information RetrievalNatural Language QueriesRAGRetrieval+2