Papers Natural Language Visual Grounding
“Natural Language Visual Grounding” 태그가 달린 논문 32편 · 필터 해제
Aria-UI: Visual Grounding for GUI Instructions
Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant…
Natural Language Visual GroundingVisual GroundingAguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Automating GUI tasks remains challenging due to reliance on textual representations, platform-specific action spaces, and limited reasoning capabilities. We introduce Aguvis, a unified vision-based framework for autonomo…
Multimodal ReasoningNatural Language Visual GroundingNavigateShowUI: One Vision-Language-Action Model for GUI Visual Agent
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (…
Instruction FollowingNatural Language Visual GroundingVision-Language-ActionImproved GUI Grounding via Iterative Narrowing
Graphical User Interface (GUI) grounding plays a crucial role in enhancing the capabilities of Vision-Language Model (VLM) agents. While general VLMs, such as GPT-4V, demonstrate strong performance across various tasks, …
Language ModelingLanguage ModellingNatural Language Visual GroundingVisual PromptingOS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs du…
Natural Language Visual GroundingNavigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across va…
Natural Language Visual GroundingNavigateVisual GroundingQwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL models that redefines the conventional predetermined-resolution approach in visual processing. Qwen2-VL introduces the Naive Dynamic Resolution …
Natural Language Visual GroundingTemporal Relation ExtractionVideo Question Answering+3OmniParser for Pure Vision Based GUI Agent
The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on mult…
Natural Language Visual GroundingGUICourse: From General Vision Language Models to Versatile GUI Agents
Utilizing Graphic User Interface (GUI) for human-computer interaction is essential for accessing a wide range of digital tools. Recent advancements in Vision Language Models (VLMs) highlight the compelling potential to d…
Natural Language Visual GroundingOptical Character Recognition (OCR)Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded and fine-grained visual perception ability. Beyond holistic image understanding, Groma is adept at region-level tasks such as region captioning a…
Language ModelingLanguage ModellingLarge Language ModelMultimodal Large Language Model+3SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data,…
Natural Language Visual GroundingCogAgent: A Visual Language Model for GUI Agents
People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks li…
Language ModelingLanguage ModellingMM-Vet+4MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
Large language models have shown their remarkable capabilities as a general interface for various language-related applications. Motivated by this, we target to build a unified interface for completing many vision-langua…
Image ClassificationImage DescriptionLanguage ModelingLanguage Modelling+8Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with vi…
Chart Question AnsweringFS-MEVQAImage CaptioningImage Description+11Localizing Moments in Long Video Via Multimodal Guidance
The recent introduction of the large-scale, long-form MAD and Ego4D datasets has enabled researchers to investigate the performance of current state-of-the-art methods for video grounding in the long-form setup, with int…
Natural Language Moment RetrievalNatural Language Visual GroundingVideo GroundingVideo UnderstandingVisual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences
Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them. We improve visual story generation by producing a new image-grounded…
Coherence EvaluationGrounded language learningNatural Language Visual GroundingSentence+3Belief Revision based Caption Re-ranker with Visual Semantic Information
In this work, we focus on improving the captions generated by image-caption generation systems. We propose a novel re-ranking approach that leverages visual-semantic measures to identify the ideal caption that maximally …
Caption GenerationImage CaptioningNatural Language UnderstandingNatural Language Visual Grounding+2TubeDETR: Spatio-Temporal Video Grounding with Transformers
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal …
DecoderLanguage-Based Temporal LocalizationNatural Language Visual Groundingobject-detection+6CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse rep…
Continuous ControlImitation LearningNatural Language Visual GroundingRobotic Grasping+1Panoptic Narrative Grounding
This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, includin…
Natural Language Visual GroundingPanoptic SegmentationVisual Grounding