Natural Language Visual Grounding
1개 벤치마크 · 논문 32편 · 이 태스크의 논문 보기 →
Benchmarks
ScreenSpot
Most implemented
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
CogAgent: A Visual Language Model for GUI Agents
Grounding of Textual Phrases in Images by Reconstruction
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Papers
Aria-UI: Visual Grounding for GUI Instructions
Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant…
Natural Language Visual GroundingVisual GroundingAguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Automating GUI tasks remains challenging due to reliance on textual representations, platform-specific action spaces, and limited reasoning capabilities. We introduce Aguvis, a unified vision-based framework for autonomo…
Multimodal ReasoningNatural Language Visual GroundingNavigateShowUI: One Vision-Language-Action Model for GUI Visual Agent
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (…
Instruction FollowingNatural Language Visual GroundingVision-Language-ActionImproved GUI Grounding via Iterative Narrowing
Graphical User Interface (GUI) grounding plays a crucial role in enhancing the capabilities of Vision-Language Model (VLM) agents. While general VLMs, such as GPT-4V, demonstrate strong performance across various tasks, …
Language ModelingLanguage ModellingNatural Language Visual GroundingVisual PromptingOS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs du…
Natural Language Visual GroundingNavigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across va…
Natural Language Visual GroundingNavigateVisual Grounding