InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions

2024-01-24 · Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, Jun Suzuki

We study the problem of completing various visual document understanding (VDU) tasks, e.g., question answering and information extraction, on real-world documents through human-written instructions. To this end, we propose InstructDoc, the first large-scale collection of 30 publicly available VDU datasets, each with diverse instructions in a unified format, which covers a wide range of 12 tasks and includes open document types/formats. Furthermore, to enhance the generalization performance on VDU tasks, we design a new instruction-based document reading and understanding model, InstructDr, that connects document images, image encoders, and large language models (LLMs) through a trainable bridging module. Experiments demonstrate that InstructDr can effectively adapt to new VDU datasets, tasks, and domains via given instructions and outperforms existing multimodal LLMs and ChatGPT without specific training.

📄 PDF Abstract BibTeX arXiv:2401.13313

Code (1)

nttmdlab-nlp/instructdoc 공식 구현

Tasks

document understandingQuestion Answering Zero-shot Generalization

Zero-Shot Semantic Segmentation via Spatial and Multi-Scale Aware Visual Class Embedding

2021-11-30 · Sungguk Cha, Yooseung Wang

Fully supervised semantic segmentation technologies bring a paradigm shift in scene understanding. However, the burden of expensive labeling cost remains as a challenge. To solve the cost problem, recent studies proposed…

Domain Adaptation Language Modeling Language Modelling Scene Understanding+4

EZ-CLIP: Efficient Zeroshot Video Action Recognition

2023-12-13 · Shahzad Ahmad, Sukalpa Chanda, Yogesh S Rawat

Recent advancements in large-scale pre-training of visual-language models on paired image-text data have demonstrated impressive generalization capabilities for zero-shot tasks. Building on this success, efforts have bee…

Action Recognition GPU Temporal Action Localization Visual Prompting+2

Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising

2025-03-26 · Yan-Bo Lin, Kevin Lin, Zhengyuan Yang, Linjie Li 외

In this paper, we introduce zero-shot audio-video editing, a novel task that requires transforming original audio-visual content to align with a specified textual prompt without additional model training. To evaluate thi…

Denoising Video Editing

Understanding Zero-Shot Adversarial Robustness for Large-Scale Models

2022-12-14 · Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang 외

Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks. Yet imperceptible adversarial perturbations can significantly reduce CLIP's performance on new tasks. In thi…

Adversarial Robustness Contrastive Learning Visual Prompt Tuning

Zero-Shot Learning in Industrial Scenarios: New Large-Scale Benchmark, Challenges and Baseline

2026-06-06 · Zekai Zhang, Qinghui Chen, Maomao Xiong, Shijiao Ding 외 arxiv

Large Visual Language Models (LVLMs) have achieved remarkable success in vision tasks. However, the significant differences between industrial and natural scenes make applying LVLMs challenging. Existing LVLMs rely on us…

Zero-Shot Learning Domain Adaptation