Dense Multitask Learning to Reconfigure Comics
In this paper, we develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels to, in turn, facilitate the transfer of comics from one publication channel to another by assisting authors in the task of reconfiguring their narratives. Our MTL method can successfully identify the semantic units as well as the embedded notion of 3D in comic panels. This is a significantly challenging problem because comics comprise disparate artistic styles, illustrations, layouts, and object scales that depend on the authors creative process. Typically, dense image-based prediction techniques require a large corpus of data. Finding an automated solution for dense prediction in the comics domain, therefore, becomes more difficult with the lack of ground-truth dense annotations for the comics images. To address these challenges, we develop the following solutions: 1) we leverage a commonly-used strategy known as unsupervised image-to-image translation, which allows us to utilize a large corpus of real-world annotations; 2) we utilize the results of the translations to develop our multitasking approach that is based on a vision transformer backbone and a domain transferable attention module; 3) we study the feasibility of integrating our MTL dense-prediction method with an existing retargeting method, thereby reconfiguring comics.
Code (0)
등록된 구현이 없습니다.
Tasks
Image-to-Image TranslationUnsupervised Image-To-Image TranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Comprehensive Gold Standard and Benchmark for Comics Text Detection and Recognition
This study focuses on improving the optical character recognition (OCR) data for panels in the COMICS dataset, the largest dataset containing text and images from comic books. To do this, we developed a pipeline for OCR …
Optical Character RecognitionOptical Character Recognition (OCR)Story GenerationText DetectionZooming into Comics: Region-Aware RL Improves Fine-Grained Comic Understanding in Vision-Language Models
Complex visual narratives, such as comics, present a significant challenge to Vision-Language Models (VLMs). Despite excelling on natural images, VLMs often struggle with stylized line art, onomatopoeia, and densely pack…
Reinforcement LearningOne missing piece in Vision and Language: A Survey on Comics Understanding
Vision-language models have recently evolved into versatile systems capable of high performance across a range of tasks, such as document understanding, visual question answering, and grounding, often in zero-shot settin…
document understandingimage-classificationImage ClassificationInstance Segmentation+6Comics Datasets Framework: Mix of Comics datasets for detection benchmarking
Comics, as a medium, uniquely combine text and images in styles often distinct from real-world visuals. For the past three decades, computational research on comics has evolved from basic object detection to more sophist…
BenchmarkingObjectobject-detectionObject Detection+1ComicScene154: A Scene Dataset for Comic Analysis
Comics offer a compelling yet under-explored domain for computational narrative analysis, combining text and imagery in ways distinct from purely textual or audiovisual media. We introduce ComicScene154, a manually annot…
Scene Segmentation