COMIC: Towards A Compact Image Captioning Model with Attention
Recent works in image captioning have shown very promising raw performance. However, we realize that most of these encoder-decoder style networks with attention do not scale naturally to large vocabulary size, making them difficult to be deployed on embedded system with limited hardware resources. This is because the size of word and output embedding matrices grow proportionally with the size of vocabulary, adversely affecting the compactness of these networks. To address this limitation, this paper introduces a brand new idea in the domain of image captioning. That is, we tackle the problem of compactness of image captioning models which is hitherto unexplored. We showed that, our proposed model, named COMIC for COMpact Image Captioning, achieves comparable results in five common evaluation metrics with state-of-the-art approaches on both MS-COCO and InstaPIC-1.1M datasets despite having an embedding vocabulary size that is 39x - 99x smaller. The source code and models are available at: https://github.com/jiahuei/COMIC-Compact-Image-Captioning-with-Attention
Code (2)
Tasks
DecoderImage CaptioningSimilar Papers 제목 키워드 기반
ComicsPAP: understanding comic strips by picking the correct panel
Large multimodal models (LMMs) have made impressive strides in image captioning, VQA, and video comprehension, yet they still struggle with the intricate temporal and spatial cues found in comics. To address this gap, we…
Image CaptioningVisual Question Answering (VQA)Visual representation of negation: Real world data analysis on comic image design
There has been a widely held view that visual representations (e.g., photographs and illustrations) do not depict negation, for example, one that can be expressed by a sentence "the train is not coming". This view is emp…
Image Captioningimage-classificationImage ClassificationNegation+1ComiCap: A VLMs pipeline for dense captioning of Comic Panels
The comic domain is rapidly advancing with the development of single- and multi-page analysis and synthesis models. Recent benchmarks and datasets have been introduced to support and assess models' capabilities in tasks …
AttributeDense CaptioningSpeaker IdentificationEstimating Image Depth in the Comics Domain
Estimating the depth of comics images is challenging as such images a) are monocular; b) lack ground-truth depth annotations; c) differ across different artistic styles; d) are sparse and noisy. We thus, use an off-the-s…
Depth EstimationDepth PredictionImage-to-Image TranslationTranslation+1Dense Multitask Learning to Reconfigure Comics
In this paper, we develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels to, in turn, facilitate the transfer of comics from one publication channel to another by assisting authors in the…
Image-to-Image TranslationUnsupervised Image-To-Image Translation