TreeTalk: Composition and Compression of Trees for Image Descriptions
We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions. We investigate two related tasks: image caption generalization and generation, where the former is an optional subtask of the latter. The high-level idea of our approach is to harvest expressive phrases (as tree fragments) from existing image descriptions, then to compose a new description by selectively combining the extracted (and optionally pruned) tree fragments. Key algorithmic components are tree composition and compression, both integrating tree structure with sequence structure. Our proposed system attains significantly better performance than previous approaches for both image caption generalization and generation. In addition, our work is the first to show the empirical benefit of automatically generalized captions for composing natural image descriptions.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningImage RetrievalSimilar Papers 제목 키워드 기반
Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding
Vision-language models such as CLIP often struggle to faithfully understand long, detail-rich captions, relying on dominant scene cues while overlooking fine-grained visual evidence. We propose a hierarchical vision-lang…
Text RetrievalEvoZip: Efficient Compression of Large Collections of Evolutionary Trees
Phylogenetic trees represent evolutionary relationships among sets of organisms. Popular phylogenetic reconstruction approaches typically yield hundreds to thousands of trees on a common leafset. Storing and sharing such…
RetrievalImage Compression Using Singular Value Decomposition
Images are a substantial portion of the internet, making efficient compression important for reducing storage and bandwidth demands. This study investigates the use of Singular Value Decomposition and low-rank matrix app…
Image CompressionText + Sketch: Image Compression at Ultra Low Rates
Recent advances in text-to-image generative models provide the ability to generate high-quality images from short text descriptions. These foundation models, when pre-trained on billion-scale datasets, are effective for …
Image CompressionText to SpeechTensor Shape Search for Optimum Data Compression
Various tensor decomposition methods have been proposed for data compression. In real world applications of the tensor decomposition, selecting the tensor shape for the given data poses a challenge and the shape of the t…
Data CompressionTensor Decomposition