Style-Aware Contrastive Learning for Multi-Style Image Captioning
Existing multi-style image captioning methods show promising results in generating a caption with accurate visual content and desired linguistic style. However, existing methods overlook the relationship between linguistic style and visual content. To overcome this drawback, we propose style-aware contrastive learning for multi-style image captioning. First, we present a style-aware visual encoder with contrastive learning to mine potential visual content relevant to style. Moreover, we propose a style-aware triplet contrast objective to distinguish whether the image, style and caption matched. To provide positive and negative samples for contrastive learning, we present three retrieval schemes: object-based retrieval, RoI-based retrieval and triplet-based retrieval, and design a dynamic trade-off function to calculate retrieval scores. Experimental results demonstrate that our approach achieves state-of-the-art performance. In addition, we conduct an extensive analysis to verify the effectiveness of our method.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningImage CaptioningRetrievalTripletMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CONSTANT: Towards High-Quality One-Shot Handwriting Generation with Patch Contrastive Enhancement and Style-Aware Quantization
One-shot styled handwriting image generation, despite achieving impressive results in recent years, remains challenging due to the difficulty in capturing the intricate and diverse characteristics of human handwriting by…
Image GenerationArtistic Style Transfer with Internal-external Learning and Contrastive Learning
Although existing artistic style transfer methods have achieved significant improvement with deep neural networks, they still suffer from artifacts such as disharmonious colors and repetitive patterns. Motivated by this,…
Contrastive LearningStyle TransferName Your Style: An Arbitrary Artist-aware Image Style Transfer
Image style transfer has attracted widespread attention in the past few years. Despite its remarkable results, it requires additional style images available as references, making it less flexible and inconvenient. Using …
Style TransferArtNeRF: A Stylized Neural Field for 3D-Aware Cartoonized Face Synthesis
Recent advances in generative visual models and neural radiance fields have greatly boosted 3D-aware image synthesis and stylization tasks. However, previous NeRF-based work is limited to single scene stylization, traini…
3D-Aware Image SynthesisContrastive LearningFace GenerationImage Generation+3Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning
In this work, we tackle the challenging problem of arbitrary image style transfer using a novel style feature representation learning method. A suitable style representation, as a key component in image stylization tasks…
Contrastive LearningImage StylizationRepresentation LearningStyle Transfer