FaceGemma: Enhancing Image Captioning with Facial Attributes for Portrait Images
Automated image caption generation is essential for improving the accessibility and understanding of visual content. In this study, we introduce FaceGemma, a model that accurately describes facial attributes such as emotions, expressions, and features. Using FaceAttdb data, we generated descriptions for 2000 faces with the Llama 3 - 70B model and fine-tuned the PaliGemma model with these descriptions. Based on the attributes and captions supplied in FaceAttDB, we created a new description dataset where each description perfectly depicts the human-annotated attributes, including key features like attractiveness, full lips, big nose, blond hair, brown hair, bushy eyebrows, eyeglasses, male, smile, and youth. This detailed approach ensures that the generated descriptions are closely aligned with the nuanced visual details present in the images. Our FaceGemma model leverages an innovative approach to image captioning by using annotated attributes, human-annotated captions, and prompt engineering to produce high-quality facial descriptions. Our method significantly improved caption quality, achieving an average BLEU-1 score of 0.364 and a METEOR score of 0.355. These metrics demonstrate the effectiveness of incorporating facial attributes into image captioning, providing more accurate and descriptive captions for portrait images.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeCaption GenerationDescriptiveImage CaptioningPrompt EngineeringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards
Generating accurate descriptions for online fashion items is important not only for enhancing customers' shopping experiences, but also for the increase of online sales. Besides the need of correctly presenting the attri…
AttributeImage CaptioningReinforcement Learning (RL)SentenceAn Attack on Facial Soft-biometric Privacy Enhancement
In the recent past, different researchers have proposed privacy-enhancing face recognition systems designed to conceal soft-biometric attributes at feature level. These works have reported impressive results, but general…
AttributeDimensionality ReductionFace RecognitionSelf-Supervised Image Captioning with CLIP
Image captioning, a fundamental task in vision-language understanding, seeks to generate accurate natural language descriptions for provided images. Current image captioning approaches heavily rely on high-quality image-…
Image CaptioningInformativenessLanguage ModellingSelf-Supervised Learning+1MyStyle++: A Controllable Personalized Generative Prior
In this paper, we propose an approach to obtain a personalized generative prior with explicit control over a set of attributes. We build upon MyStyle, a recently introduced method, that tunes the weights of a pre-trained…
Exp-Graph: How Connections Learn Facial Attributes in Graph-based Expression Recognition
Facial expression recognition is crucial for human-computer interaction applications such as face animation, video surveillance, affective computing, medical analysis, etc. Since the structure of facial attributes varies…
Facial Expression Recognition