S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning
VQA Natural Language Explanation (VQA-NLE) task aims to explain the decision-making process of VQA models in natural language. Unlike traditional attention or gradient analysis, free-text rationales can be easier to understand and gain users' trust. Existing methods mostly use post-hoc or self-rationalization models to obtain a plausible explanation. However, these frameworks are bottlenecked by the following challenges: 1) the reasoning process cannot be faithfully responded to and suffer from the problem of logical inconsistency. 2) Human-annotated explanations are expensive and time-consuming to collect. In this paper, we propose a new Semi-Supervised VQA-NLE via Self-Critical Learning (S3C), which evaluates the candidate explanations by answering rewards to improve the logical consistency between answers and rationales. With a semi-supervised learning framework, the S3C can benefit from a tremendous amount of samples without human-annotated explanations. A large number of automatic measures and human evaluations all show the effectiveness of our method. Meanwhile, the framework achieves a new state-of-the-art performance on the two VQA-NLE datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Recently generating natural language explanations has shown very promising results in not only offering interpretable explanations but also providing additional information and supervision for prediction. However, existi…
Explanation GenerationNatural Language UnderstandingTask-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding
Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the major semi-supervised approaches to improve natural language understanding (NLU) tasks with massive amount of unlabeled data. However, it's unc…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Natural Language Inference+4Self-training Improves Pre-training for Natural Language Understanding
Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain add…
Data AugmentationFew-Shot LearningKnowledge DistillationNatural Language Understanding+3A Comparative Study of Pre-training and Self-training
Pre-training and self-training are two approaches to semi-supervised learning. The comparison between pre-training and self-training has been explored. However, the previous works led to confusing findings: self-training…
Data AugmentationNatural Language InferenceSentiment AnalysisCross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem
Natural language explanations (NLEs) are vital for elucidating the reasoning behind large language model (LLM) decisions. Many techniques have been developed to generate NLEs using LLMs. However, like humans, LLMs might …
Explanation GenerationLanguage ModellingLarge Language Model