Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation
We propose a new active learning (AL) framework, Active Learning++, which can utilize an annotator's labels as well as its rationale. Annotators can provide their rationale for choosing a label by ranking input features based on their importance for a given query. To incorporate this additional input, we modified the disagreement measure for a bagging-based Query by Committee (QBC) sampling strategy. Instead of weighing all committee models equally to select the next instance, we assign higher weight to the committee model with higher agreement with the annotator's ranking. Specifically, we generated a feature importance-based local explanation for each committee model. The similarity score between feature rankings provided by the annotator and the local model explanation is used to assign a weight to each corresponding committee model. This approach is applicable to any kind of ML model using model-agnostic techniques to generate local explanation such as LIME. With a simulation study, we show that our framework significantly outperforms a QBC based vanilla AL framework.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningFeature ImportanceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales
Beyond exploring disaggregated labels for modeling perspectives, annotator rationales provide fine-grained signals of individual perspectives. In this work, we propose a framework for jointly modeling annotator-specific …
Natural Language InferenceExplanation GenerationFree-text Rationale Generation under Readability Level Control
Free-text rationales justify model decisions in natural language and thus become likable and accessible among approaches to explanation across many tasks. However, their effectiveness can be hindered by misinterpretation…
HallucinationText GenerationKnowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations
Models that generate extractive rationales (i.e., subsets of features) or natural language explanations (NLEs) for their predictions are important for explainable AI. While an extractive rationale provides a quick view o…
Decision MakingBeyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture
Real-world domain experts (e.g., doctors) rarely annotate only a decision label in their day-to-day workflow without providing explanations. Yet, existing low-resource learning techniques, such as Active Learning (AL), t…
Active LearningDecision MakingDiversityExplanation Generation+1Table-Text Alignment: Explaining Claim Verification Against Tables in Scientific Papers
Scientific claim verification against tables typically requires predicting whether a claim is supported or refuted given a table. However, we argue that predicting the final label alone is insufficient: it reveals little…
Claim Verification