Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild
We argue that we need to evaluate model interpretability methods 'in the wild', i.e., in situations where professionals make critical decisions, and models can potentially assist them. We present an in-the-wild evaluation of token attribution based on Deep Taylor Decomposition, with professional journalists performing reliability assessments. We find that using this method in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to faster and better human decision-making, as well as a more critical attitude toward news sources among the journalists. We present a comparison of human and model rationales, as well as a qualitative analysis of the journalists' experiences with machine-in-the-loop decision making.
Code (1)
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
A Rigorous Study Of The Deep Taylor Decomposition
Saliency methods attempt to explain deep neural networks by highlighting the most salient features of a sample. Some widely used methods are based on a theoretical framework called Deep Taylor Decomposition (DTD), which …
Exploring Kolmogorov-Arnold networks for realistic image sharpness assessment
Score prediction is crucial in evaluating realistic image sharpness based on collected informative features. Recently, Kolmogorov-Arnold networks (KANs) have been developed and witnessed remarkable success in data fittin…
Image Quality AssessmentKolmogorov-Arnold NetworksA Relation Spectrum Inheriting Taylor Series: Muscle Synergy and Coupling for Hand
There are two famous function decomposition methods in math: Taylor Series and Fourier Series. Fourier series developed into Fourier spectrum, which was applied to signal decomposition\analysis. However, because the Tayl…
MathRelationTaylor expansion-based Kolmogorov-Arnold network for blind image quality assessment
Kolmogorov-Arnold Network (KAN) has attracted growing interest for its strong function approximation capability. In our previous work, KAN and its variants were explored in score regression for blind image quality assess…
Blind Image Quality AssessmentComputational EfficiencyImage Quality AssessmentregressionHaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild
Hallucinations pose a significant challenge to the reliability of large language models (LLMs) in critical domains. Recent benchmarks designed to assess LLM hallucinations within conventional NLP tasks, such as knowledge…
HallucinationQuestion AnsweringRAGRetrieval+1