Can Offline Metrics Measure Explanation Goals? A Comparative Survey Analysis of Offline Explanation Metrics in Recommender Systems
Explanations in a Recommender System (RS) provide reasons for recommendations to users and can enhance transparency, persuasiveness, engagement, and trust-known as explanation goals. Evaluating the effectiveness of explanation algorithms offline remains challenging due to subjectivity. Initially, we conducted a literature review on current offline metrics, revealing that algorithms are often assessed with anecdotal evidence, offering convincing examples, or with metrics that don't align with human perception. We investigated whether, in explanations connecting interacted and recommended items based on shared content, the selection of item attributes and interacted items affects explanation goals. Metrics measuring the diversity and popularity of attributes and the recency of item interactions were used to evaluate explanations from three state-of-the-art agnostic algorithms across six recommendation systems. These offline metrics were compared with results from an online user study. Our findings reveal a trade-off: transparency and trust relate to popular properties, while engagement and persuasiveness are linked to diversified properties. This study contributes to the development of more robust evaluation methods for explanation algorithms in recommender systems.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityPersuasivenessRecommendation SystemsSentenceSimilar Papers 제목 키워드 기반
Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity
The use of eXplainable Artificial Intelligence (XAI) systems has introduced a set of challenges that need resolution. The XAI robustness, or stability, has been one of the goals of the community from its beginning. Multi…
AvgExplainable artificial intelligenceExplainable Artificial Intelligence (XAI)SensitivityINFACT: An Online Human Evaluation Framework for Conversational Recommendation
Conversational recommender systems (CRS) are interactive agents that support their users in recommendation-related goals through multi-turn conversations. Generally, a CRS can be evaluated in various dimensions. Today's …
Conversational RecommendationRecommendation SystemsA Comparative User Evaluation of XRL Explanations using Goal Identification
Debugging is a core application of explainable reinforcement learning (XRL) algorithms; however, limited comparative evaluations have been conducted to understand their relative performance. We propose a novel evaluation…
Reinforcement LearningComparative Explanations of Recommendations
As recommendation is essentially a comparative (or ranking) process, a good explanation should illustrate to users why an item is believed to be better than another, i.e., comparative explanations about the recommended i…
Explainable RecommendationRecommendation SystemsSentenceComparing Fair Ranking Metrics
Ranked lists are frequently used by information retrieval (IR) systems to present results believed to be relevant to the users information need. Fairness is a relatively new but important aspect of these rankings to meas…
FairnessInformation RetrievalRecommendation SystemsRetrieval