Better Training Data Attribution via Better Inverse Hessian-Vector Products
Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
Influence functions offer a principled way to trace model predictions back to training data, but their use in deep learning is hampered by the need to invert a large, ill-conditioned Hessian matrix. Approximations such a…
Robust Attribution Regularization
An emerging problem in trustworthy machine learning is to train models that produce robust interpretations for their predictions. We take a step towards solving this problem through the lens of axiomatic attribution of n…
CAWA: An Attention-Network for Credit Attribution
Credit attribution is the task of associating individual parts in a document with their most appropriate class labels. It is an important task with applications to information retrieval and text summarization. When label…
Information RetrievalMultilabel Text ClassificationRetrievalSentence+3See Beyond a Single View: Multi-Attribution Learning Leads to Better Conversion Rate Prediction
Conversion rate (CVR) prediction is a core component of online advertising systems, where the attribution mechanisms-rules for allocating conversion credit across user touchpoints-fundamentally determine label generation…
Efficient Ensembles Improve Training Data Attribution
Training data attribution (TDA) methods aim to quantify the influence of individual training data points on the model predictions, with broad applications in data-centric AI, such as mislabel detection, data selection, a…
AttributeComputational Efficiency