Efficient Sketches for Training Data Attribution and Studying the Loss Landscape
The study of modern machine learning models often necessitates storing vast quantities of gradients or Hessian vector products (HVPs). Traditional sketching methods struggle to scale under these memory constraints. We present a novel framework for scalable gradient and HVP sketching, tailored for modern hardware. We provide theoretical guarantees and demonstrate the power of our methods in applications like training data attribution, Hessian spectrum analysis, and intrinsic dimension computation for pre-trained language models. Our work sheds new light on the behavior of pre-trained language models, challenging assumptions about their intrinsic dimensionality and Hessian properties.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Category-Epitomes : Discriminatively Minimalist Representations for Object Categories
Freehand line sketches are an interesting and unique form of visual representation. Typically, such sketches are studied and utilized as an end product of the sketching process. However, we have found it instructive to s…
ObjectA hierarchical residual network with compact triplet-center loss for sketch recognition
With the widespread use of touch-screen devices, it is more and more convenient for people to draw sketches on screen. This results in the demand for automatically understanding the sketches. Thus, the sketch recognition…
Sketch RecognitionTripletBetter Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
Influence functions offer a principled way to trace model predictions back to training data, but their use in deep learning is hampered by the need to invert a large, ill-conditioned Hessian matrix. Approximations such a…
Exploring Crowd Co-creation Scenarios for Sketches
As a first step towards studying the ability of human crowds and machines to effectively co-create, we explore several human-only collaborative co-creation scenarios. The goal in each scenario is to create a digital sket…
GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
Scalable data attribution methods typically assign isolated utility scores to individual training examples. This prevalent additive assumption fundamentally fails to capture critical subset dynamics, including data redun…