Inverse distance weighting attention
We report the effects of replacing the scaled dot-product (within softmax) attention with the negative-log of Euclidean distance. This form of attention simplifies to inverse distance weighting interpolation. Used in simple one hidden layer networks and trained with vanilla cross-entropy loss on classification problems, it tends to produce a key matrix containing prototypes and a value matrix with corresponding logits. We also show that the resulting interpretable networks can be augmented with manually-constructed prototypes to perform low-impact handling of special cases.
Code (1)
Similar Papers 제목 키워드 기반
Active Learning for Regression by Inverse Distance Weighting
This paper proposes an active learning (AL) algorithm to solve regression problems based on inverse-distance weighting functions for selecting the feature vectors to query. The algorithm has the following features: (i) s…
Active LearningClassificationregressionGlobal optimization via inverse distance weighting and radial basis functions
Global optimization problems whose objective function is expensive to evaluate can be solved effectively by recursively fitting a surrogate function to function samples and minimizing an acquisition function to generate …
Bayesian Optimizationglobal-optimizationAugmented Inverse Hybrid Weighting: Robust Inference under Deterministic and Random Distribution Shifts
Reweighting source samples to match a target covariate distribution is a standard response to distribution shift when generalizing evidence from one population to another. This strategy is well suited to deterministic, l…
Inverse Distance Aggregation for Federated Learning with Non-IID Data
Federated learning (FL) has been a promising approach in the field of medical imaging in recent years. A critical problem in FL, specifically in medical scenarios is to have a more accurate shared model which is robust t…
Federated LearningOrthographic Codes and the Neighborhood Effect: Lessons from Information Theory
We consider the orthographic neighborhood effect: the effect that words with more orthographic similarity to other words are read faster. The neighborhood effect serves as an important control variable in psycholinguisti…