SSA: Improving Performance With a Better Scoring Function

2025-08-20 · Omar Naim, Swarnadeep Bhar, Jérôme Bolte, Nicholas Asher arxiv

While transformer models exhibit strong in-context learning (ICL) abilities, they often fail to generalize under simple distribution shifts. We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor. We propose \textbf{Scaled Signed Averaging (SSA)}, a novel attention scoring function that mitigates these failures. SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures.

📄 PDF Abstract BibTeX arXiv:2508.14685

Code (0)

등록된 구현이 없습니다.

Efficient Relation-aware Scoring Function Search for Knowledge Graph Embedding

2021-04-22 · Shimin Di, Quanming Yao, Yongqi Zhang, Lei Chen

The scoring function, which measures the plausibility of triplets in knowledge graphs (KGs), is the key to ensure the excellent performance of KG embedding, and its design is also an important problem in the literature. …

AutoML Graph Embedding Knowledge Graph Embedding Knowledge Graphs+1

On the Efficacy of Generalization Error Prediction Scoring Functions

2023-03-23 · Puja Trivedi, Danai Koutra, Jayaraman J. Thiagarajan

Generalization error predictors (GEPs) aim to predict model performance on unseen distributions by deriving dataset-level error estimates from sample-level scores. However, GEPs often utilize disparate mechanisms (e.g., …

Diversity Prediction

Bilinear Scoring Function Search for Knowledge Graph Learning

2021-07-01 · Yongqi Zhang, Quanming Yao, James Tin-Yau Kwok

Learning embeddings for entities and relations in knowledge graph (KG) have benefited many downstream tasks. In recent years, scoring functions, the crux of KG learning, have been human-designed to measure the plausibili…

AutoML Graph Embedding Graph Learning Knowledge Graph Embedding

A transfer learning based approach for pronunciation scoring

2021-11-01 · Marcelo Sancinetti, Jazmin Vidal, Cyntia Bonomi, Luciana Ferrer

Phone-level pronunciation scoring is a challenging task, with performance far from that of human annotators. Standard systems generate a score for each phone in a phrase using models trained for automatic speech recognit…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Phone-level pronunciation scoring speech-recognition+2

Ligand Pose Optimization with Atomic Grid-Based Convolutional Neural Networks

2017-10-20 · Matthew Ragoza, Lillian Turner, David Ryan Koes

Docking is an important tool in computational drug discovery that aims to predict the binding pose of a ligand to a target protein through a combination of pose scoring and optimization. A scoring function that is differ…

Drug Discovery