Asterisk*: Keep it Simple

2024-11-08 · Andrew Semenov

This paper describes Asterisk, a compact GPT-based model for generating text embeddings. The model uses a minimalist architecture with two layers, two attention heads, and 256 embedding dimensions. By applying knowledge distillation from larger pretrained models, we explore the trade-offs between model size and performance while minimizing computational and memory requirements. The model is primarily evaluated and optimized for classification tasks, with experimental results showing its moderate performance in zero-shot classification across various downstream applications. With additional configuration, the model performance can approach or even surpass that of larger architectures on specific classification tasks.

📄 PDF Abstract BibTeX arXiv:2411.05691

Code (0)

등록된 구현이 없습니다.

Tasks

Classification Knowledge Distillationzero-shot-classificationZero-Shot Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Uma Ferramenta para Projeto de Sistemas de Di\'alogos para Call Center Baseados em Asterisk (A Tool for the Design of Dialog Systems for Asterisk-based Call Centers) [in Portuguese]

2013-01-01 · WS 2013 1 · Estillac Borges Filho, Pedro Batista, Aldebaro Klautau

Asterisk Operator

2025-09-15 · Zixi Li arxiv

We propose the \textbf{Asterisk Operator} ($\ast$-operator), a novel unified framework for abstract reasoning based on Adjacency-Structured Parallel Propagation (ASPP). The operator formalizes structured reasoning tasks …

Online pseudo labeling for polyp segmentation with momentum networks

2022-09-29 · Toan Pham Van, Linh Bao Doan, Thanh Tung Nguyen, Duc Trung Tran 외

Semantic segmentation is an essential task in developing medical image diagnosis systems. However, building an annotated medical dataset is expensive. Thus, semi-supervised methods are significant in this circumstance. I…

Semantic Segmentation

CIRCUS: Circuit Consensus under Uncertainty via Stability Ensembles

2026-02-28 · Swapnil Parekh arxiv

Every mechanistic circuit carries an invisible asterisk: it reflects not just the model's computation, but the analyst's choice of pruning threshold. Change that choice and the circuit changes, yet current practice treat…

Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text

2021-07-07 · ACL 2021 5 · Philippe Laban, Tobias Schnabel, Paul Bennett, Marti A. Hearst

This work presents Keep it Simple (KiS), a new approach to unsupervised text simplification which learns to balance a reward across three properties: fluency, salience and simplicity. We train the model with a novel algo…

Reading Comprehension Text Simplification