Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers
For simple classification tasks, we show that users can benefit from the advantages of using small, local, generative language models instead of large commercial models without a trade-off in performance or introducing extra labelling costs. These advantages, including those around privacy, availability, cost, and explainability, are important both in commercial applications and in the broader democratisation of AI. Through experiments on 17 sentence classification tasks (2-4 classes), we show that penalised logistic regression on the embeddings from a small LLM equals (and usually betters) the performance of a large LLM in the "tens-of-shot" regime. This requires no more labelled instances than are needed to validate the performance of the large LLM. Finally, we extract stable and sensible explanations for classification decisions.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationregressionSentenceSentence ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Online Cascade Learning for Efficient Inference over Streams
Large Language Models (LLMs) have a natural role in answering complex queries about data streams, but the high computational cost of LLM inference makes them infeasible in many such tasks. We propose online cascade learn…
Imitation LearningMachine Learning, Linear and Bayesian Models for Logistic Regression in Failure Detection Problems
In this work, we study the use of logistic regression in manufacturing failures detection. As a data set for the analysis, we used the data from Kaggle competition Bosch Production Line Performance. We considered the use…
BIG-bench Machine LearningGeneral ClassificationregressionAn Efficient Pseudo-likelihood Method for Sparse Binary Pairwise Markov Network Estimation
The pseudo-likelihood method is one of the most popular algorithms for learning sparse binary pairwise Markov networks. In this paper, we formulate the $L_1$ regularized pseudo-likelihood problem as a sparse multiple log…
regressionImplicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability
Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen, et al., 2021], where the stepsizes are set to be large, resulting in non-mon…
regressionQueue Imbalance as a One-Tick-Ahead Price Predictor in a Limit Order Book
We investigate whether the bid/ask queue imbalance in a limit order book (LOB) provides significant predictive power for the direction of the next mid-price movement. We consider this question both in the context of a si…
regression