Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models
Recently, work in NLP has shifted to few-shot (in-context) learning, with large language models (LLMs) performing well across a range of tasks. However, while fairness evaluations have become a standard for supervised methods, little is known about the fairness of LLMs as prediction systems. Further, common standard methods for fairness involve access to models weights or are applied during finetuning, which are not applicable in few-shot learning. Do LLMs exhibit prediction biases when used for standard NLP tasks? In this work, we explore the effect of shots, which directly affect the performance of models, on the fairness of LLMs as NLP classification systems. We consider how different shot selection strategies, both existing and new demographically sensitive methods, affect model fairness across three standard fairness datasets. We discuss how future work can include LLM fairness evaluations.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessFew-Shot LearningIn-Context LearningSimilar Papers 제목 키워드 기반
Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches
This paper investigates algorithmic bias in language-based models for automated depression detection, focusing on socio-demographic disparities related to gender and race/ethnicity. Models trained using deep neural netwo…
Few-Shot LearningRecommendation Fairness in Social Networks Over Time
In social recommender systems, it is crucial that the recommendation models provide equitable visibility for different demographic groups, such as gender or race. Most existing research has addressed this problem by only…
counterfactualFairnessRecommendation SystemsImproving LLM Group Fairness on Tabular Data via In-Context Learning
Large language models (LLMs) have been shown to be effective on tabular prediction tasks in the low-data regime, leveraging their internal knowledge and ability to learn from instructions and examples. However, LLMs can …
FairnessIn-Context LearningSociocultural knowledge is needed for selection of shots in hate speech detection tasks
We introduce HATELEXICON, a lexicon of slurs and targets of hate speech for the countries of Brazil, Germany, India and Kenya, to aid training and interpretability of models. We demonstrate how our lexicon can be used to…
Few-Shot LearningHate Speech DetectionEvaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
Large vision-language models (LVLMs) have recently achieved significant progress, demonstrating strong capabilities in open-world visual understanding. However, it is not yet clear how LVLMs address demographic biases in…
FairnessQuestion AnsweringSingle Choice QuestionVisual Question Answering