Pipelines for Social Bias Testing of Large Language Models
The maturity level of language models is now at a stage in which many companies rely on them to solve various tasks. However, while research has shown how biased and harmful these models are, systematic ways of integrating social bias tests into development pipelines are still lacking. This short paper suggests how to use these verification techniques in development pipelines. We take inspiration from software testing and suggest addressing social bias evaluation as software testing. We hope to open a discussion on the best methodologies to handle social bias testing in language models.
Code (1)
Tasks
software testingSimilar Papers 제목 키워드 기반
BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models
Pretrained Language Models (PLMs) harbor inherent social biases that can result in harmful real-world implications. Such social biases are measured through the probability values that PLMs output for different social gro…
SentenceText GenerationSocial Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias
The rapid growth of Large Language Models (LLMs) has put forward the study of biases as a crucial field. It is important to assess the influence of different types of biases embedded in LLMs to ensure fair use in sensiti…
BenchmarkingBias DetectionFrom If-Statements to ML Pipelines: Revisiting Bias in Code-Generation
Prior work evaluates code generation bias primarily through simple conditional statements, which represent only a narrow slice of real-world programming and reveal solely overt, explicitly encoded bias. We demonstrate th…
Code GenerationTrustworthy Social Bias Measurement
How do we design measures of social bias that we trust? While prior work has introduced several measures, no measure has gained widespread trust: instead, mounting evidence argues we should distrust these measures. In th…
Social Biases in NLP Models as Barriers for Persons with Disabilities
Building equitable and inclusive NLP technologies demands consideration of whether and how social attitudes are represented in ML models. In particular, representations encoded in models often inadvertently perpetuate un…
Sentiment Analysis