Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection
Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs. This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs. Given the exponential growth of possible test cases, exhaustive testing is impractical; therefore, prioritizing MRs based on their effectiveness in detecting fairness violations is crucial. We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection. Experimental results demonstrate that our proposed prioritization approach improves fault detection rates by 22% compared to random prioritization and 12% compared to distance-based prioritization, while reducing the time to the first failure by 15% and 8%, respectively. Furthermore, our approach performs within 5% of fault-based prioritization in effectiveness, while significantly reducing the computational cost associated with fault labeling. These results validate the effectiveness of diversity-based MR prioritization in enhancing fairness testing for LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Bias DetectionDiversityFairnessFault DetectionSentenceSimilar Papers 제목 키워드 기반
Metamorphic Testing for Fairness Evaluation in Large Language Models: Identifying Intersectional Bias in LLaMA and GPT
Large Language Models (LLMs) have made significant strides in Natural Language Processing but remain vulnerable to fairness-related issues, often reflecting biases inherent in their training data. These biases pose risks…
FairnessMETAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
Large-Language Models (LLMs) have shifted the paradigm of natural language data processing. However, their black-boxed and probabilistic characteristics can lead to potential risks in the quality of outputs in diverse LL…
FairnessLanguage ModelingLanguage ModellingLarge Language ModelFairness Testing in Retrieval-Augmented Generation: How Small Perturbations Reveal Bias in Small Language Models
Large Language Models (LLMs) are widely used across multiple domains but continue to raise concerns regarding security and fairness. Beyond known attack vectors such as data poisoning and prompt injection, LLMs are also …
Sentiment AnalysisText GenerationBias Testing and Mitigation in Black Box LLMs using Metamorphic Relations
The widespread deployment of Large Language Models (LLMs) has intensified concerns about subtle social biases embedded in their outputs. Existing guardrails often fail when faced with indirect or contextually complex bia…
Object-based Metamorphic Testing through Image Structuring
Testing software is often costly due to the need of mass-producing test cases and providing a test oracle for it. This is often referred to as the oracle problem. One method that has been proposed in order to alleviate t…
Object