Active Fairness Auditing

2022-06-16 · Tom Yan, Chicheng Zhang

The fast spreading adoption of machine learning (ML) by companies across industries poses significant regulatory challenges. One such challenge is scalability: how can regulatory bodies efficiently audit these ML models, ensuring that they are fair? In this paper, we initiate the study of query-based auditing algorithms that can estimate the demographic parity of ML models in a query-efficient manner. We propose an optimal deterministic algorithm, as well as a practical randomized, oracle-efficient algorithm with comparable guarantees. Furthermore, we make inroads into understanding the optimal query complexity of randomized active fairness estimation algorithms. Our first exploration of active fairness estimation aims to put AI governance on firmer theoretical foundations.

📄 PDF Abstract BibTeX arXiv:2206.08450

Code (0)

등록된 구현이 없습니다.

Tasks

Fairness

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

2026-01-06 · David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing 외 arxiv

Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We concep…

The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning

2022-06-20 · Cherie M Poland

In recent years, discussions about fairness in machine learning, AI ethics and algorithm audits have increased. Many entities have developed framework guidance to establish a baseline rubric for fairness and accountabili…

BIG-bench Machine Learning Ethics Fairness

Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-\texorpdfstring{$δ$}{delta} Alignment

2026-06-09 · Junbo Ding, Xin Zang, Chenchen Pan, Donghao Song 외 arxiv

Lipschitz-style individual fairness formalizes the idea that semantically similar examples should receive similar predictions, but its evaluation in multi-task learning (MTL) can be confounded by method-induced represent…

Multi-Task Learning

RECAST: Interactive Auditing of Automatic Toxicity Detection Models

2020-01-07 · Austin P. Wright, Omar Shaikh, Haekyu Park, Will Epperson 외

As toxic language becomes nearly pervasive online, there has been increasing interest in leveraging the advancements in natural language processing (NLP), from very large transformer models to automatically detecting and…

Adversarial Robustness Fairness

Fairness Auditing: Lower Bounds on Company Manipulation

2026-08-01 · Rachit Verma, Padala Manisha, Sujit Gujar arxiv

Fairness audits are increasingly mandated in high-stakes applications such as hiring, lending, and automated decision-making. Recent work has established fundamental impossibility results for black-box fairness auditing,…