Measuring the right thing: justifying metrics in AI impact assessments
AI Impact Assessments are only as good as the measures used to assess the impact of these systems. It is therefore paramount that we can justify our choice of metrics in these assessments, especially for difficult to quantify ethical and social values. We present a two-step approach to ensure metrics are properly motivated. First, a conception needs to be spelled out (e.g. Rawlsian fairness or fairness as solidarity) and then a metric can be fitted to that conception. Both steps require separate justifications, as conceptions can be judged on how well they fit with the function of, for example, fairness. We argue that conceptual engineering offers helpful tools for this step. Second, metrics need to be fitted to a conception. We illustrate this process through an examination of competing fairness metrics to illustrate that here the additional content that a conception offers helps us justify the choice for a specific metric. We thus advocate that impact assessments are not only clear on their metrics, but also on the conceptions that motivate those metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessSimilar Papers 제목 키워드 기반
Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity
Is it possible to evaluate the moral cognition of complex artificial agents? In this work, we take a look at one aspect of morality: `doing the right thing for the right reasons.' We propose a behavior-based analysis of …
Deep Reinforcement LearningMeta Reinforcement Learningreinforcement-learningReinforcement LearningUnsupervised learning for economic risk evaluation in the context of Covid-19 pandemic
Justifying draconian measures during the Covid-19 pandemic was difficult not only because of the restriction of individual rights, but also because of its economic impact. The objective of this work is to present a machi…
Time SeriesTime Series AnalysisTime Series ForecastingRethinking Oversmoothing in Graph Neural Networks: A Rank-Based Perspective
Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, node embeddings become increasingly similar, and model performance drops sharply. Traditionally, oversmoothing …
Towards the Right Kind of Fairness in AI
Fairness is a concept of justice. Various definitions exist, some of them conflicting with each other. In the absence of an uniformly accepted notion of fairness, choosing the right kind for a specific situation has alwa…
EthicsFairnessAre We Scaling the Right Thing? A System Perspective on Test-Time Scaling
Test-time scaling (TTS) has recently emerged as a promising direction to exploit the hidden reasoning capabilities of pre-trained large language models (LLMs). However, existing scaling methods narrowly focus on the comp…