A Pitfall of Learning from User-generated Data: In-depth Analysis of Subjective Class Problem
Research in the supervised learning algorithms field implicitly assumes that training data is labeled by domain experts or at least semi-professional labelers accessible through crowdsourcing services like Amazon Mechanical Turk. With the advent of the Internet, data has become abundant and a large number of machine learning based systems started being trained with user-generated data, using categorical data as true labels. However, little work has been done in the area of supervised learning with user-defined labels where users are not necessarily experts and might be motivated to provide incorrect labels in order to improve their own utility from the system. In this article, we propose two types of classes in user-defined labels: subjective class and objective class - showing that the objective classes are as reliable as if they were provided by domain experts, whereas the subjective classes are subject to bias and manipulation by the user. We define this as a subjective class issue and provide a framework for detecting subjective labels in a dataset without querying oracle. Using this framework, data mining practitioners can detect a subjective class at an early stage of their projects, and avoid wasting their precious time and resources by dealing with subjective class problem with traditional machine learning techniques.
Code (1)
Tasks
BIG-bench Machine LearningSimilar Papers 제목 키워드 기반
Not All Explanations are Created Equal: Investigating the Pitfalls of Current XAI Evaluation
Explainable Artificial Intelligence (XAI) aims to create transparency in modern AI models by offering explanations of the models to human users. There are many ways in which researchers have attempted to evaluate the qua…
Single-GPU GNN Systems: Traps and Pitfalls
The current graph neural network (GNN) systems have established a clear trend of not showing training accuracy results, and directly or indirectly relying on smaller datasets for evaluations majorly. Our in-depth analysi…
GPUGraph Neural NetworkGenerative Explore-Exploit: Training-free Optimization of Generative Recommender Systems using LLM Optimizers
Recommender systems are widely used to suggest engaging content, and Large Language Models (LLMs) have given rise to generative recommenders. Such systems can directly generate items, including for open-set tasks like qu…
General KnowledgeQuestion GenerationQuestion-GenerationRecommendation Systems+1Can Users Detect Biases or Factual Errors in Generated Responses in Conversational Information-Seeking?
Information-seeking dialogues span a wide range of questions, from simple factoid to complex queries that require exploring multiple facets and viewpoints. When performing exploratory searches in unfamiliar domains, user…
DiversityMisinformationResponse GenerationHuman-AI Interactions and Societal Pitfalls
When working with generative artificial intelligence (AI), users may see productivity gains, but the AI-generated content may not match their preferences exactly. To study this effect, we introduce a Bayesian framework i…