Redefining Research Crowdsourcing: Incorporating Human Feedback with LLM-Powered Digital Twins
Crowd work platforms like Amazon Mechanical Turk and Prolific are vital for research, yet workers' growing use of generative AI tools poses challenges. Researchers face compromised data validity as AI responses replace authentic human behavior, while workers risk diminished roles as AI automates tasks. To address this, we propose a hybrid framework using digital twins, personalized AI models that emulate workers' behaviors and preferences while keeping humans in the loop. We evaluate our system with an experiment (n=88 crowd workers) and in-depth interviews with crowd workers (n=5) and social science researchers (n=4). Our results suggest that digital twins may enhance productivity and reduce decision fatigue while maintaining response quality. Both researchers and workers emphasized the importance of transparency, ethical data use, and worker agency. By automating repetitive tasks and preserving human engagement for nuanced ones, digital twins may help balance scalability with authenticity.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Evaluating LLM-corrupted Crowdsourcing Data Without Ground Truth
The recent success of generative AI highlights the crucial role of high-quality human feedback in building trustworthy AI systems. However, the increasing use of large language models (LLMs) by crowdsourcing workers pose…
Multiple-choiceCrowd-PrefRL: Preference-Based Reward Learning from Crowds
Preference-based reinforcement learning (RL) provides a framework to train AI agents using human feedback through preferences over pairs of behaviors, enabling agents to learn desired behaviors when it is difficult to sp…
Reinforcement Learning (RL)PlanIt: A Crowdsourcing Approach for Learning to Plan Paths from Large Scale Preference Feedback
We consider the problem of learning user preferences over robot trajectories for environments rich in objects and humans. This is challenging because the criterion defining a good trajectory varies with users, tasks and …
Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcing
To better serve users' demands in mobile applications (e.g., navigation), mobile crowdsourcing platforms can iteratively align large language model (LLM)-generated content (e.g., AI-generated traffic condition prediction…
The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices
The technical progression of artificial intelligence (AI) research has been built on breakthroughs in fields such as computer science, statistics, and mathematics. However, in the past decade AI researchers have increasi…
Ethics