Evaluating Machine Expertise: How Graduate Students Develop Frameworks for Assessing GenAI Content
This paper examines how graduate students develop frameworks for evaluating machine-generated expertise in web-based interactions with large language models (LLMs). Through a qualitative study combining surveys, LLM interaction transcripts, and in-depth interviews with 14 graduate students, we identify patterns in how these emerging professionals assess and engage with AI-generated content. Our findings reveal that students construct evaluation frameworks shaped by three main factors: professional identity, verification capabilities, and system navigation experience. Rather than uniformly accepting or rejecting LLM outputs, students protect domains central to their professional identities while delegating others--with managers preserving conceptual work, designers safeguarding creative processes, and programmers maintaining control over core technical expertise. These evaluation frameworks are further influenced by students' ability to verify different types of content and their experience navigating complex systems. This research contributes to web science by highlighting emerging human-genAI interaction patterns and suggesting how platforms might better support users in developing effective frameworks for evaluating machine-generated expertise signals in AI-mediated web environments.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Statistician Teaches Deep Learning
Deep learning (DL) has gained much attention and become increasingly popular in modern data science. Computer scientists led the way in developing deep learning techniques, so the ideas and perspectives can seem alien to…
Cultural Vocal Bursts Intensity PredictionDeep Learning"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students
This study evaluates the effectiveness of various large language models (LLMs) in performing tasks common among undergraduate computer science students. Although a number of research studies in the computing education co…
Code GenerationEmora STDM: A Versatile Framework for Innovative Dialogue System Development
This demo paper presents Emora STDM (State Transition Dialogue Manager), a dialogue system development framework that provides novel workflows for rapid prototyping of chat-based dialogue managers as well as collaborativ…
Dialogue ManagementManagementData Science Education in Undergraduate Physics: Lessons Learned from a Community of Practice
It is becoming increasingly important that physics educators equip their students with the skills to work with data effectively. However, many educators may lack the necessary training and expertise in data science to te…
A Low-Cost Robot Science Kit for Education with Symbolic Regression for Hypothesis Discovery and Validation
The next generation of physical science involves robot scientists - autonomous physical science systems capable of experimental design, execution, and analysis in a closed loop. Such systems have shown real-world success…
Experimental DesignSymbolic Regression