Long-form analogies generated by chatGPT lack human-like psycholinguistic properties
Psycholinguistic analyses provide a means of evaluating large language model (LLM) output and making systematic comparisons to human-generated text. These methods can be used to characterize the psycholinguistic properties of LLM output and illustrate areas where LLMs fall short in comparison to human-generated text. In this work, we apply psycholinguistic methods to evaluate individual sentences from long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT. We perform a supervised classification analysis using 78 features extracted from Coh-metrix that analyze text cohesion, language, and readability (Graesser et. al., 2004). Results illustrate high performance for classifying student-generated and chatGPT-generated analogies. To evaluate which features contribute most to model performance, we use a hierarchical clustering approach. Results from this analysis illustrate several linguistic differences between the two sources.
Code (0)
등록된 구현이 없습니다.
Tasks
FormLanguage ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
"Like a Nesting Doll": Analyzing Recursion Analogies Generated by CS Students using Large Language Models
Grasping complex computing concepts often poses a challenge for students who struggle to anchor these new ideas to familiar experiences and understandings. To help with this, a good analogy can bridge the gap between unf…
DiversityStoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding
Analogy-making between narratives is crucial for human reasoning. In this paper, we evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus, \tex…
Multiple-choiceNatural Language UnderstandingSentenceSentence Embedding+1Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction
The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. Despite the attention previous research has given to wo…
Novel ConceptsQuestion AnsweringCan AI be a Teaching Partner? Evaluating ChatGPT, Gemini, and DeepSeek across Three Teaching Strategies
There are growing promises that Large Language Models (LLMs) can support students' learning by providing explanations, feedback, and guidance. However, despite their rapid adoption and widespread attention, there is stil…
ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies
Analogy-making is central to human cognition, allowing us to adapt to novel situations -- an ability that current AI systems still lack. Most analogy datasets today focus on simple analogies (e.g., word analogies); datas…
Multiple-choice