The human unlikeness of neural language models in next-word prediction
The training objective of unidirectional language models (LMs) is similar to a psycholinguistic benchmark known as the cloze task, which measures next-word predictability. However, LMs lack the rich set of experiences that people do, and humans can be highly creative. To assess human parity in these models{'} training objective, we compare the predictions of three neural language models to those of human participants in a freely available behavioral dataset (Luke {\&} Christianson, 2016). Our results show that while neural models show a close correspondence to human productions, they nevertheless assign insufficient probability to how often speakers guess upcoming words, especially for open-class content words.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction
Contemporary autoregressive language models (LMs) trained purely on corpus data have been shown to capture numerous features of human incremental processing. However, past work has also suggested dissociations between co…
Humans and language models diverge when predicting repeating text
Language models that are trained on the next-word prediction task have been shown to accurately model human behavior in word prediction and reading speed. In contrast with these findings, we present a scenario in which t…
In-Context LearningLearning to vary: Teaching LMs to reproduce human linguistic variability in next-word prediction
Natural language generation (NLG) tasks are often subject to inherent variability; e.g. predicting the next word given a context has multiple valid responses, evident when asking multiple humans to complete the task. Whi…
Next word prediction based on the N-gram model for Kurdish Sorani and Kurmanji
Next word prediction is an input technology that simplifies the process of typing by suggesting the next word to a user to select, as typing in a conversation consumes time. A few previous studies have focused on the Kur…
PredictionTo model human linguistic prediction, make LLMs less superhuman
When we read, we make predictions about upcoming words; these predictions influence our reading behavior. The success of large language models (LLMs), which, like humans, make predictions about upcoming words, has motiva…