Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?
Pre-trained language models consider the context of neighboring words and documents but lack any author context of the human generating the text. However, language depends on the author's states, traits, social, situational, and environmental attributes, collectively referred to as human context (Soni et al., 2024). Human-centered natural language processing requires incorporating human context into language models. Currently, two methods exist: pre-training with 1) group-wise attributes (e.g., over-45-year-olds) or 2) individual traits. Group attributes are simple but coarse -- not all 45-year-olds write the same way -- while individual traits allow for more personalized representations, but require more complex modeling and data. It is unclear which approach benefits what tasks. We compare pre-training models with human context via 1) group attributes, 2) individual users, and 3) a combined approach on five user- and document-level tasks. Our results show that there is no best approach, but that human-centered language modeling holds avenues for different methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Age EstimationLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
DevBench: A multimodal developmental benchmark for language learning
How (dis)similar are the learning trajectories of vision-language models and children? Recent modeling work has attempted to understand the gap between models' and humans' data efficiency by constructing models trained o…
Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned
In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer a…
Speaking Multiple Languages Affects the Moral Bias of Language Models
Pre-trained multilingual language models (PMLMs) are commonly used when dealing with data from multiple languages and cross-lingual transfer. However, PMLMs are trained on varying amounts of data for each language. In pr…
Cross-Lingual TransferBabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?
Language models have seen significant growth in the size of their corpus, leading to notable performance improvements. Yet, there has been limited progress in developing models that handle smaller, more human-like datase…
Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?
To decipher the algorithm underlying the human brain's language representation, previous work probed brain responses to language input with pre-trained artificial neural network (ANN) models fine-tuned on NLU tasks. Howe…
ChunkingMulti-Task Learning