Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions
We introduce and study artificial impressions--patterns in LLMs' internal representations of prompts that resemble human impressions and stereotypes based on language. We fit linear probes on generated prompts to predict impressions according to the two-dimensional Stereotype Content Model (SCM). Using these probes, we study the relationship between impressions and downstream model behavior as well as prompt features that may inform such impressions. We find that LLMs inconsistently report impressions when prompted, but also that impressions are more consistently linearly decodable from their hidden representations. Additionally, we show that artificial impressions of prompts are predictive of the quality and use of hedging in model responses. We also investigate how particular content, stylistic, and dialectal features in prompts impact LLM impressions.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Predicting Social Perception from Faces: A Deep Learning Approach
Warmth and competence represent the fundamental traits in social judgment that determine emotional reactions and behavioral intentions towards social targets. This research investigates whether an algorithm can learn vis…
Deep LearningGeneral ClassificationThe current status of large language models in summarizing radiology report impressions
Large language models (LLMs) like ChatGPT show excellent capabilities in various natural language processing tasks, especially for text generation. The effectiveness of LLMs in summarizing radiology report impressions re…
Text GenerationTraining Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression
To improve user engagement during conversations with dialogue systems, we must improve individual dialogue responses and dialogue impressions such as consistency, personality, and empathy throughout the entire dialogue. …
Evaluating Large Language Models for Radiology Natural Language Processing
The rise of large language models (LLMs) has marked a pivotal shift in the field of natural language processing (NLP). LLMs have revolutionized a multitude of domains, and they have made a significant impact in the medic…
Automatic Personalized Impression Generation for PET Reports Using Large Language Models
In this study, we aimed to determine if fine-tuned large language models (LLMs) can generate accurate, personalized impressions for whole-body PET reports. Twelve language models were trained on a corpus of PET reports u…