Inferred vs traditional personality assessment: are we predicting the same thing?
Machine learning methods are widely used by researchers to predict psychological characteristics from digital records. To find out whether automatic personality estimates retain the properties of the original traits, we reviewed 220 recent articles. First, we put together the predictive quality estimates from a subset of the studies which declare separation of training, validation, and testing phases, which is critical for ensuring the correctness of quality estimates in machine learning. Only 20% of the reviewed papers met this criterion. To compare the reported quality estimates, we converted them to approximate Pearson correlations. The credible upper limits for correlations between predicted and self-reported personality traits vary in a range between 0.42 and 0.48, depending on the specific trait. The achieved values are substantially below the correlations between traits measured with distinct self-report questionnaires. This suggests that we cannot readily interpret personality predictions as estimates of the original traits or expect predicted personality traits to reproduce known relationships with life outcomes regularly. Next, we complement quality estimates evaluation with evidence on psychometric properties of predicted traits. The few existing results suggest that predicted traits are less stable with time and have lower effective dimensionality than self-reported personality. The predictive text-based models perform substantially worse outside their training domains but stay above a random baseline. The evidence on the relationships between predicted traits and external variables is mixed. Predictive features are difficult to use for validation, due to the lack of prior hypotheses. Thus, predicted personality traits fail to retain important properties of the original characteristics. This calls for the cautious use and targeted validation of the predictive models.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesSimilar Papers 제목 키워드 기반
MentalGame: Predicting Personality-Job Fitness for Software Developers Using Multi-Genre Games and Machine Learning Approaches
Personality assessment in career guidance and personnel selection traditionally relies on self-report questionnaires, which are susceptible to response bias, fatigue, and intentional distortion. Game-based assessment off…
Self-Assessment Tests are Unreliable Measures of LLM Personality
As large language models (LLM) evolve in their capabilities, various recent studies have tried to quantify their behavior using psychological tools created to study human behavior. One such example is the measurement of …
Multiple-choiceHave Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs
Have Large Language Models (LLMs) developed a personality? The short answer is a resounding "We Don't Know!". In this paper, we show that we do not yet have the right tools to measure personality in language models. Pers…
Multiple-choiceDynamic Generation of Personalities with Large Language Models
In the realm of mimicking human deliberation, large language models (LLMs) show promising performance, thereby amplifying the importance of this research area. Deliberation is influenced by both logic and personality. Ho…
Personality GenerationTop-Down Influence? Predicting CEO Personality and Risk Impact from Speech Transcripts
How much does a CEO's personality impact the performance of their company? Management theory posits a great influence, but it is difficult to show empirically---there is a lack of publicly available self-reported persona…
Management