Simulating and Analysing Human Survey Responses with Large Language Models: A Case Study in Energy Stated Preference
Survey research plays a crucial role in studies by capturing consumer preferences and informing policy decisions. Stated preference (SP) surveys help researchers understand how individuals make trade-offs in hypothetical, potentially futuristic, scenarios. However, traditional methods are costly, time-consuming, and affected by respondent fatigue and ethical constraints. Large language models (LLMs) have shown remarkable capabilities in generating human-like responses, prompting interest in their use in survey research. This study investigates LLMs for simulating consumer choices in energy-related SP surveys and explores their integration into data collection and analysis workflows. Test scenarios were designed to assess the simulation performance of several LLMs (LLaMA 3.1, Mistral, GPT-3.5, DeepSeek-R1) at individual and aggregated levels, considering prompt design, in-context learning (ICL), chain-of-thought (CoT) reasoning, model types, integration with traditional choice models, and potential biases. While LLMs achieve accuracy above random guessing, performance remains insufficient for practical simulation use. Cloud-based LLMs do not consistently outperform smaller local models. DeepSeek-R1 achieves the highest average accuracy (77%) and outperforms non-reasoning LLMs in accuracy, factor identification, and choice distribution alignment. Previous SP choices are the most effective input; longer prompts with more factors reduce accuracy. Mixed logit models can support LLM prompt refinement. Reasoning LLMs show potential in data analysis by indicating factor significance, offering a qualitative complement to statistical models. Despite limitations, pre-trained LLMs offer scalability and require minimal historical data. Future work should refine prompts, further explore CoT reasoning, and investigate fine-tuning techniques.
Code (0)
등록된 구현이 없습니다.
Tasks
In-Context LearningSurveyMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using Large Language Models
In recent years, large language models (LLMs) have attracted attention due to their ability to generate human-like text. As surveys and opinion polls remain key tools for gauging public attitudes, there is increasing int…
SurveyExtracting Keywords from Open-Ended Business Survey Questions
Open-ended survey data constitute an important basis in research as well as for making business decisions. Collecting and manually analysing free-text survey data is generally more costly than collecting and analysing su…
Multiple-choiceSurveyOverstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibility
Large language models (LLMs) are increasingly used as proxies for human judgment in computational social science, yet their ability to reproduce patterns of susceptibility to misinformation remains unclear. We test wheth…
An Analysis of Large Language Models for Simulating User Responses in Surveys
Using Large Language Models (LLMs) to simulate user opinions has received growing attention. Yet LLMs, especially trained with reinforcement learning from human feedback (RLHF), are known to exhibit biases toward dominan…
Reinforcement LearningQuestion AnsweringPsychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the c…