Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters
The advancement of Large Language Models (LLMs) has led to their widespread use across a broad spectrum of tasks including decision making. Prior studies have compared the decision making abilities of LLMs with those of humans from a psychological perspective. However, these studies have not always properly accounted for the sensitivity of LLMs' behavior to hyperparameters and variations in the prompt. In this study, we examine LLMs' performance on the Horizon decision making task studied by Binz and Schulz (2023) analyzing how LLMs respond to variations in prompts and hyperparameters. By experimenting on three OpenAI language models possessing different capabilities, we observe that the decision making abilities fluctuate based on the input prompts and temperature settings. Contrary to previous findings language models display a human-like exploration exploitation tradeoff after simple adjustments to the prompt.
Code (1)
Tasks
Decision MakingSensitivitySimilar Papers 제목 키워드 기반
Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models
In this paper, we investigate the phenomena of "selection biases" in Large Language Models (LLMs), focusing on problems where models are tasked with choosing the optimal option from an ordered sequence. We delve into bia…
Decision MakingSensitivityReinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting
Reinforcement learning (RL) is a promising approach for aligning large language models (LLMs) knowledge with sequential decision-making tasks. However, few studies have thoroughly investigated the impact on LLM agents ca…
Decision MakingReinforcement Learning (RL)SensitivitySequential Decision MakingSkin-in-the-Game: Decision Making via Multi-Stakeholder Alignment in LLMs
Large Language Models (LLMs) have shown remarkable capabilities in tasks such as summarization, arithmetic reasoning, and question answering. However, they encounter significant challenges in the domain of moral reasonin…
Arithmetic ReasoningDecision MakingQuestion AnsweringBias in Decision-Making for AI's Ethical Dilemmas: A Comparative Study of ChatGPT and Claude
Recent advances in Large Language Models (LLMs) have enabled human-like responses across various tasks, raising questions about their ethical decision-making capabilities and potential biases. This study investigates pro…
AttributeDecision MakingEthicsSensitivityPredicting and Understanding Human Action Decisions: Insights from Large Language Models and Cognitive Instance-Based Learning
Large Language Models (LLMs) have demonstrated their capabilities across various tasks, from language translation to complex reasoning. Understanding and predicting human behavior and biases are crucial for artificial in…
Decision MakingSequential Decision Making