홈 › Papers

Non-Determinism of "Deterministic" LLM Settings

2024-08-06 · Berk Atıl, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, Breck Baldwin

LLM (large language model) practitioners commonly notice that outputs can vary for the same inputs under settings expected to be deterministic. Yet the questions of how pervasive this is, and with what impact on results, have not to our knowledge been systematically investigated. We investigate non-determinism in five LLMs configured to be deterministic when applied to eight common tasks in across 10 runs, in both zero-shot and few-shot settings. We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%. In fact, none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings. Sharing preliminary results with insiders has revealed that non-determinism perhaps essential to the efficient use of compute resources via co-mingled data in input buffers so this issue is not going away anytime soon. To better quantify our observations, we introduce metrics focused on quantifying determinism, TARr@N for the total agreement rate at N runs over raw output, and TARa@N for total agreement rate of parsed-out answers. Our code and data are publicly available at https://github.com/breckbaldwin/llm-stability.

📄 PDF Abstract BibTeX arXiv:2408.04667

Code (2)

Comcast/llm-stability 공식 구현

breckbaldwin/llm-stability 공식 구현

Tasks

Language ModelingLanguage ModellingLarge Language Model

Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

2026-01-17 · Raffi Khatchadourian arxiv

LLM agents struggle with regulatory audit replay: when asked to reproduce a flagged transaction decision with identical inputs, many deployments fail to return consistent results. We introduce the Determinism-Faithfulnes…

Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

2018-09-15 · Prabhat Nagarajan, Garrett Warnell, Peter Stone

While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminis…

Deep Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models

2026-04-15 · Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiaoge Zhang 외 arxiv

Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remains poorly understood. The existing non-determinism evaluations for L…

Question Answering Code Generation

Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism

2026-01-03 · Tairan Fu, Gonzalo Martínez, Javier Conde, Carlos Arriaga 외 arxiv

The execution of Large Language Models (LLMs) has been shown to produce nondeterministic results when run on Graphics Processing Units (GPUs), even when they are configured to produce deterministic results. This is due t…

Explorability in Pushdown Automata

2025-11-06 · Ayaan Bedi, Karoliina Lehtinen arxiv

We study explorability, a measure of nondeterminism in pushdown automata, which generalises history-determinism. An automaton is k-explorable if, while reading the input, it suffices to follow k concurrent runs, built st…