Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost
Coding-agent efficiency cannot be characterized by token count or model price alone. We study how end-to-end cost and task success depend jointly on prompt semantics, inference effort, harness policy, model, task difficulty, tool use, context management, and provider accounting. Controlled prompt experiments show that wording can change reasoning and verification behavior without changing the task. A separate SWE-bench Verified study shows that additional inference effort can improve difficult tasks for some models but can also add cost without benefit. A DeepSeek Harness extension shows that the effect of an effort-control intervention changes substantially when the harness changes, even when the model, tasks, prompts, and controller logic are held fixed. These results show that prompt, effort, and harness are interacting experimental factors rather than independent efficiency controls. We model efficiency as cost per successful task induced by the agent trajectory. Token and cache counts are measurements of that trajectory, not sufficient optimization targets. Agent evaluations should therefore measure success and end-to-end cost while controlling the system variables that determine how the trajectory is produced
Code (0)
등록된 구현이 없습니다.
Tasks
Prompt EngineeringSimilar Papers 제목 키워드 기반
"Well, Keep Thinking": Enhancing LLM Reasoning with Adaptive Injection Decoding
Large language models (LLMs) exhibit strong reasoning abilities, often attributed to few-shot or zero-shot chain-of-thought (CoT) prompting. While effective, these methods require labor-intensive prompt engineering, rais…
Prompt EngineeringAgent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Models
Large Language Models (LLMs) have demonstrated strong capabilities in complex reasoning tasks, while recent prompting strategies such as Chain-of-Thought (CoT) have further elevated their performance in handling complex …
Logit-Entropy Adaptive Stopping Heuristic for Efficient Chain-of-Thought Reasoning
Chain-of-Thought (CoT) prompting is a key technique for enabling complex reasoning in large language models. However, generating full, fixed-length rationales is computationally wasteful, inflating both token usage and l…
Chain-of-Thought Augmentation with Logit Contrast for Enhanced Reasoning in Language Models
Rapidly increasing model scales coupled with steering methods such as chain-of-thought prompting have led to drastic improvements in language model reasoning. At the same time, models struggle with compositional generali…
Language ModelingLanguage ModellingOLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents
Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks. In deployed settings where agents repeatedly handle related multi-step tasks, small action-se…
Decision Making