Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software
Are AI agents tools, co-authors, or researchers? We present a quantified case study ($N=1$): a physicist supervising an AI coding agent (Claude Code, Sonnet and Opus models) over 12 work days and 57 sessions to build CLAX-PT, a differentiable one-loop perturbation theory module in JAX. We documented and classified 15 supervision events by intervention level. The agent resolved ten autonomously by iterating against oracle tests. Two more by the physicist's domain knowledge. The three it could not -- all evaded oracle detection -- share a common property: the agent treated symptom reduction as root-cause resolution. It spent 33 of the 57 sessions adjusting coefficients within a code architecture that could not represent the target physics, and could not re-evaluate its CLASS-PT branch choice even when prompted to reconsider; only an injected physics concept (anisotropic BAO damping) triggered the redesign. Separately, the agent committed a calibrated correction that passed all oracle tests but corresponded to no quantity in the theory, predicting wrong values at any other cosmology. The fudge factor was caught and replaced within the same session. Three supervision practices proved critical for catching what oracle tests missed: testing at diverse parameter points beyond the fiducial calibration; shared changelogs that surfaced stalled exploration across sessions; and an explicit rule against unphysical numerical patches. In this case, supervision design, not model capability, determined whether the agent's output was trustworthy. Closing the gap would require agents that propose architectural alternatives rather than optimize within a given structure, and distinguish predictive adequacy from explanatory correctness -- capabilities not exhibited here, not obviously addressed by scaling alone. [Abridged.]
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Antibiotic resistance: a physicist's view
The problem of antibiotic resistance poses challenges across many disciplines. One such challenge is to understand the fundamental science of how antibiotics work, and how resistance to them can emerge. This is an area w…
A high-bias, low-variance introduction to Machine Learning for physicists
Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a man…
BIG-bench Machine LearningClusteringData VisualizationVocal Bursts Intensity PredictionWhat is AI, what is it not, how we use it in physics and how it impacts... you
Artificial Intelligence (AI) and Machine Learning (ML) have been prevalent in particle physics for over three decades, shaping many aspects of High Energy Physics (HEP) analyses. As AI's influence grows, it is essential …
Anomaly DetectionMisconceptionsGenerative Models for Fast Calorimeter Simulation.LHCb case
Simulation is one of the key components in high energy physics. Historically it relies on the Monte Carlo methods which require a tremendous amount of computation resources. These methods may have difficulties with the e…
Why is AI hard and Physics simple?
We discuss why AI is hard and why physics is simple. We discuss how physical intuition and the approach of theoretical physics can be brought to bear on the field of artificial intelligence and specifically machine learn…
BIG-bench Machine LearningLearning TheoryPhysical Intuition