WMF-AM: Probing LLM Working Memory via Depth-Parameterized Cumulative State Tracking
Existing large language models (LLMs) evaluations use fixed-difficulty benchmarks that cannot adapt as models improve, and rarely isolate specific cognitive processes. We introduce Working Memory Fidelity-Active Manipulation (WMF-AM), a probe of cumulative state tracking, the ability to maintain and update intermediate results across K sequential operations within a single query, without a scratchpad. Unlike multi-step agent benchmarks that stress task orchestration, WMF-AM isolates within-pass cumulative load by parameterizing depth K. The core probe uses arithmetic accumulation on 28 models from 12 families (0.5B to frontier); a matched non-arithmetic extension (permissions, schedules, inventories) confirms the design generalizes beyond arithmetic. Three construct-isolation ablations confirm that cumulative load, not arithmetic skill or entity tracking, drives difficulty. We release WMF-AM as a lightweight, recalibratable diagnostic for characterizing where models degrade under cumulative load. Code and data can be accessed at https://github.com/dengzhe-hou/WMF-AM
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation
Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its no…
Learning to Remember, Forget and Ignore using Attention Control in Memory
Typical neural networks with external memory do not effectively separate capacity for episodic and working memory as is required for reasoning in humans. Applying knowledge gained from psychological studies, we designed …
Are they human? Detecting large language models by probing human memory constraints
The validity of online behavioral research relies on study participants being human rather than machine. In the past, it was possible to detect machines by posing simple challenges that were easily solved by humans but n…
(Frisch-Waugh-Lovell)': On the Estimation of Regression Models by Row
We demonstrate that regression models can be estimated by working independently in a row-wise fashion. We document a simple procedure which allows for a wide class of econometric estimators to be implemented cumulatively…
regressionMeSH: Memory-as-State-Highways for Recursive Transformers
Recursive transformers reuse parameters and iterate over hidden states multiple times, decoupling compute depth from parameter depth. However, under matched compute, recursive models with fewer parameters often lag behin…