Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns
Stateful multimodal assistants encode an image once but may answer questions about it many turns later. Attention-guided visual-KV eviction assumes that evidence irrelevant now will remain dispensable, although future questions are unknown. We ask when a visual fact is actually safe to forget and introduce the Causal Visual Memory Audit (CVMA), a paired single-prefill framework that tests what later answers lose when a visual region, the whole image, or prior assistant text becomes unavailable. On VisDial and ConvBench, current attention can rank future-useful regions worse than random even though a diagnostic marginal-utility control shows substantial selection headroom. Aggregate scores hide this failure when later turns do not need vision; controlled and stock-generated histories reveal a second escape route, in which assistant-text KV replaces image KV for facts already stated but not reliably for unstated facts. In the tested stacks, safe forgetting is supported by low future visual dependence or fact-specific verbalization---not by low current attention.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
SMI: Statistical Membership Inference for Reliable Unlearned Model Auditing
Machine unlearning (MU) is essential for enforcing the right to be forgotten in machine learning systems. A key challenge of MU is how to reliably audit whether a model has truly forgotten specified training data. Member…
The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference a…
Noisy Agents: Self-supervised Exploration by Predicting Auditory Events
Humans integrate multiple sensory modalities (e.g. visual and audio) to build a causal understanding of the physical world. In this work, we propose a novel type of intrinsic motivation for Reinforcement Learning (RL) th…
Atari GamesReinforcement Learning (RL)Auditing Language Model Unlearning via Information Decomposition
We expose a critical limitation in current approaches to machine unlearning in language models: despite the apparent success of unlearning algorithms, information about the forgotten data remains linearly decodable from …
An Explainable AI Approach to Large Language Model Assisted Causal Model Auditing and Development
Causal networks are widely used in many fields, including epidemiology, social science, medicine, and engineering, to model the complex relationships between variables. While it can be convenient to algorithmically infer…
Causal InferenceEpidemiologyLanguage ModelingLanguage Modelling+2