Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers
Formal languages have proven to be effective conduits to understand the inner mechanisms of transformers. Past work has shown that transformers trained on next token prediction over counter languages learn representations consistent with an underlying stack structure. Beyond representational analysis, this paper investigates the causal role of these representations. Linear probes are trained to predict the stack depth at each token from the model's hidden states, and a principal representation direction is extracted from the probe. Ablation of this direction from the model causes sequential accuracy to collapse to near 0%, providing strong empirical evidence that the stack representation is not just learned, but is causally necessary for model performance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Towards a Causal Probabilistic Framework for Prediction, Action-Selection & Explanations for Robot Block-Stacking Tasks
Uncertainties in the real world mean that is impossible for system designers to anticipate and explicitly design for all scenarios that a robot might encounter. Thus, robots designed like this are fragile and fail outsid…
Causal InferencecounterfactualCounterfactual Reasoning for Fine-Grained Evidence Disentanglement in VideoQA
Recent advances in video multimodal models have significantly improved VideoQA performance. However, these systems often rely on spurious statistical correlations rather than answer-relevant causal evidence, resulting in…
Causal-Counterfactual RAG: The Integration of Causal-Counterfactual Reasoning into RAG
Large language models (LLMs) have transformed natural language processing (NLP), enabling diverse applications by integrating large-scale pre-trained knowledge. However, their static knowledge limits dynamic reasoning ov…
Semantic SimilarityCaFlow: Enhancing Long-Term Action Quality Assessment with Causal Counterfactual Flow
Action Quality Assessment (AQA) predicts fine-grained execution scores from action videos and is widely applied in sports, rehabilitation, and skill evaluation. Long-term AQA, as in figure skating or rhythmic gymnastics,…
Action Quality AssessmentExploring Logically Dependent Multi-task Learning with Causal Inference
Previous studies have shown that hierarchical multi-task learning (MTL) can utilize task dependencies by stacking encoders and outperform democratic MTL. However, stacking encoders only considers the dependencies of feat…
Causal InferencecounterfactualCounterfactual ReasoningMulti-Task Learning