Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models
We investigate the geometry of decision-making in Multiple Choice Question Answering (MCQA) through the lens of isotropy. Analyzing five open-weight models across diverse datasets, we identify decision-critical transition layers characterized by a shift in isotropy, coinciding with a major representational change and the emergence of task-relevant clusters. We demonstrate that this synchronized geometric behavior is strongly correlated with downstream accuracy ($r\approx0.84$), displaying its relevance for successful decision-making. Furthermore, we show that this transition is robust to prompt variations, suggesting that it reflects a general mechanism of model behavior.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSimilar Papers 제목 키워드 기반
Anisotropic Random Feature Regression in High Dimensions
In contrast to standard statistical wisdom, modern learning algorithms typically find their best performance in the overparameterized regime in which the model has many more parameters than needed to fit the training dat…
regressionVocal Bursts Intensity PredictionFinding signatures of low-dimensional geometric landscapes in high-dimensional cell fate transitions
Multicellular organisms develop a wide variety of highly-specialized cell types. The consistency and robustness of developmental cell fate trajectories suggests that complex gene regulatory networks effectively act as lo…
Decision MakingHow Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy
It is widely accepted that fine-tuning pre-trained language models usually brings about performance improvements in downstream tasks. However, there are limited studies on the reasons behind this effectiveness, particula…
Detecting LLM Hallucinations via Embedding Cluster Geometry: A Three-Type Taxonomy with Measurable Signatures
We propose a geometric taxonomy of large language model hallucinations based on observable signatures in token embedding cluster structure. By analyzing the static embedding spaces of 11 transformer models spanning encod…
Signature-Informed Transformer for Asset Allocation
Modern deep learning for asset allocation typically separates forecasting from optimization. We argue this creates a fundamental mismatch where minimizing prediction errors fails to yield robust portfolios. We propose th…
Decision Making