How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective

2025-04-10 · Qi Liu, Jiaxin Mao, Ji-Rong Wen

Recent studies have shown that large language models (LLMs) can assess relevance and support information retrieval (IR) tasks such as document ranking and relevance judgment generation. However, the internal mechanisms by which off-the-shelf LLMs understand and operationalize relevance remain largely unexplored. In this paper, we systematically investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability. Using activation patching techniques, we analyze the roles of various model components and identify a multi-stage, progressive process in generating either pointwise or pairwise relevance judgment. Specifically, LLMs first extract query and document information in the early layers, then process relevance information according to instructions in the middle layers, and finally utilize specific attention heads in the later layers to generate relevance judgments in the required format. Our findings provide insights into the mechanisms underlying relevance assessment in LLMs, offering valuable implications for future research on leveraging LLMs for IR tasks.

📄 PDF Abstract BibTeX arXiv:2504.07898

Code (1)

liuqi6777/llm-relevance 공식 구현 jax

Tasks

Document Ranking Information Retrieval

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Patching Activation patching studies the model's computation by altering its latent representations, the token embeddings in transformer-based language models, during the inference process

Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

2023-08-27 · Vedant Palit, Rohan Pandey, Aryaman Arora, Paul Pu Liang

Mechanistic interpretability seeks to understand the neural mechanisms that enable specific behaviors in Large Language Models (LLMs) by leveraging causality-based methods. While these approaches have identified neural c…

Question Answering Text Generation Visual Question Answering

Mechanistic Interpretability for AI Safety -- A Review

2024-04-22 · Leonard Bereska, Efstratios Gavves

Understanding AI systems' inner workings is critical for ensuring value alignment and safety. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and representations learne…

Axiomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models

2024-05-03 · Catherine Chen, Jack Merullo, Carsten Eickhoff

Neural models have demonstrated remarkable performance across diverse ranking tasks. However, the processes and internal mechanisms along which they determine relevance are still largely unknown. Existing approaches for …

Decision Making Retrieval

Beyond the Black Box: Interpretability of LLMs in Finance

2025-05-14 · Hariom Tatsat, Ariye Shater

Large Language Models (LLMs) exhibit remarkable capabilities across a spectrum of tasks in financial services, including report generation, chatbots, sentiment analysis, regulatory compliance, investment advisory, financ…

Fairness Hallucination Sentiment Analysis

Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity

2026-05-13 · Cristian Hinostroza, Rodrigo Toro Icarte, Christ Devia, Andres Carvallo De Ferari 외 arxiv

Large language models (LLMs) have revolutionized natural language processing. Understanding their internal mechanisms is crucial for developing more interpretable and optimized architectures. Mechanistic interpretability…