DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
We introduce a novel index, the Distribution of Cosine Similarity (DOCS), for quantitatively assessing the similarity between weight matrices in Large Language Models (LLMs), aiming to facilitate the analysis of their complex architectures. Leveraging DOCS, our analysis uncovers intriguing patterns in the latest open-source LLMs: adjacent layers frequently exhibit high weight similarity and tend to form clusters, suggesting depth-wise functional specialization. Additionally, we prove that DOCS is theoretically effective in quantifying similarity for orthogonal matrices, a crucial aspect given the prevalence of orthogonal initializations in LLMs. This research contributes to a deeper understanding of LLM architecture and behavior, offering tools with potential implications for developing more efficient and interpretable models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DocScanner: Robust Document Image Rectification with Progressive Learning
Compared with flatbed scanners, portable smartphones provide more convenience for physical document digitization. However, such digitized documents are often distorted due to uncontrolled physical deformations, camera po…
Optical Character Recognition (OCR)Evaluating Self-Generated Documents for Enhancing Retrieval-Augmented Generation with Large Language Models
The integration of documents generated by LLMs themselves (Self-Docs) alongside retrieved documents has emerged as a promising strategy for retrieval-augmented generation systems. However, previous research primarily foc…
Language ModellingLarge Language ModelQuestion AnsweringRAG+2Contrastive Similarity Learning for Market Forecasting: The ContraSim Framework
We introduce the Contrastive Similarity Space Embedding Algorithm (ContraSim), a novel framework for uncovering the global semantic relationships between daily financial headlines and market movements. ContraSim operates…
Contrastive LearningA cohomology-based Gromov-Hausdorff metric approach for quantifying molecular similarity
We introduce, for the first time, a cohomology-based Gromov-Hausdorff ultrametric method to analyze 1-dimensional and higher-dimensional (co)homology groups, focusing on loops, voids, and higher-dimensional cavity struct…
ClusteringTowards Quantifying the Distance between Opinions
Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easi…
Navigatetext similarity