Leveraging Entanglement Entropy for Deep Understanding of Attention Matrix in Text Matching
The formal understanding of deep learning has made great progress based on quantum many-body physics. For example, the entanglement entropy in quantum many-body systems can interpret the inductive bias of neural network and then guide the design of network structure and parameters for certain tasks. However, there are two unsolved problems in the current study of entanglement entropy, which limits its application potential. First, the theoretical benefits of entanglement entropy was only investigated in the representation of a single object (e.g., an image or a sentence), but has not been well studied in the matching of two objects (e.g., question-answering pairs). Second, the entanglement entropy can not be qualitatively calculated since the exponentially increasing dimension of the matching matrix. In this paper, we are trying to address these two problem by investigating the fundamental connections between the entanglement entropy and the attention matrix. We prove that by a mapping (via the trace operator) on the high-dimensional matching matrix, a low-dimensional attention matrix can be derived. Based on such a attention matrix, we can provide a feasible solution to the entanglement entropy that describes the correlation between the two objects in matching tasks. Inspired by the theoretical property of the entanglement entropy, we can design the network architecture adaptively in a typical text matching task, i.e., question-answering task.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive BiasQuestion AnsweringSentenceText MatchingSimilar Papers 제목 키워드 기반
Artificial Entanglement in the Fine-Tuning of Large Language Models
Large language models (LLMs) can be adapted to new tasks using parameter-efficient fine-tuning (PEFT) methods that modify only a small number of trainable parameters, often through low-rank updates. In this work, we adop…
parameter-efficient fine-tuningReinforcement Learning for Optimizing Large Qubit Array based Quantum Sensor Circuits
As the number of qubits in a sensor increases, the complexity of designing and controlling the quantum circuits grows exponentially. Manually optimizing these circuits becomes infeasible. Optimizing entanglement distribu…
Quantum Machine LearningReinforcement LearningQuantum Machine Learning for Optimizing Entanglement Distribution in Quantum Sensor Circuits
In the rapidly evolving field of quantum computing, optimizing quantum circuits for specific tasks is crucial for enhancing performance and efficiency. More recently, quantum sensing has become a distinct and rapidly gro…
Quantum Machine LearningReinforcement LearningGradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions
Understanding the intricate non-convex training dynamics of softmax-based models is crucial for explaining the empirical success of transformers. In this article, we analyze the gradient flow dynamics of the value-softma…
Generative Modeling of Quantum Distribution with Functional Flow Matching
The emergence of powerful deep generative models based on diffusion and flow matching has enabled the learning and modeling of complex distributions. Learning quantum distributions, however, remains challenging due to th…