Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks

2024-07-24 · Jingze Shi, Bingheng Wu, Lu He, Luchang Jiang

We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which can improve the expressiveness of the attention algorithm and avoid the sequence noise significantly affecting the accuracy of the attention score. We also design cross domain mixture of experts, which can improve the granularity of the sparse activation feedforward network while maintaining the efficiency of parameter utilization and retrieval. The combination of these methods constitutes our foundation model architecture: Wonderful Matrices. We conduct experiments on the language modeling task and find that Wonderful Matrices are more efficient and effective in handling complex language tasks.

📄 PDF Abstract BibTeX arXiv:2407.16958

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingMixture-of-ExpertsPositionRetrieval

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture

2024-12-16 · Jingze Shi, Bingheng Wu

In order to make the foundation model more efficient and effective, our idea is combining sequence transformation and state transformation. First, we prove the availability of rotary position embedding in the state space…

Mixture-of-Experts Position Retrieval

A DNN Framework For Text Image Rectification From Planar Transformations

2016-11-14 · Chengzhe Yan, Jie Hu, Chang-Shui Zhang

In this paper, a novel neural network architecture is proposed attempting to rectify text images with mild assumptions. A new dataset of text images is collected to verify our model and open to public. We explored the ca…

Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs

2024-07-26 · Zidan Wang, Rui Shen, Bradly Stadie

We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework for executing high-level robotic planning in a zero-shot regime. In our context, zero-shot high-level planning means that for a nove…

Action Generation Large Language Model Task Planning Visual Reasoning

Tangent classes of matroids and wonderful compactifications

2026-07-07 · Ronnie Cheng, Shurui Liu, Guoxiong Gao arxiv

For every loopless matroid $M$ and every Feichtner--Yuzvinsky building set $\mathcal{G}$ containing the top flat, we construct an integral tangent class $T_{M,\mathcal{G}}^{\mathbb{Z}}\in K_{\mathbb{Z}}(M,\mathcal{G})$; …

Mathematical Reasoning

DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models

2025-01-28 · Zeping Min, Xinshang Wang

We introduce a novel index, the Distribution of Cosine Similarity (DOCS), for quantitatively assessing the similarity between weight matrices in Large Language Models (LLMs), aiming to facilitate the analysis of their co…