Graph Machine: Towards Better Pretraining via Edges
We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Can Graph Neural Networks Go "Online"? An Analysis of Pretraining and Inference
Large-scale graph data in real-world applications is often not static but dynamic, i. e., new nodes and edges appear over time. Current graph convolution approaches are promising, especially, when all the graph's nodes a…
Transformers as Graph-to-Graph Models
We argue that Transformers are essentially graph-to-graph models, with sequences just being a special case. Attention weights are functionally equivalent to graph edges. Our Graph-to-Graph Transformer architecture makes …
Graph Context Encoder: Graph Feature Inpainting for Graph Generation and Self-supervised Pretraining
We propose the Graph Context Encoder (GCE), a simple but efficient approach for graph representation learning based on graph feature masking and reconstruction. GCE models are trained to efficiently reconstruct input gra…
Graph GenerationGraph Representation LearningRepresentation LearningDo Larger Language Models Imply Better Reasoning? A Pretraining Scaling Law for Reasoning
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks requiring complex reasoning. However, the effects of scaling on their reasoning abilities remain insufficiently understo…
Knowledge GraphsMemorizationAre All Edges Necessary? A Unified Framework for Graph Purification
Graph Neural Networks (GNNs) as deep learning models working on graph-structure data have achieved advanced performance in many works. However, it has been proved repeatedly that, not all edges in a graph are necessary f…
All