Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective
Foundation Models (FMs) have demonstrated remarkable insights into the relational dynamics of the world, leading to the crucial question: how do these models acquire an understanding of world hybrid relations? Traditional statistical learning, particularly for prediction problems, may overlook the rich and inherently structured information from the data, especially regarding the relationships between objects. We introduce a mathematical model that formalizes relational learning as hypergraph recovery to study pre-training of FMs. In our framework, the world is represented as a hypergraph, with data abstracted as random samples from hyperedges. We theoretically examine the feasibility of a Pre-Trained Model (PTM) to recover this hypergraph and analyze the data efficiency in a minimax near-optimal style. By integrating rich graph theories into the realm of PTMs, our mathematical framework offers powerful tools for an in-depth understanding of pre-training from a unique perspective and can be used under various scenarios. As an example, we extend the framework to entity alignment in multimodal learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Entity AlignmentRelational ReasoningSimilar Papers 제목 키워드 기반
Self-Supervised Dynamic Hypergraph Recommendation based on Hyper-Relational Knowledge Graph
Knowledge graphs (KGs) are commonly used as side information to enhance collaborative signals and improve recommendation quality. In the context of knowledge-aware recommendation (KGR), graph neural networks (GNNs) have …
Collaborative FilteringKnowledge-Aware RecommendationKnowledge GraphsSelf-Supervised LearningPlanted Hitting Set Recovery in Hypergraphs
In various application areas, networked data is collected by measuring interactions involving some specific set of core nodes. This results in a network dataset containing the core nodes along with a potentially much lar…
Hypergraph as Language
Large language models (LLMs) have recently shown strong potential in modeling relational structures. However, existing approaches remain fundamentally graph-centric: they focus on processing pairwise graph structures int…
Nonparametric Modeling of Higher-Order Interactions via Hypergraphons
We study statistical and algorithmic aspects of using hypergraphons, that are limits of large hypergraphs, for modeling higher-order interactions. Although hypergraphons are extremely powerful from a modeling perspective…
Achieving the Kesten-Stigum bound in the non-uniform hypergraph stochastic block model
We study the community detection problem in the non-uniform hypergraph stochastic block model (HSBM), where hyperedges of varying sizes coexist. This setting captures higher-order and multi-view interactions and raises a…
Spectral ReconstructionCommunity Detection