Cliqueformer: Model-Based Optimization with Structured Transformers
Large neural networks excel at prediction tasks, but their application to design problems, such as protein engineering or materials discovery, requires solving offline model-based optimization (MBO) problems. While predictive models may not directly translate to effective design, recent MBO algorithms incorporate reinforcement learning and generative modeling approaches. Meanwhile, theoretical work suggests that exploiting the target function's structure can enhance MBO performance. We present Cliqueformer, a transformer-based architecture that learns the black-box function's structure through functional graphical models (FGM), addressing distribution shift without relying on explicit conservative approaches. Across various domains, including chemical and genetic design tasks, Cliqueformer demonstrates superior performance compared to existing methods.
Code (1)
Tasks
modelSimilar Papers 제목 키워드 기반
Systolic Arrays and Structured Pruning Co-design for Efficient Transformers in Edge Systems
Efficient deployment of resource-intensive transformers on edge devices necessitates cross-stack optimization. We thus study the interrelation between structured pruning and systolic acceleration, matching the size of pr…
Machine TranslationQuantizationspeech-recognitionSpeech RecognitionFrom S4 to Mamba: A Comprehensive Survey on Structured State Space Models
Recent advancements in sequence modeling have led to the emergence of Structured State Space Models (SSMs) as an efficient alternative to Recurrent Neural Networks (RNNs) and Transformers, addressing challenges in long-r…
Computational EfficiencyMambaspeech-recognitionSpeech Recognition+3CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers
Transformers achieve strong accuracy but incur high compute and memory cost. Structured pruning reduces inference cost, but most methods rely on retraining or multi-stage optimization, which limits post-training deployme…
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training. Motivated by this observation, we introduce \emph{Depth-wis…
Image ClassificationOptimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions
Transformer architectures can solve unseen tasks based on input-output pairs in a given prompt due to in-context learning (ICL). Existing theoretical studies on ICL have mainly focused on linear regression tasks, often w…