BSA: Ball Sparse Attention for Large-scale Geometries

2025-06-14 · Catalin E. Brita, Hieu Nguyen, Lohithsai Yadala Chanchu, Domonkos Nagy, Maksim Zhdanov

Self-attention scales quadratically with input size, limiting its use for large-scale physical systems. Although sparse attention mechanisms provide a viable alternative, they are primarily designed for regular structures such as text or images, making them inapplicable for irregular geometries. In this work, we present Ball Sparse Attention (BSA), which adapts Native Sparse Attention (NSA) (Yuan et al., 2025) to unordered point sets by imposing regularity using the Ball Tree structure from the Erwin Transformer (Zhdanov et al., 2025). We modify NSA's components to work with ball-based neighborhoods, yielding a global receptive field at sub-quadratic cost. On an airflow pressure prediction task, we achieve accuracy comparable to Full Attention while significantly reducing the theoretical computational complexity. Our implementation is available at https://github.com/britacatalin/bsa.

📄 PDF Abstract BibTeX arXiv:2506.12541

Code (1)

britacatalin/bsa 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

MSPT: Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention

2025-12-01 · Pedro M. P. Curvo, Jan-Willem van de Meent, Maksim Zhdanov arxiv

A key scalability challenge in neural solvers for industrial-scale physics simulations is efficiently capturing both fine-grained local interactions and long-range global dependencies across millions of spatial elements.…

Sparse Array Design via Fractal Geometries

2020-01-05 · Regev Cohen, Yonina C. Eldar

Sparse sensor arrays have attracted considerable attention in various fields such as radar, array processing, ultrasound imaging and communications. In the context of correlation-based processing, such arrays enable to r…

Flow reconstruction in time-varying geometries using graph neural networks

2024-11-13 · Bogdan A. Danciu, Vito A. Pagone, Benjamin Böhm, Marius Schmidt 외

The paper presents a Graph Attention Convolutional Network (GACN) for flow reconstruction from very sparse data in time-varying geometries. The model incorporates a feature propagation algorithm as a preprocessing step t…

Graph Attention

Hybrid Unet-Transformer Model for Generating Stress and Strain Fields from Composite Geometrics

2026-06-30 · Shrey Patel arxiv

Accurate prediction of stress and strain fields in hierarchical composite microstructures is critical for physics-informed material design, yet conventional finite element method (FEM) simulations are computationally pro…

Transolver: A Fast Transformer Solver for PDEs on General Geometries

2024-02-04 · Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang 외

Transformers have empowered many milestones across various fields and have recently been applied to solve partial differential equations (PDEs). However, since PDEs are typically discretized into large-scale meshes with …