Supernova: Achieving More with Less in Transformer Architectures
We present Supernova, a 650M-parameter decoder-only transformer that demonstrates how careful architectural design and tokenization innovation can achieve the performance of larger models while maintaining computational efficiency. Our architecture combines Rotary Positional Embeddings (RoPE), Grouped Query Attention (GQA) with a 3:1 compression ratio, RMSNorm for computational efficiency, and SwiGLU activation functions. A critical innovation is our custom 128,000-vocabulary byte-level BPE tokenizer, which achieves state-of-the-art compression performance. Through detailed analysis, we show that Supernova achieves 90% of the performance of 1B-parameter models while using 35% fewer parameters and requiring only 100B training tokens--an order of magnitude less than competing models. Our findings challenge the prevailing scaling paradigm, demonstrating that architectural efficiency and tokenization quality can compensate for reduced parameter counts.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencySimilar Papers 제목 키워드 기반
A COMPASS to Model Comparison and Simulation-Based Inference in Galactic Chemical Evolution
We present COMPASS, a novel simulation-based inference framework that combines score-based diffusion models with transformer architectures to jointly perform parameter estimation and Bayesian model comparison across comp…
Bayesian InferenceProbabilistic Dalek -- Emulator framework with probabilistic prediction for supernova tomography
Supernova spectral time series can be used to reconstruct a spatially resolved explosion model known as supernova tomography. In addition to an observed spectral time series, a supernova tomography requires a radiative t…
Active LearningCPUTime SeriesTime Series Analysis+1Identifying Transients in the Dark Energy Survey using Convolutional Neural Networks
The ability to discover new transients via image differencing without direct human intervention is an important task in observational astronomy. For these kind of image classification problems, machine Learning technique…
Astronomyimage-classificationImage ClassificationAnomaly Detection for Multivariate Time Series of Exotic Supernovae
Supernovae mark the explosive deaths of stars and enrich the cosmos with heavy elements. Future telescopes will discover thousands of new supernovae nightly, creating a need to flag astrophysically interesting events rap…
Anomaly DetectionTime SeriesTime Series AnalysisPhysics Supernova: AI Agent Matches Elite Gold Medalists at IPhO 2025
Physics provides fundamental laws that describe and predict the natural world. AI systems aspiring toward more general, real-world intelligence must therefore demonstrate strong physics problem-solving abilities: to form…