LACE: Loss-Adaptive Capacity Expansion for Continual Learning

2026-03-30 · Shivnath Tathe arxiv

Fixed representational capacity is a fundamental constraint in continual learning: practitioners must guess an appropriate model width before training, without knowing how many distinct concepts the data contains. We propose LACE (Loss-Adaptive Capacity Expansion), a simple online mechanism that expands a model's representational capacity during training by monitoring its own loss signal. When sustained loss deviation exceeds a threshold - indicating that the current capacity is insufficient for newly encountered data - LACE adds new dimensions to the projection layer and trains them jointly with existing parameters. Across synthetic and real-data experiments, LACE triggers expansions exclusively at domain boundaries (100% boundary precision, zero false positives), matches the accuracy of a large fixed-capacity model while starting from a fraction of its dimensions, and produces adapter dimensions that are collectively critical to performance (3% accuracy drop when all adapters removed). We further demonstrate unsupervised domain separation in GPT-2 activations via layer-wise clustering, showing a U-shaped separability curve across layers that motivates adaptive capacity allocation in deep networks. LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.

📄 PDF Abstract BibTeX arXiv:2603.28611

Code (0)

등록된 구현이 없습니다.

Tasks

Continual Learning

ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning

2025-10-11 · Jinyang Zhang, Yue Fang, Hongxin Ding, Weibin Liao 외 arxiv

Conventional continual pretraining (CPT) for large language model (LLM) domain adaptation often suffers from catastrophic forgetting and limited domain capacity. Existing strategies adopt layer expansion, introducing add…

Continual Pretraining General Knowledge Domain Adaptation

Dynamic Mixture of Experts Against Severe Distribution Shifts

2025-11-24 · Donghu Kim arxiv

The challenge of building neural networks that can continuously learn and adapt to evolving data streams is central to the fields of continual learning (CL) and reinforcement learning (RL). This lifelong learning problem…

Reinforcement Learning Continual Learning

SpaceNet: Make Free Space For Continual Learning

2020-07-15 · Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy

The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catastrophic forgetting previously learned ta…

class-incremental learning Class Incremental Learning Continual Learning Incremental Learning+1

ATSplat: Compact Feed-forward 3D Gaussian Splatting with Adaptive Token Expansion

2026-07-22 · Cho In, Jeonghwan Cho, Mijin Yoo, Gim Hee Lee 외 hf

3D Gaussian Splatting (3DGS) achieves high-quality novel-view synthesis by optimizing freely placed primitives in 3D and adaptively densifying them in under-reconstructed regions. However, this scene-adaptive capacity al…

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning

2026-05-10 · Xing Han, Shravan Chaudhari, Tanvi Ranade, Rama Chellappa 외 arxiv

Real-world model deployment across multiple domains requires multimodal models to operate under two complementary regimes: (1) multi-task pretraining, tasks are co-available at design time where related tasks could borro…

Multi-Task Learning Continual Learning