Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignment
Unlike primates, training artificial neural networks on changing data distributions leads to a rapid decrease in performance on old tasks. This phenomenon is commonly referred to as catastrophic forgetting. In this paper, we investigate the representational changes that underlie this performance decrease and identify three distinct processes that together account for the phenomenon. The largest component is a misalignment between hidden representations and readout layers. Misalignment occurs due to learning on additional tasks and causes internal representations to shift. Representational geometry is partially conserved under this misalignment and only a small part of the information is irrecoverably lost. All types of representational changes scale with the dimensionality of hidden representations. These insights have implications for deep learning applications that need to be continuously updated, but may also aid aligning ANN models to the rather robust biological vision.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningSimilar Papers 제목 키워드 기반
Valuation of contingent convertible catastrophe bonds - the case for equity conversion
Within the context of the banking-related literature on contingent convertible bonds, we comprehensively formalise the design and features of a relatively new type of insurance-linked security, called a contingent conver…
LSTM-Based Detection of Structural Breaks in Property Insurance Loss Reserving: A Climate-Informed Approach
Accurate loss reserving is foundational to insurer solvency, yet accelerating climate driven catastrophes systematically violate the stability assumptions on which traditional actuarial methods depend. This white paper p…
A Unified Bayesian Framework for Pricing Catastrophe Bond Derivatives
Catastrophe (CAT) bond markets are incomplete and hence carry uncertainty in instrument pricing. As such various pricing approaches have been proposed, but none treat the uncertainty in catastrophe occurrences and intere…
ClusteringUncertainty QuantificationOvercoming Forgetting Catastrophe in Quantization-Aware Training
Quantization is an effective approach for memory cost reduction by compressing networks to lower bits. However, existing quantization processes learned only from the current data tend to suffer from forgetting catast…
Lifelong learningQuantizationArbitrage-free catastrophe reinsurance valuation for compound dynamic contagion claims
In this paper, we consider catastrophe stop-loss reinsurance valuation for a reinsurance company with dynamic contagion claims. To deal with conventional and emerging catastrophic events, we propose the use of a compound…