A Granger-Causal Perspective on Gradient Descent with Application to Pruning
Stochastic Gradient Descent (SGD) is the main approach to optimizing neural networks. Several generalization properties of deep networks, such as convergence to a flatter minima, are believed to arise from SGD. This article explores the causality aspect of gradient descent. Specifically, we show that the gradient descent procedure has an implicit granger-causal relationship between the reduction in loss and a change in parameters. By suitable modifications, we make this causal relationship explicit. A causal approach to gradient descent has many significant applications which allow greater control. In this article, we illustrate the significance of the causal approach using the application of Pruning. The causal approach to pruning has several interesting properties - (i) We observe a phase shift as the percentage of pruned parameters increase. Such phase shift is indicative of an optimal pruning strategy. (ii) After pruning, we see that minima becomes "flatter", explaining the increase in accuracy after pruning weights.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GCAD: Anomaly Detection in Multivariate Time Series from the Perspective of Granger Causality
Multivariate time series anomaly detection has numerous real-world applications and is being extensively studied. Modeling pairwise correlations between variables is crucial. Existing methods employ learnable graph struc…
Anomaly DetectionTime SeriesTime Series Anomaly DetectionGraphs in State-Space Models for Granger Causality in Climate Science
Granger causality (GC) is often considered not an actual form of causality. Still, it is arguably the most widely used method to assess the predictability of a time series from another one. Granger causality has been wid…
EconometricsState Space ModelsTime SeriesA Gradient-based Causal Discovery Framework with Applications to Complex Industrial Processes
With the advancement of deep learning technologies, various neural network-based Granger causality models have been proposed. Although these models have demonstrated notable improvements, several limitations remain. Most…
Time Series ForecastingTime Series PredictionGranger Causality Detection with Kolmogorov-Arnold Networks
Discovering causal relationships in time series data is central in many scientific areas, ranging from economics to climate science. Granger causality is a powerful tool for causality detection. However, its original for…
Kolmogorov-Arnold NetworksVariable-lag Granger Causality for Time Series Analysis
Granger causality is a fundamental technique for causal inference in time series data, commonly used in the social and biological sciences. Typical operationalizations of Granger causality make a strong assumption that e…
Causal InferenceLeadership InferenceTime SeriesTime Series Analysis