Papers Stochastic Optimization
“Stochastic Optimization” 태그가 달린 논문 1,567편 · 필터 해제
Stochastic Gradient Descent over P2
Stochastic gradient descent (SGD) admits diffusion approximations that replace the complicated randomness of stochastic gradients by Gaussian noise, providing a powerful tool for understanding its dynamics and long-time …
Stochastic OptimizationConvergence rates for the RMSprop optimizer with full control of the hyperparameters
Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivity parts in Adam and AdamW basically jus…
Stochastic OptimizationBeyond Optimal Rates in Stochastic Optimization: Trajectory-Adaptive Stopping Rules
Stochastic gradient descent (SGD) is typically analyzed at a deterministic horizon chosen before the algorithm is run, even though practical stopping decisions are made adaptively by inspecting the evolving trajectory. T…
Stochastic OptimizationStochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework
Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation. This condition gives a direct way to rule o…
Stochastic OptimizationQuantum Speedups for Stochastic Optimization with Heavy-Tailed Noise
We study stochastic optimization with heavy-tailed gradient noise. We first propose a novel quantum mean estimator for multivariate heavy-tailed random variables that achieves lower query complexity than optimal classica…
Stochastic OptimizationEnhanced Neural Quantum State via Annealed Gradient Descent
Neural quantum states offer expressive representations of quantum many-body wave functions, yet their practical accuracy can be limited by stochastic optimization rather than representational capacity. Here we identify a…
Stochastic OptimizationTamed Stochastic Gradient Hamiltonian Monte Carlo
In this paper, we propose a novel tamed stochastic gradient Hamiltonian Monte Carlo (tSGHMC) algorithm for sampling and stochastic optimization problems with superlinearly growing stochastic gradients. Under a certain co…
Stochastic OptimizationInterleaved Noise Injection Improves Clean, Corrupted, and OOD Performance
Noise injection is a well-known technique in stochastic optimization. We report its surprising effectiveness with an interleaved (on-off-on-off...) rather than the usual monotonic decay schedule. We present a theoretical…
Stochastic OptimizationHow much Data do We Need? Sequential Data Collection for Stochastic Programming
Data-driven optimization often requires collecting data to estimate uncertain model parameters before solving the underlying decision problem. In practice, however, data acquisition may incur non-negligible costs, making…
Stochastic OptimizationIs Variational Monte Carlo Robust? Sharp Moment Thresholds and Heavy-tailed Stochastic Optimization
Variational Monte Carlo (VMC) is a central algorithm in electronic structure theory and has gained renewed importance through modern neural-network ansätze such as FermiNet. At its core, VMC seeks ground states by minimi…
Stochastic OptimizationMGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
Efficient optimization is essential for training large language models. Although intra-layer selective updates have been explored, a general mechanism that enables fine-grained control while ensuring convergence guarante…
Stochastic OptimizationDecision-Weighted Flow Matching for Contextual Stochastic Optimization
Conditional generative models are increasingly used as scenario generators for stochastic optimization, but standard training objectives emphasize uniform distributional fit rather than the downstream decisions induced b…
Stochastic OptimizationAdaptive Weighted Averaging
We study the problem of selecting the largest among $n$ unknown values $x_1,\dots,x_n$ given only a single unbiased estimate $y_i$ for each $x_i$. We design strategies that are simultaneously admissible (not uniformly do…
Stochastic OptimizationLearning What to Remember: Observability-Safe Memory Retention via Constrained Optimization for Long-Horizon Language Agents
Long-horizon language agents accumulate observations, reasoning traces, and retrieved facts exceeding context windows, making memory retention a fundamental resource-allocation problem. Existing systems treat retention a…
Stochastic OptimizationTheoretical Foundations of Continual Learning via Drift-Plus-Penalty
In many real-world settings, data streams are nonstationary and arrive sequentially, requiring learning systems to adapt continuously without retraining from scratch. Continual learning (CL) addresses this challenge by i…
Stochastic OptimizationContinual LearningOptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning. However, most current orthogonalized methods are still paired with fixed, ex…
Stochastic OptimizationSecond-Order Path Kernel Interpolation Formulas in Machine Learning
Understanding how training data shape neural network predictions is a central problem in modern learning theory. In 2020, Pedro Domingos proposed an interpolation formula valid for every model learned by deterministic gr…
Stochastic OptimizationAccelerated Decentralized Stochastic Gradient Descent for Strongly Convex Optimization
Decentralized stochastic optimization is a fundamental paradigm for large-scale learning over networks, where agents communicate only with their neighbors and no central coordinator is required. For strongly convex probl…
Stochastic OptimizationLarge-scale empirical tuning and comparison of default optimizers for variational inference
Black-box variational inference (BBVI) is a methodology for posterior approximation that relies on stochastic optimization. In practice, the stochastic optimizers underpinning BBVI generally require extensive problem-spe…
Stochastic OptimizationBayesian InferenceDiff2SP: Diffusion Models for Correlated Scenario Generation in Stochastic Programming
Scenario generation is a critical component in stochastic programming (SP), as it directly influences the quality of decision-making under uncertainty. Existing approaches predominantly rely on either sampling-based tech…
Stochastic Optimization