Tao Jiang – Optimization Online

Second-Moment Stochastic Approximation Methods

Published: 2026/10/14

Data Science Algorithms, Data Science Theory, Optimization in Data Science deep learning, second moment preconditioning, stochastic approximation

Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal … Read more

Stochastic Approximation with Block Coordinate Optimal Stepsizes

Published: 2025/07/16

Data Science Algorithms, Optimization in Data Science adaptive algorithms, block coordinate stepsizes, stochastic approximation

We consider stochastic approximation with block-coordinate stepsizes and propose adaptive stepsize rules that aim to minimize the expected distance from the next iterate to an optimal point. These stepsize rules employ online estimates of the second moment of the search direction along each block coordinate. The popular Adam algorithm can be interpreted as a particular … Read more