Rui Shi – Optimization Online

A Fully Stochastic Second-Order Trust Region Method

Published: 2019/11/15

Nonlinear Optimization, Stochastic Programming, Unconstrained Optimization deep neural networks, finite-sum optimization, machine learning, stochastic newton methods, stochastic optimization, time series forecasting, trust-region methods

A stochastic second-order trust region method is proposed, which can be viewed as a second-order extension of the trust-region-ish (TRish) algorithm proposed by Curtis et al. [INFORMS J. Optim. 1(3) 200–220, 2019]. In each iteration, a search direction is computed by (approximately) solving a trust region subproblem defined by stochastic gradient and Hessian estimates. The … Read more

A Stochastic Trust Region Algorithm Based on Careful Step Normalization

Published: 2017/12/29, Updated: 2018/06/26

Frank E. Curtis

Katya Scheinberg

Rui Shi

Nonlinear Optimization, Stochastic Programming, Unconstrained Optimization deep neural networks, finite sum minimization, logistic regression, machine learning, stochastic gradient method, stochastic optimization, trust-region methods

An algorithm is proposed for solving stochastic and finite sum minimization problems. Based on a trust region methodology, the algorithm employs normalized steps, at least as long as the norms of the stochastic gradient estimates are within a specified interval. The complete algorithm—which dynamically chooses whether or not to employ normalized steps—is proved to have … Read more