symmetric rank-one update – Optimization Online

Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximations

Published: 2017/10/26, Updated: 2019/05/22

Applications - Science and Engineering, Unconstrained Optimization machine learning, nonconvex optimization, quasi-newton methods, symmetric rank-one update

Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the limited-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) update typically do not require manually tuning hyper-parameters but … Read more

ALGORITHM XXX: SC-SR1: MATLAB SOFTWARE FOR SOLVING SHAPE-CHANGING L-SR1 TRUST-REGION SUBPROBLEMS

Published: 2016/07/12, Updated: 2018/02/17

Unconstrained Optimization large-scale unconstrained optimization, limited memory quasi-newton methods, nonconvex optimization, shape-changing norm, symmetric rank-one update, trust-region methods

We present a MATLAB implementation of the shape-changing sym- metric rank-one (SC-SR1) method that solves trust-region subproblems when a limited-memory symmetric rank-one (L-SR1) matrix is used in place of the true Hessian matrix. The method takes advantage of two shape-changing norms [4, 3] to decompose the trust-region subproblem into two separate problems. Using one of … Read more

On solving large-scale limited-memory quasi-Newton equations

Published: 2015/10/21, Updated: 2016/01/17

Jennifer B. Erway

Roummel F. Marcia

Nonlinear Optimization broyden-fletcher-goldfarb-shanno update, compact representation, davidon-fletcher-powell update, limited memory quasi-newton methods, restricted broyden class of updates, sherman-morrison-woodbury formula, symmetric rank-one update

We consider the problem of solving linear systems of equations with limited- memory members of the restricted Broyden class and symmetric rank-one matrices. In this paper, we present various methods for solving these linear systems, and propose a new approach based on a practical implementation of the compact representation for the inverse of these limited-memory … Read more

On Solving L-SR1 Trust-Region Subproblems

Published: 2015/06/18, Updated: 2016/08/12

Johannes Brust

Jennifer B. Erway

Roummel F. Marcia

Optimization Software and Modeling Systems, Unconstrained Optimization large-scale unconstrained optimization, limited memory quasi-newton methods, symmetric rank-one update, trust-region methods

In this article, we consider solvers for large-scale trust-region subproblems when the quadratic model is defined by a limited-memory symmetric rank-one (L-SR1) quasi-Newton matrix. We propose a solver that exploits the compact representation of L-SR1 matrices. Our approach makes use of both an orthonormal basis for the eigenspace of the L-SR1 matrix and the Sherman- … Read more