inexact method – Optimization Online

Accelerating Inexact Successive Quadratic Approximation for Regularized Optimization Through Manifold Identification

Published: 2020/12/04, Updated: 2023/04/28

Convex and Nonsmooth Optimization, Nonlinear Optimization inexact method, manifold identification, regularized optimization, superlinear convergence, variable metric

For regularized optimization that minimizes the sum of a smooth term and a regularizer that promotes structured solutions, inexact proximal-Newton-type methods, or successive quadratic approximation (SQA) methods, are widely used for their superlinear convergence in terms of iterations. However, unlike the counter parts in smooth optimization, they suffer from lengthy running time in solving regularized … Read more

A Distributed Quasi-Newton Algorithm for Primal and Dual Regularized Empirical Risk Minimization

Published: 2019/12/12, Updated: 2019/12/17

Stephen Wright

Cong Han Lim

Ching-pei Lee

Nonsmooth Optimization, Parallel Algorithms distributed optimization, empirical riskminimization, inexact method, nonsmooth optimization, proximal method, quasi-newton methods, regularized optimization, variable metrics

We propose a communication- and computation-efficient distributed optimization algorithm using second-order information for solving empirical risk minimization (ERM) problems with a nonsmooth regularization term. Our algorithm is applicable to both the primal and the dual ERM problem. Current second-order and quasi-Newton methods for this problem either do not work well in the distributed setting or … Read more

A Distributed Quasi-Newton Algorithm for Empirical Risk Minimization with Nonsmooth Regularization

Published: 2018/03/04, Updated: 2018/05/26

Stephen Wright

Cong Han Lim

Ching-pei Lee

Data-Mining, Nonlinear Optimization, Parallel Algorithms distributed optimization, empirical risk minimization, inexact method, nonsmooth optimization, proximal method, quasi-newton methods, regularized optimization, variable metrics

We propose a communication- and computation-efficient distributed optimization algorithm using second-order information for solving ERM problems with a nonsmooth regularization term. Current second-order and quasi-Newton methods for this problem either do not work well in the distributed setting or work only for specific regularizers. Our algorithm uses successive quadratic approximations, and we describe how to … Read more

Inexact Successive Quadratic Approximation for Regularized Optimization

Published: 2018/03/03, Updated: 2019/01/24

Stephen Wright

Ching-pei Lee

Nonlinear Optimization, Nonsmooth Optimization convex optimization, inexact method, nonconvex optimization, proximal method, regularized optimization, second-order approximation, variable metric

Successive quadratic approximations, or second-order proximal methods, are useful for minimizing functions that are a sum of a smooth part and a convex, possibly nonsmooth part that promotes regularization. Most analyses of iteration complexity focus on the special case of proximal gradient method, or accelerated variants thereof. There have been only a few studies of … Read more