heavy-ball method – Optimization Online

Projected Stochastic Momentum Methods for Nonlinear Equality-Constrained Optimization for Machine Learning

Published: 2026/01/16, Updated: 2026/01/20

Constrained Nonlinear Optimization, Nonlinear Optimization, Stochastic Programming Adam, constrained optimization, continuous optimization, heavy-ball method, machine learning, Newton-SQP method

Two algorithms are proposed, analyzed, and tested for solving continuous optimization problems with nonlinear equality constraints. Each is an extension of a stochastic momentum-based method from the unconstrained setting to the setting of a stochastic Newton-SQP-type algorithm for solving equality-constrained problems. One is an extension of the heavy-ball method and the other is an extension … Read more

Local Convergence of the Heavy-ball Method and iPiano for Non-convex Optimization

Published: 2017/06/09, Updated: 2018/01/29

Peter Ochs

Nonsmooth Optimization alternating projection, averaged projection, gradient of moreau envelopes, heavy-ball method, inertial forward-backward splitting, ipiano, non-convex feasibility, prox-regularity

A local convergence result for abstract descent methods is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial … Read more

Global convergence of the Heavy-ball method for convex optimization

Published: 2014/12/23

Hamid Reza Feyzmahdavian

Euhanna Ghadimi

Mikael Johansson

Convex Optimization, Unconstrained Optimization complexity, gradient descent, heavy-ball method, performance of first-order algorithms, rate of convergence, smooth convex optimization

This paper establishes global convergence and provides global bounds of the convergence rate of the Heavy-ball method for convex optimization problems. When the objective function has Lipschitz-continuous gradient, we show that the Cesa ́ro average of the iterates converges to the optimum at a rate of $O(1/k)$ where k is the number of iterations. When … Read more

iPiano: Inertial Proximal Algorithm for Nonconvex Optimization

Published: 2014/06/28, Updated: 2015/01/10

Nonlinear Optimization heavy-ball method, inertial forward-backward splitting, kurdyka-lojasiewicz inequality, nonconvex optimization, proof of convergence

In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly nonconvex) and a convex (possibly nondifferentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a nonsmooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm … Read more