Adam – Optimization Online

Projected Stochastic Momentum Methods for Nonlinear Equality-Constrained Optimization for Machine Learning

Published: 2026/01/16, Updated: 2026/01/20

Constrained Nonlinear Optimization, Nonlinear Optimization, Stochastic Programming Adam, constrained optimization, continuous optimization, heavy-ball method, machine learning, Newton-SQP method

Two algorithms are proposed, analyzed, and tested for solving continuous optimization problems with nonlinear equality constraints. Each is an extension of a stochastic momentum-based method from the unconstrained setting to the setting of a stochastic Newton-SQP-type algorithm for solving equality-constrained problems. One is an extension of the heavy-ball method and the other is an extension … Read more

rAdam: restart Adam method to escape from local minima for bound constrained non-linear optimization problems

Published: 2025/08/13

Thi-Thoi Tran

Nonlinear Optimization, Stochastic Programming Adam, bound-constrained problems, design of experiments, globalization, latin hypercube sampling, Restart Adam, Restart strategy, SGD, Weight Decay

This paper presents a restart version of the Adaptive Moment Estimation (Adam) method for bound constrained nonlinear optimization problems. It aims to avoid getting trapped in a local minima and enable exploration the global optimum. The proposed method combines an adapted restart strategy coupling with barrier methodology to handle the bound constraints. Computational comparison with … Read more