adaptive first-order methods

Gradient Methods with Online Scaling Part I. Theoretical Foundations

  • Wenzhi Gao
    • Categories Convex Optimization, Nonlinear Optimization, Unconstrained Optimization Tags , ,

    This paper establishes the theoretical foundations of the online scaled gradient methods (OSGM), a framework that utilizes online learning to adapt stepsizes and provably accelerate first-order methods. OSGM quantifies the effectiveness of a stepsize by a feedback function motivated from a convergence measure and uses the feedback to adjust the stepsize through an online learning … Read more

    prunAdag: an adaptive pruning-aware gradient method

  • Margherita Porcelli
  • Giovanni Seraghiti
  • Philippe L. Toint
    • Categories Data Science Algorithms, Nonlinear Optimization Tags , , ,

    A pruning-aware adaptive gradient method is proposed which classifies the variables in two sets before updating them using different strategies. This technique extends the “relevant/irrelevant” approach of Ding (2019) and Zimmer et al. (2022) and allows a posteriori sparsification of the solution of model parameter fitting problems. The new method is proved to be convergent … Read more