Nonlinear Optimization

New efficient accelerated and stochastic gradient descent algorithms based on locally Lipschitz gradient constants

  • Pham Thi Hoai
    • Categories Convex Optimization, Nonlinear Optimization, Stochastic Programming Tags , , , , , , ,

    In this paper, we revisit the recent stepsize applied for the gradient descent scheme which is called NGD proposed by [Hoai et al., A novel stepsize for gradient descent method, Operations Research Letters (2024) 53, doi: \href{10.1016/j.orl.2024.107072}{10.1016/j.orl.2024.107072}]. We first investigate NGD stepsize with two well-known accelerated techniques which are Heavy ball and Nesterov’s methods. In … Read more

    Gradient Methods with Online Scaling

  • Wenzhi Gao
    • Categories Convex Optimization, Nonlinear Optimization, Other Topics Tags , ,

    We introduce a framework to accelerate the convergence of gradient-based methods with online learning. The framework learns to scale the gradient at each iteration through an online learning algorithm and provably accelerates gradient-based methods asymptotically. In contrast with previous literature, where convergence is established based on worst-case analysis, our framework provides a strong convergence guarantee … Read more

    A Trust-Region Algorithm for Noisy Equality Constrained Optimization

  • Shigeng Sun
  • Jorge Nocedal
    • Categories Constrained Nonlinear Optimization, Nonlinear Optimization

    This paper introduces a modified Byrd-Omojokun (BO) trust region algorithm to address the challenges posed by noisy function and gradient evaluations. The original BO method was designed to solve equality constrained problems and it forms the backbone of some interior point methods for general large-scale constrained optimization. A key strength of the BO method is … Read more

    New results related to cutters and to an extrapolated block-iterative method for finding a common fixed point of a collection of them

  • Yair Censor
  • Daniel Reem
  • Maroun Zaknoon
    • Categories Convex Optimization, Nonlinear Optimization, Parallel Algorithms Tags , , , ,

    Given a Hilbert space and a finite family of operators defined on the space, the common fixed point problem (CFPP) is to find a point in the intersection of the fixed point sets of these operators.  Instances of the problem have numerous applications in science and engineering. We consider an extrapolated block-iterative method with dynamic … Read more

    A tutorial on properties of the epigraph reformulation

  • Oliver Stein
    • Categories Nonlinear Optimization Tags , , , , , ,

    This paper systematically surveys useful properties of the epigraph reformulation for optimization problems, and complements them by some new results. We focus on the complete compatibility of the original formulation and the epigraph reformulation with respect to solvability and unsolvability, the compatibility with respect to some, but not all, basic constraint qualifications, the formulation of … Read more

    Parameter-free proximal bundle methods with adaptive stepsizes for hybrid convex composite optimization problems

  • Renato D.C. Monteiro
  • Honghao Zhang
    • Categories Convex and Nonsmooth Optimization, Convex Optimization, Nonlinear Optimization, Other Topics Tags , , ,

    This paper develops a parameter-free adaptive proximal bundle method with two important features: 1) adaptive choice of variable prox stepsizes that “closely fits” the instance under consideration; and 2) adaptive criterion for making the occurrence of serious steps easier. Computational experiments show that our method performs substantially fewer consecutive null steps (i.e., a shorter cycle) … Read more

    Fully First-Order Methods for Decentralized Bilevel Optimization

  • Xiaoyu Wang
  • Xuxing Chen
  • Shiqian Ma
  • Tong Zhang
    • Categories Nonlinear Optimization, Parallel Algorithms, Stochastic Programming Tags , , ,

    This paper focuses on decentralized stochastic bilevel optimization (DSBO) where agents only communicate with their neighbors. We propose Decentralized Stochastic Gradient Descent and Ascent with Gradient Tracking (DSGDA-GT), a novel algorithm that only requires first-order oracles that are much cheaper than second-order oracles widely adopted in existing works. We further provide a finite-time convergence analysis … Read more

    An inexact ADMM for separable nonconvex and nonsmooth optimization

  • Jianchao Bai
  • Miao Zhang
  • Hongchao Zhang
    • Categories Nonsmooth Optimization, Quadratic Programming

    An Inexact Alternating Direction Method of Multiplies (I-ADMM) with an expansion linesearch step was developed for solving a family of separable minimization problems subject to linear constraints, where the objective function is the sum of a smooth but possibly nonconvex function and a possibly nonsmooth nonconvex function. Global convergence and linear convergence rate of the … Read more

    Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications

  • Yue Huang
  • Zhaoxian Wu
  • Shiqian Ma
  • Qing Ling
    • Categories Nonlinear Optimization, Stochastic Programming Tags , , ,

    Stochastic approximation (SA) that involves multiple coupled sequences, known as multiple-sequence SA (MSSA), finds diverse applications in the fields of signal processing and machine learning. However, existing theoretical understandings of MSSA are limited: the multi-timescale analysis implies a slow convergence rate, whereas the single-timescale analysis relies on a stringent fixed point smoothness assumption. This paper … Read more

    Efficient parameter-free restarted accelerated gradient methods for convex and strongly convex optimization

  • Arnesh Sujanani
  • Renato D.C. Monteiro
    • Categories Convex and Nonsmooth Optimization, Convex Optimization, Nonlinear Optimization Tags , , , , ,

    This paper develops a new parameter-free restarted method, namely RPF-SFISTA, and a new parameter-free aggressive regularization method, namely A-REG, for solving strongly convex and convex composite optimization problems, respectively. RPF-SFISTA has the major advantage that it requires no knowledge of both the strong convexity parameter of the entire composite objective and the Lipschitz constant of … Read more