Unconstrained Optimization

An adaptive regularization algorithm for unconstrained optimization with inexact function and derivatives values

  • Nicholas I. M. Gould
  • Philippe L. Toint
    • Categories Nonlinear Systems and Least-Squares, Unconstrained Optimization Tags , , ,

    An adaptive regularization algorithm for unconstrained nonconvex optimization is proposed that is capable of handling inexact objective-function and derivative values, and also of providing approximate minimizer of arbitrary order. In comparison with a similar algorithm proposed in Cartis, Gould, Toint (2022), its distinguishing feature is that it is based on controlling the relative error between … Read more

    Trust-region algorithms: probabilistic complexity and intrinsic noise with applications to subsampling techniques

  • Stefania Bellavia
  • Gianmarco Gurioli
  • Benedetta Morini
  • Philippe L. Toint
    • Categories Nonlinear Systems and Least-Squares, Unconstrained Optimization Tags , , , , ,

    A trust-region algorithm is presented for finding approximate minimizers of smooth unconstrained functions whose values and derivatives are subject to random noise. It is shown that, under suitable probabilistic assumptions, the new method finds (in expectation) an epsilon-approximate minimizer of arbitrary order q > 0 in at most O(epsilon^{-(q+1)}) inexact evaluations of the function and … Read more

    OPM, a collection of Optimization Problems in Matlab

  • Serge Gratton
  • Philippe L. Toint
    • Categories Bound-constrained Optimization, Optimization Software Benchmark, Unconstrained Optimization Tags , , ,

    OPM is a small collection of CUTEst unconstrained and bound-constrained nonlinear optimization problems, which can be used in Matlab for testing optimization algorithms directly (i.e. without installing additional software). Article Download View OPM, a collection of Optimization Problems in Matlab

    Convergence Analysis of Block Majorize-Minimize Subspace Approaches

  • Jean-Baptiste Fest
  • Émilie Chouzenoux
    • Categories Applications - Science and Engineering, Unconstrained Optimization Tags , , ,

    Majorization-Minimization (MM) consists of a class of efficient and effective optimization algorithms that benefit from solid theoretical foundations. MM methods have shown their great ability to tackle efficiently challenging optimization problems from signal processing, image processing, inverse problems and machine learning. When processing large amount of data/variable, as it may happen in 3D image processing, … Read more

    Analysis non-sparse recovery for non-convex relaxed $\ell_q$ minimization

  • Jianwen Huang
  • Feng Zhang
  • Xinling Liu
  • Jianjun Wang
    • Categories Data-Mining, Nonsmooth Optimization, Unconstrained Optimization Tags , , , ,

    This paper studies construction of signals, which are sparse or nearly sparse with respect to a tight frame $D$ from underdetermined linear systems. In the paper, we propose a non-convex relaxed $\ell_q(0 Article Download View Analysis non-sparse recovery for non-convex relaxed $ell_q$ minimization

    Nonlinear conjugate gradient for smooth convex functions

  • Sahar Karimi
  • Stephen A. Vavasis
    • Categories Convex Optimization, Unconstrained Optimization Tags , ,

    The method of nonlinear conjugate gradients (NCG) is widely used in practice for unconstrained optimization, but it satisfies weak complexity bounds at best when applied to smooth convex functions. In contrast, Nesterov’s accelerated gradient (AG) method is optimal up to constant factors for this class. However, when specialized to quadratic function, conjugate gradient is optimal … Read more

    Bolstering Stochastic Gradient Descent with Model Building

  • Ş. İlker Birbil
  • Ozgur Martin
  • Gonec Onay
  • Figen Oztoprak
    • Categories Unconstrained Optimization Tags , , ,

    Stochastic gradient descent method and its variants constitute the core optimization algorithms that achieve good convergence rates for solving machine learning problems. These rates are obtained especially when these algorithms are fine-tuned for the application at hand. Although this tuning process can require large computational costs, recent work has shown that these costs can be … Read more

    Regularized Step Directions in Conjugate Gradient Minimization for Machine Learning

  • Cassidy K. Buhler
  • Hande Benson
  • David F. Shanno
    • Categories Nonlinear Optimization, Unconstrained Optimization Tags , ,

    Conjugate gradient minimization methods (CGM) and their accelerated variants are widely used in machine learning applications. We focus on the use of cubic regularization to improve the CGM direction independent of the steplength (learning rate) computation. Using Shanno’s reformulation of CGM as a memoryless BFGS method, we derive new formulas for the regularized step direction, … Read more

    A quasi-Newton method with Wolfe line searches for multiobjective optimization

  • Leandro Fonseca Prudente
  • D. R. Souza
    • Categories Multi-Criteria Optimization, Unconstrained Optimization Tags , , , ,

    We propose a BFGS method with Wolfe line searches for unconstrained multiobjective optimization problems. The algorithm is well defined even for general nonconvex problems. Global convergence and R-linear convergence to a Pareto optimal point are established for strongly convex problems. In the local convergence analysis, if the objective functions are locally strongly convex with Lipschitz … Read more

    An extended delayed weighted gradient algorithm for solving strongly convex optimization problems

  • Roberto Andreani
  • Harry Oviedo
  • Marcos Raydan
  • Leonardo D. Secchin
    • Categories Unconstrained Optimization Tags , , ,

    The recently developed delayed weighted gradient method (DWGM) is competitive with the well-known conjugate gradient (CG) method for the minimization of strictly convex quadratic functions. As well as the CG method, DWGM has some key optimality and orthogonality properties that justify its practical performance. The main difference with the CG method is that, instead of … Read more