line-search

Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking

  • Frederik Kunstner
  • Mark Schmidt
    • Categories Convex Optimization, Nonlinear Optimization Tags , ,

    The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically optimal step-size. Many approaches have been developed to instead tune per-coordinate step-sizes, also known as diagonal preconditioners, but none of the existing methods are provably competitive with the optimal per-coordinate stepsizes. We … Read more

    An Explicit Three-Term Polak-Ribière-Polyak Conjugate Gradient Method for Bicriteria Optimization

  • Mounir El Maghri
  • Youssef Elboulqe
    • Categories Multi-Criteria Optimization, Unconstrained Optimization Tags , , , , ,

    We propose in this paper a Polak-Ribière-Polyak conjugate gradient type method for solving bicriteria optimization problems by avoiding scalarization techniques. Two particular advantages in this contribution are to be noted. First, the suggested descent direction common to both criteria may be directly computed by a given formula without solving any intermediate subproblem. Second, the descent … Read more

    A Sequential Quadratic Programming Method for Optimization with Stochastic Objective Functions, Deterministic Inequality Constraints and Robust Subproblems

  • Songqiang Qiu
  • Vyacheslav Kungurtsev
    • Categories Constrained Nonlinear Optimization, Other Topics, Stochastic Programming Tags , , ,

    In this paper, a robust sequential quadratic programming method of Burke and Han (Math Programming, 1989)  for constrained optimization is generalized to problem with stochastic objective function, deterministic equality and inequality constraints. A stochastic line search scheme in Paquette and Scheinberg (SIOPT, 2020) is employed to globalize the steps. We show that in the case … Read more

    An Explicit Spectral Fletcher-Reeves Conjugate Gradient Method for Bi-criteria Optimization

  • Youssef Elboulqe
  • Mounir El Maghri
    • Categories Multi-Criteria Optimization, Nonlinear Optimization, Unconstrained Optimization Tags , , , ,

    In this paper we propose a spectral Fletcher-Reeves conjugate gradient-like method (SFRCG) for solving unconstrained bi-criteria minimisation problems without using any technique of scalarization. We suggest an explicit formulae for computing a descent direction common to both criteria. This latter verifies furthermore a sufficient descent property which does not depend on the line search nor … Read more

    Spectral Projected Subgradient Method for Nonsmooth Convex Optimization Problems

  • Nataša Krejić
  • Nataša Krklec Jerinkić
  • Tijana Ostojić
    • Categories Constrained Nonlinear Optimization, Convex and Nonsmooth Optimization, Convex Optimization Tags , , , , ,

    We consider constrained optimization problems with a nonsmooth objective function in the form of mathematical expectation. The Sample Average Approximation (SAA) is used to estimate the objective function and variable sample size strategy is employed. The proposed algorithm combines an SAA subgradient with the spectral coefficient in order to provide a suitable direction which improves … Read more

    LSOS: Line-search Second-Order Stochastic optimization methods for nonconvex finite sums

  • Daniela di Serafino
  • Nataša Krejić
  • Nataša Krklec Jerinkić
  • Marco Viola
    • Categories Nonlinear Optimization, Stochastic Programming Tags , , , ,

    We develop a line-search second-order algorithmic framework for minimizing finite sums. We do not make any convexity assumptions, but require the terms of the sum to be continuously differentiable and have Lipschitz-continuous gradients. The methods fitting into this framework combine line searches and suitably decaying step lengths. A key issue is a two-step sampling at … Read more

    A Proximal Interior Point Algorithm with Applications to Image Processing

  • Émilie Chouzenoux
  • Marie-Caroline Corbineau
  • Jean-Christophe Pesquet
    • Categories Constrained Nonlinear Optimization, Convex Optimization, Nonsmooth Optimization Tags , , , , , , , ,

    In this article, we introduce a new proximal interior point algorithm (PIPA). This algorithm is able to handle convex optimization problems involving various constraints where the objective function is the sum of a Lipschitz differentiable term and a possibly nonsmooth one. Each iteration of PIPA involves the minimization of a merit function evaluated for decaying … Read more

    Line search and convergence in bound-constrained optimization

  • Behzad Azmi
  • Arnold Neumaier
    • Categories Bound-constrained Optimization Tags , , , ,

    The first part of this paper discusses convergence properties of a new line search method for the optimization of continuously differentiable functions with Lipschitz continuous gradient. The line search uses (apart from the gradient at the current best point) function values only. After deriving properties of the new, in general curved, line search, global convergence … Read more

    Efficient global unconstrained black box optimization

  • Morteza Kimiaei
  • Arnold Neumaier
    • Categories Stochastic Approaches, Unconstrained Optimization Tags , , , ,

    For the unconstrained optimization of black box functions, this paper introduces a new randomized algorithm called VRBBO. In practice, VRBBO matches the quality of other state-of-the-art algorithms for finding, in small and large dimensions, a local minimizer with reasonable accuracy. Although our theory guarantees only local minimizers our heuristic techniques turn VRBBO into an efficient … Read more