first-order methods

Hankel Matrix Rank Minimization with Applications to System Identification and Realization

  • Maryam Fazel
  • Ting Kei Pong
  • Defeng Sun
  • Paul Tseng
    • Categories Applications - Science and Engineering, Convex and Nonsmooth Optimization, Linear, Cone and Semidefinite Programming Tags , , , , ,

    We introduce a flexible optimization framework for nuclear norm minimization of matrices with linear structure, including Hankel, Toeplitz and moment structures, and catalog applications from diverse fields under this framework. We discuss various first-order methods for solving the resulting optimization problem, including alternating direction methods of multipliers, proximal point algorithms and gradient projection methods. We … Read more

    Fast global convergence of gradient methods for high-dimensional statistical recovery

  • Alekh Agarwal
  • Martin J Wainwright
  • Sahand Negahban
    • Categories Convex Optimization, Statistics Tags , ,

    Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framework that allows the data dimension $\pdim$ to grow with (and possibly exceed) … Read more

    An acceleration procedure for optimal first-order methods

  • Michel Baes
  • Michael Bürgisser
    • Categories Convex Optimization, Semi-definite Programming Tags , ,

    We introduce in this paper an optimal first-order method that allows an easy and cheap evaluation of the local Lipschitz constant of the objective’s gradient. This constant must ideally be chosen at every iteration as small as possible, while serving in an indispensable upper bound for the value of the objective function. In the previously … Read more

    CONJUGATE GRADIENT WITH SUBSPACE OPTIMIZATION

  • Stephen A. Vavasis
  • Sahar Karimi
    • Categories Convex Optimization, Unconstrained Optimization Tags , ,

    In this paper we present a variant of the conjugate gradient (CG) algorithm in which we invoke a subspace minimization subproblem on each iteration. We call this algorithm CGSO for “conjugate gradient with subspace optimization”. It is related to earlier work by Nemirovsky and Yudin. We apply the algorithm to solve unconstrained strictly convex problems. … Read more

    Stochastic first order methods in smooth convex optimization.

  • Olivier Devolder
    • Categories Convex Optimization, Stochastic Programming Tags , , , , , ,

    In this paper, we are interested in the development of efficient first-order methods for convex optimization problems in the simultaneous presence of smoothness of the objective function and stochasticity in the first-order information. First, we consider the Stochastic Primal Gradient method, which is nothing else but the Mirror Descent SA method applied to a smooth … Read more

    Fast First-Order Methods for Stable Principal Component Pursuit

  • Necdet Serhat Aybat
  • Donald Goldfarb
  • Garud Iyengar
    • Categories Applications - Science and Engineering, Convex and Nonsmooth Optimization Tags , , , , ,

    The stable principal component pursuit (SPCP) problem is a non-smooth convex optimization problem, the solution of which has been shown both in theory and in practice to enable one to recover the low rank and sparse components of a matrix whose elements have been corrupted by Gaussian noise. In this paper, we first show how … Read more

    A Sparsity Preserving Stochastic Gradient Method for Composite Optimization

  • Xi Chen
  • Qihang Lin
  • Javier F. Peña
    • Categories Convex Optimization Tags , ,

    We propose new stochastic gradient algorithms for solving convex composite optimization problems. In each iteration, our algorithms utilize a stochastic oracle of the gradient of the smooth component in the objective function. Our algorithms are based on a stochastic version of the estimate sequence technique introduced by Nesterov (Introductory Lectures on Convex Optimization: A Basic … Read more

    FAST FIRST-ORDER METHODS FOR COMPOSITE CONVEX OPTIMIZATION WITH BACKTRACKING

  • Xi Bai
  • Donald Goldfarb
  • Katya Scheinberg
    • Categories Nonsmooth Optimization Tags , , , , ,

    We propose new versions of accelerated first order methods for convex composite optimization, where the prox parameter is allowed to increase from one iteration to the next. In particular we show that a full backtracking strategy can be used within the FISTA \cite{Beck-Teboulle-2009} and FALM algorithms \cite{Goldfarb-Ma-Scheinberg-2010} while preserving their worst-case iteration complexities of $O(\sqrt{L(f)/\epsilon})$. … Read more

    First-order Methods of Smooth Convex Optimization with Inexact Oracle

  • Olivier Devolder
  • François Glineur
  • Yurii Nesterov
    • Categories Convex Optimization Tags , , , , ,

    We introduce the notion of inexact first-order oracle and analyze the behaviour of several first-order methods of smooth convex optimization used with such an oracle. This notion of inexact oracle naturally appears in the context of smoothing techniques, Moreau-Yosida regularization, Augmented Lagrangians and many other situations. We derive complexity estimates for primal, dual and fast … Read more

    On the oracle complexity of first-order and derivative-free algorithms for smooth nonconvex minimization

  • Coralia Cartis
  • Nicholas I. M. Gould
  • Philippe L. Toint
    • Categories Nonlinear Systems and Least-Squares, Unconstrained Optimization Tags , , , , ,

    The (optimal) function/gradient evaluations worst-case complexity analysis available for the Adaptive Regularizations algorithms with Cubics (ARC) for nonconvex smooth unconstrained optimization is extended to finite-difference versions of this algorithm, yielding complexity bounds for first-order and derivative free methods applied on the same problem class. A comparison with the results obtained for derivative-free methods by Vicente … Read more