convergence analysis

Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications

  • Yue Huang
  • Zhaoxian Wu
  • Shiqian Ma
  • Qing Ling
    • Categories Nonlinear Optimization, Stochastic Programming Tags , , ,

    Stochastic approximation (SA) that involves multiple coupled sequences, known as multiple-sequence SA (MSSA), finds diverse applications in the fields of signal processing and machine learning. However, existing theoretical understandings of MSSA are limited: the multi-timescale analysis implies a slow convergence rate, whereas the single-timescale analysis relies on a stringent fixed point smoothness assumption. This paper … Read more

    MUSE-BB: A Decomposition Algorithm for Nonconvex Two-Stage Problems using Strong Multisection Branching

  • Marco Langiu
  • Manuel Dahmen
  • D. Bongartz
  • Alexander Mitsos
    • Categories Global Optimization, Parallel Algorithms, Stochastic Programming Tags , , , ,

    We present MUSE-BB, a branch-and-bound (B&B) based decomposition algorithm for the deterministic global solution of nonconvex two-stage stochastic programming problems. In contrast to three recent decomposition algorithms, which solve this type of problem in a projected form by nesting an inner B&B in an outer B&B on the first-stage variables, we branch on all variables … Read more

    Higher-Order Newton Methods with Polynomial Work per Iteration

  • Amir Ali Ahmadi
  • Abraar Chaudhry
  • Jeffrey Zhang
    • Categories Convex Optimization, Linear, Cone and Semidefinite Programming, Semi-definite Programming Tags , , , ,

    We present generalizations of Newton’s method that incorporate derivatives of an arbitrary order \(d\) but maintain a polynomial dependence on dimension in their cost per iteration. At each step, our \(d^{\text{th}}\)-order method uses semidefinite programming to construct and minimize a sum of squares-convex approximation to the \(d^{\text{th}}\)-order Taylor expansion of the function we wish to … Read more

    A Reduced Jacobian Scheme with Full Convergence for Multicriteria Optimization

  • Mounir El Maghri
  • Youssef Elboulqe
    • Categories Constrained Nonlinear Optimization, Linear Programming, Multi-Criteria Optimization Tags , , , , ,

    In this paper, we propose a variant of the reduced Jacobian method (RJM) introduced by El Maghri and Elboulqe in [JOTA, 179 (2018) 917–943] for multicriteria optimization under linear constraints. Motivation is that, contrarily to RJM which has only global convergence to Pareto KKT-stationary points in the classical sense of accumulation points, this new variant … Read more

    Convergence Results for Primal-Dual Algorithms in the Presence of Adjoint Mismatch

  • Andres Contreras
  • Émilie Chouzenoux
  • Jean-Christophe Pesquet
  • Marion Savanier
    • Categories Convex Optimization, Nonsmooth Optimization Tags , , , , , , ,

    Most optimization problems arising in imaging science involve high-dimensional linear operators and their adjoints. In the implementations of these operators, approximations may be introduced for various practical considerations (e.g., memory limitation, computational cost, convergence speed), leading to an adjoint mismatch. This occurs for the X-ray tomographic inverse problems found in Computed Tomography (CT), where the … Read more

    Unmatched Preconditioning of the Proximal Gradient Algorithm

  • Marion Savanier
  • Émilie Chouzenoux
  • Jean-Christophe Pesquet
  • Cyril Riddel
    • Categories Convex Optimization, Nonsmooth Optimization Tags , , , ,

    This works addresses the resolution of penalized least-squares problems using the proximal gradient algorithm (PGA). It is known that PGA can be accelerated by preconditioning strategies. However, typical effective choices of preconditioners may correspond to intricate matrices that are not easily inverted, and lead to an increased complexity in the computation of the proximity step. … Read more

    Bolstering Stochastic Gradient Descent with Model Building

  • Ş. İlker Birbil
  • Ozgur Martin
  • Gonec Onay
  • Figen Oztoprak
    • Categories Unconstrained Optimization Tags , , ,

    Stochastic gradient descent method and its variants constitute the core optimization algorithms that achieve good convergence rates for solving machine learning problems. These rates are obtained especially when these algorithms are fine-tuned for the application at hand. Although this tuning process can require large computational costs, recent work has shown that these costs can be … Read more

    SABRINA: A Stochastic Subspace Majorization-Minimization Algorithm

  • Jean-Baptiste Fest
  • Émilie Chouzenoux
    • Categories Convex Optimization, Nonlinear Optimization Tags , , , , ,

    A wide class of problems involves the minimization of a coercive and differentiable function $F$ on $\mathbb{R}^N$ whose gradient cannot be evaluated in an exact manner. In such context, many existing convergence results from standard gradient-based optimization literature cannot be directly applied and robustness to errors in the gradient is not necessarily guaranteed. This work … Read more

    Time-Domain Decomposition for Mixed-Integer Optimal Control Problems

  • Falk M. Hante
  • Richard Krug
  • Martin Schmidt
    • Categories (Mixed) Integer Nonlinear Programming, Systems governed by Differential Equations Optimization Tags , , ,

    We consider mixed-integer optimal control problems, whose optimality conditions involve global combinatorial optimization aspects for the corresponding Hamiltonian pointwise in time. We propose a time-domain decomposition, which makes this problem class accessible for mixed-integer programming using parallel-in-time direct discretizations. The approach is based on a decomposition of the optimality system and the interpretation of the … Read more

    Time-Domain Decomposition for Optimal Control Problems Governed by Semilinear Hyperbolic Systems with Mixed Two-Point Boundary Conditions

  • Richard Krug
  • Günter Leugering
  • Martin Schmidt
  • Dieter Weninger
  • Alexander Martin
    • Categories Systems governed by Differential Equations Optimization Tags , , , ,

    In this article, we continue our work (Krug et al., 2021) on time-domain decomposition of optimal control problems for systems of semilinear hyperbolic equations in that we now consider mixed two-point boundary value problems and provide an in-depth well-posedness analysis. The more general boundary conditions significantly enlarge the scope of applications, e.g., to hyperbolic problems … Read more