Stochastic Approaches

ASPEN: An Additional Sampling Penalty Method for Finite-Sum Optimization Problems with Nonlinear Equality Constraints

  • Nataša Krejić
  • Nataša Krklec Jerinkić
  • Tijana Ostojić
  • Nemanja Vučićević
    • Categories Nonlinear Optimization, Stochastic Approaches, Stochastic Programming Tags , , , , , ,

    We propose a novel algorithm for solving non-convex, nonlinear equality-constrained finite-sum optimization problems. The proposed algorithm incorporates an additional sampling strategy for sample size update into the well-known framework of quadratic penalty methods. Thus, depending on the problem at hand, the resulting method may exhibit a sample size strategy ranging from a mini-batch on one … Read more

    An Adaptive Stochastic Dual Progressive Hedging Algorithm for Stochastic Programming

  • Di Zhang
  • Yihang Zhang
  • Suvrajeet Sen
    • Categories Stochastic Approaches, Stochastic Programming Tags , , ,

    The Progressive Hedging (PH) algorithm is one of the cornerstones in large-scale stochastic programming. However, its traditional development requires that all scenario subproblems are solved per iteration, and a probability distribution with finitely many outcomes. This paper introduces a stochastic dual PH algorithm (SDPH) to overcome these challenges. We introduce an adaptive sampling process and … Read more

    Variable metric proximal stochastic gradient methods with additional sampling

  • Ilaria Trombini
  • Federica Porta
  • Nataša Krklec Jerinkić
  • Valeria Ruggiero
    • Categories Stochastic Approaches Tags , , ,

    Regularized empirical risk minimization problems arise in a variety of applications, including machine learning, signal processing, and image processing. Proximal stochastic gradient algorithms are a standard approach to solve these problems due to their low computational cost per iteration and a relatively simple implementation. This paper introduces a class of proximal stochastic gradient methods built … Read more

    Spectral Stochastic Gradient Method with Additional Sampling for Finite and Infinite Sums

  • Nataša Krklec Jerinkić
  • Ilaria Trombini
  • Valeria Ruggiero
    • Categories Stochastic Approaches Tags , , , ,

    In this paper, we propose a new stochastic gradient method for numerical minimization of finite sums. We also propose a modified version of this method applicable on more general problems referred to as infinite sum problems, where the objective function is in the form of mathematical expectation. The method is based on a strategy to … Read more

    Global Optimization Algorithm through High-Resolution Sampling

  • Daniel Cortild
  • Juan Peypouquet
    • Categories Global Optimization, Global Optimization Theory, Stochastic Approaches Tags , , , , ,

    We present an optimization algorithm that can identify a global minimum of a potentially nonconvex smooth function with high probability, assuming the Gibbs measure of the potential satisfies a logarithmic Sobolev inequality. Our contribution is twofold: on the one hand we propose a global optimization method, which is built on an oracle sampling algorithm producing … Read more

    A Markovian Model for Learning-to-Optimize

  • Michael Sucker
  • Peter Ochs
    • Categories Optimization in Data Science, Stochastic Approaches, Stochastic Programming Tags , , , ,

    We present a probabilistic model for stochastic iterative algorithms with the use case of optimization algorithms in mind. Based on this model, we present PAC-Bayesian generalization bounds for functions that are defined on the trajectory of the learned algorithm, for example, the expected (non-asymptotic) convergence rate and the expected time to reach the stopping criterion. … Read more

    Convergence analysis on a data-driven inexact proximal-indefinite stochastic ADMM

  • Jianchao Bai
    • Categories Convex Optimization, Stochastic Approaches Tags , , , , ,

    In this paper, we propose an Inexact Proximal-indefinite Stochastic ADMM (abbreviated as IPS-ADMM) to solve a class of separable convex optimization problems whose objective functions consist of two parts: one is an average of many smooth convex functions and the other is a convex but potentially nonsmooth function. The involved smooth subproblem is tackled by … Read more

    Nonexpansive Markov Operators and Random Function Iterations for Stochastic Fixed Point Problems

  • Russell Luke
    • Categories Global Optimization Theory, Stochastic Approaches, Stochastic Programming Tags , ,

    We study the convergence of random function iterations for finding an invariant measure of the corresponding Markov operator. We call the problem of finding such an invariant mea- sure the stochastic fixed point problem. This generalizes earlier work studying the stochastic feasibility problem, namely, to find points that are, with probability 1, fixed points of … Read more

    Expected Value of Matrix Quadratic Forms with Wishart distributed Random Matrices

  • Melinda Hagedorn
    • Categories Convex Optimization, Data Science Theory, Stochastic Approaches Tags , , , , ,

    To explore the limits of a stochastic gradient method, it may be useful to consider an example consisting of an infinite number of quadratic functions. In this context, it is appropriate to determine the expected value and the covariance matrix of the stochastic noise, i.e. the difference of the true gradient and the approximated gradient … Read more