stochastic optimization

Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers

  • Coralia Cartis
  • Jan Fiala
  • Benjamin Marteau
  • Lindon Roberts
    • Categories Nonlinear Optimization Tags , , , , ,

    We present DFO-LS, a software package for derivative-free optimization (DFO) for nonlinear Least-Squares (LS) problems, with optional bound constraints. Inspired by the Gauss-Newton method, DFO-LS constructs simplified linear regression models for the residuals. DFO-LS allows flexible initialization for expensive problems, whereby it can begin making progress from as few as two objective evaluations. Numerical results … Read more

    Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods

  • Albert S. Berahas
  • Richard H. Byrd
  • Jorge Nocedal
    • Categories Nonlinear Optimization, Unconstrained Optimization Tags , ,

    This paper presents a finite difference quasi-Newton method for the minimization of noisy functions. The method takes advantage of the scalability and power of BFGS updating, and employs an adaptive procedure for choosing the differencing interval h based on the noise estimation techniques of Hamming (2012) and Moré and Wild (2011). This noise estimation procedure … Read more

    A Progressive Batching L-BFGS Method for Machine Learning

  • Raghu Bollapragada
  • Jorge Nocedal
  • Hao-Jun Shi
  • Ping Tak Peter Tang
  • Dheevatsa Mudigere
    • Categories Convex Optimization, Nonlinear Optimization, Stochastic Programming Tags , , ,

    The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objective function. All of this appears to call for a full batch approach, but since small batch sizes give rise … Read more

    A Stochastic Trust Region Algorithm Based on Careful Step Normalization

  • Frank E. Curtis
  • Katya Scheinberg
  • Rui Shi
    • Categories Nonlinear Optimization, Stochastic Programming, Unconstrained Optimization Tags , , , , , ,

    An algorithm is proposed for solving stochastic and finite sum minimization problems. Based on a trust region methodology, the algorithm employs normalized steps, at least as long as the norms of the stochastic gradient estimates are within a specified interval. The complete algorithm—which dynamically chooses whether or not to employ normalized steps—is proved to have … Read more

    Random Gradient Extrapolation for Distributed and Stochastic Optimization

  • Guanghui Lan
  • Yi Zhou
    • Categories Convex Optimization, Stochastic Programming Tags , , , ,

    In this paper, we consider a class of finite-sum convex optimization problems defined over a distributed multiagent network with $m$ agents connected to a central server. In particular, the objective function consists of the average of $m$ ($\ge 1$) smooth components associated with each network agent together with a strongly convex term. Our major contribution … Read more

    Computational Aspects of Bayesian Solution Estimators in Stochastic Optimization

  • Gérard Cornuéjols
  • Danial Davarnia
  • Burak Kocuk
    • Categories Statistics, Stochastic Approaches, Stochastic Programming Tags , , , ,

    We study a class of stochastic programs where some of the elements in the objective function are random, and their probability distribution has unknown parameters. The goal is to find a good estimate for the optimal solution of the stochastic program using data sampled from the distribution of the random elements. We investigate two common … Read more

    Adaptive Sampling Strategies for Stochastic Optimization

  • Raghu Bollapragada
  • Richard H. Byrd
  • Jorge Nocedal
    • Categories Convex Optimization, Nonlinear Optimization, Stochastic Programming Tags , ,

    In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by increasing the sample size as needed. The decision to increase … Read more

    From Estimation to Optimization via Shrinkage

  • Gérard Cornuéjols
  • Danial Davarnia
    • Categories Quadratic Programming, Statistics, Stochastic Programming Tags , , , ,

    We study a class of quadratic stochastic programs where the distribution of random variables has unknown parameters. A traditional approach is to estimate the parameters using a maximum likelihood estimator (MLE) and to use this as input in the optimization problem. For the unconstrained case, we show that an estimator that “shrinks” the MLE towards … Read more

    Optimizing power generation in the presence of micro-grids

  • Boris Detienne
  • Michael Poss
  • De Boeck Jérome
  • Pan Stefania
  • Wim van Ackooij
    • Categories 0-1 Programming, Applications - Science and Engineering, Stochastic Programming Tags , , ,

    In this paper we consider energy management optimization problems in a future wherein an interaction with micro-grids has to be accounted for. We will model this interaction through a set of contracts between the generation companies owning centralized assets and the micro-grids. We will formulate a general stylized model that can, in principle, account for … Read more

    The Adaptive Sampling Gradient Method: Optimizing Smooth Functions with an Inexact Oracle

  • Fatemeh Hashemi
  • Raghu Pasupathy
  • Michael Taaffe
    • Categories Nonlinear Optimization, Optimization of Simulated Systems, Stochastic Programming Tags , ,

    Consider settings such as stochastic optimization where a smooth objective function $f$ is unknown but can be estimated with an \emph{inexact oracle} such as quasi-Monte Carlo (QMC) or numerical quadrature. The inexact oracle is assumed to yield function estimates having error that decays with increasing oracle effort. For solving such problems, we present the Adaptive … Read more