stochastic gradient descent – Page 2

Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes

Published: 2018/10/04, Updated: 2020/06/06

Robust Optimization, Stochastic Programming adversarial, comparative statics, distributionally robust optimization, optimal transport, rate of convergence, stochastic gradient descent, strong convexity, wasserstein distance

We consider optimal transport based distributionally robust optimization (DRO) problems with locally strongly convex transport cost functions and affine decision rules. Under conventional convexity assumptions on the underlying loss function, we obtain structural results about the value function, the optimal policy, and the worst-case optimal transport adversarial model. These results expose a rich structure embedded … Read more

Barzilai-Borwein Step Size for Stochastic Gradient Descent

Published: 2016/05/22

Convex Optimization, Stochastic Approaches barzilai-borwein method, step size, stochastic gradient descent

One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a fixed step … Read more

Factoring nonnegative matrices with linear programs

Published: 2012/06/06

Data-Mining, Linear Programming linear programming, machine learning, multicore, nonnegative matrix factorization, parallel computing, stochastic gradient descent

This paper describes a new approach for computing nonnegative matrix factorizations (NMFs) with linear programming. The key idea is a data-driven model for the factorization, in which the most salient features in the data are used to express the remaining features. More precisely, given a data matrix X, the algorithm identifies a matrix C that … Read more

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Published: 2011/06/28, Updated: 2011/11/11

Data-Mining, Nonlinear Optimization, Stochastic Programming incremental gradient methods, machine learning, multicore, parallel computing, stochastic gradient descent

Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks. Several researchers have recently proposed schemes to parallelize SGD, but all require performance-destroying memory locking and synchronization. This work aims to show using novel theoretical analysis, algorithms, and implementation that SGD can be implemented *without … Read more