incremental gradient methods – Optimization Online

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Published: 2011/06/28, Updated: 2011/11/11

Data-Mining, Nonlinear Optimization, Stochastic Programming incremental gradient methods, machine learning, multicore, parallel computing, stochastic gradient descent

Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks. Several researchers have recently proposed schemes to parallelize SGD, but all require performance-destroying memory locking and synchronization. This work aims to show using novel theoretical analysis, algorithms, and implementation that SGD can be implemented *without … Read more

Parallel Stochastic Gradient Algorithms for Large-Scale Matrix Completion

Published: 2011/04/26, Updated: 2013/03/22

Christopher Re

Benjamin Recht

Data-Mining, Nonlinear Optimization, Optimization Software and Modeling Systems incremental gradient methods, matrix completion, multicore, parallel computing

This paper develops Jellyfish, an algorithm for solving data-processing problems with matrix-valued decision variables regularized to have low rank. Particular examples of problems solvable by Jellyfish include matrix completion problems and least-squares problems regularized by the nuclear norm or the max-norm. Jellyfish implements a projected incremental gradient method with a biased, random ordering of the … Read more