Adaptive Gradient Descent

bAdag: an adaptive block coordinate gradient method for smooth nonconvex functions

Categories Data Science Algorithms, Unconstrained Optimization Tags , , ,

A new Block Coordinate Gradient (BCG) method, dubbed bAdag, for smooth, nonconvex minimization problem is proposed; it falls in the class of Objective Function Free Optimization (OFFO) methods, and it is based on the AdaGrad algorithm. At each iteration, our method computes an adaptive step size based on the cumulative sum of block gradients, instead … Read more

AdaBB: Adaptive Barzilai-Borwein Method for Convex Optimization

Categories Convex Optimization Tags , , , , ,

In this paper, we propose AdaBB, an adaptive gradient method based on the Barzilai-Borwein stepsize. The algorithm is line-search-free and parameter-free, and essentially provides a convergent variant of the Barzilai-Borwein method for general unconstrained convex optimization. We analyze the ergodic convergence of the objective function value and the convergence of the iterates for solving general … Read more