A new class of asynchronous adaptive first-order optimization methods is introduced, comprising asynchronous variants of several popular
algorithms. Versions of these methods using momentum and/or inexact normalization are also considered. The convergence of methods in the
class on non-convex functions is analyzed in a fully stochastic setting, and is shown to be (up to logarithmic factors) of order lO(1/sqrt{t}) under
reasonable assumptions. Numerical experiments suggest that such asynchronous adaptive algorithms are very relevant in heterogeneous large-scale machine learning system