Parallel Algorithms

Selection of variables in parallel space decomposition for the mesh adaptive direct search algorithm

  • Stéphane Alarie
  • Nadir Amaioua
  • Charles Audet
  • Sébastien Le Digabel
  • Louis-Alexandre Leclaire
    • Categories Nonlinear Optimization, Parallel Algorithms

    The parallel space decomposition of the Mesh Adaptive Direct Search algorithm (PSDMADS proposed in 2008) is an asynchronous parallel method for constrained derivative-free optimization with large number of variables. It uses a simple generic strategy to decompose a problem into smaller dimension subproblems. The present work explores new strategies for selecting subset of variables defining … Read more

    A Distributed Quasi-Newton Algorithm for Empirical Risk Minimization with Nonsmooth Regularization

  • Stephen Wright
  • Cong Han Lim
  • Ching-pei Lee
    • Categories Data-Mining, Nonlinear Optimization, Parallel Algorithms Tags , , , , , , ,

    We propose a communication- and computation-efficient distributed optimization algorithm using second-order information for solving ERM problems with a nonsmooth regularization term. Current second-order and quasi-Newton methods for this problem either do not work well in the distributed setting or work only for specific regularizers. Our algorithm uses successive quadratic approximations, and we describe how to … Read more

    An Accelerated Communication-Efficient Primal-Dual Optimization Framework for Structured Machine Learning

  • Frank E. Curtis
  • Martin Jaggi
  • Chenxin Ma
  • Nathan Srebro
  • Martin Takac
    • Categories Nonlinear Optimization, Parallel Algorithms

    Distributed optimization algorithms are essential for training machine learning models on very large-scale datasets. However, they often suffer from communication bottlenecks. Confronting this issue, a communication-efficient primal-dual coordinate ascent framework (CoCoA) and its improved variant CoCoA+ have been proposed, achieving a convergence rate of $\mathcal{O}(1/t)$ for solving empirical risk minimization problems with Lipschitz continuous losses. … Read more

    DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization

  • Weizhu Chen
  • Qihang Lin
  • Lin Xiao
  • Wei Yu
    • Categories Convex and Nonsmooth Optimization, Parallel Algorithms Tags , , , , ,

    Machine learning with big data often involves large optimization models. For distributed optimization over a cluster of machines, frequent communication and synchronization of all model parameters (optimization variables) can be very costly. A promising solution is to use parameter servers to store different subsets of the model parameters, and update them asynchronously at different machines … Read more

    Parallel Solvers for Mixed Integer Linear Optimization

  • Timo Berthold
  • Thorsten Koch
  • Ted K. Ralphs
  • Yuji Shinano
    • Categories (Mixed) Integer Linear Programming, Branch and Cut Algorithms, Parallel Algorithms Tags , , ,

    In this article, we provide an overview of the current state of the art with respect to solution of mixed integer linear optimization problems (MILPS) in parallel. Sequential algorithms for solving MILPs have improved substantially in the last two decades and commercial MILP solvers are now considered effective off-the-shelf tools for optimization. Although concerted development … Read more

    FiberSCIP – A shared memory parallelization of SCIP

  • Stefan Heinz
  • Yuji Shinano
  • Stefan Vigerske
  • Michael Winkler
    • Categories Integer Programming, Parallel Algorithms

    Recently, parallel computing environments have become significantly popular. In order to obtain the benefit of using parallel computing environments, we have to deploy our programs for these effectively. This paper focuses on a parallelization of SCIP (Solving Constraint Integer Programs), which is a MIP solver and constraint integer programming framework available in source code. There … Read more

    Symmetric ADMM with Positive-Indefinite Proximal Regularization for Linearly Constrained Convex Optimization

  • Bin Gao
  • Feng Ma
    • Categories Complementarity and Variational Inequalities, Convex Optimization, Parallel Algorithms Tags , , ,

    The proximal ADMM which adds proximal regularizations to ADMM’s subproblems is a popular and useful method for linearly constrained separable convex problems, especially its linearized case. A well-known requirement on guaranteeing the convergence of the method in the literature is that the proximal regularization must be positive semidefinite. Recently it was shown by He et … Read more

    Asynchronous Parallel Algorithms for Nonconvex Big-Data Optimization. Part I: Model and Convergence

  • Loris Cannelli
  • Francisco Facchinei
  • Vyacheslav Kungurtsev
  • Gesualdo Scutari
    • Categories Convex and Nonsmooth Optimization, Nonlinear Optimization, Parallel Algorithms Tags , , ,

    We propose a novel asynchronous parallel algorithmic framework for the minimization of the sum of a smooth nonconvex function and a convex nonsmooth regularizer, subject to both convex and nonconvex constraints. The proposed framework hinges on successive convex approximation techniques and a novel probabilistic model that captures key elements of modern computational architectures and asynchronous … Read more

    Asynchronous Parallel Algorithms for Nonconvex Big-Data Optimization. Part II: Complexity and Numerical Results

  • Loris Cannelli
  • Francisco Facchinei
  • Vyacheslav Kungurtsev
  • Gesualdo Scutari
    • Categories Convex and Nonsmooth Optimization, Nonlinear Optimization, Parallel Algorithms Tags , , ,

    We present complexity and numerical results for a new asynchronous parallel algorithmic method for the minimization of the sum of a smooth nonconvex function and a convex nonsmooth regularizer, subject to both convex and nonconvex constraints. The proposed method hinges on successive convex approximation techniques and a novel probabilistic model that captures key elements of … Read more

    On the Convergence of Asynchronous Parallel Iteration with Arbitrary Delays

  • Zhimin Peng
  • Yangyang Xu
  • Ming Yan
  • Wotao Yin
    • Categories Nonlinear Optimization, Parallel Algorithms Tags , , , ,

    Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables. Because of asynchrony, the iterates are computed with outdated information, and the age of the outdated information, which we call \emph{delay}, is the number of times it has been … Read more