machine learning

Substantiation of the Backpropagation Technique via the Hamilton-Pontryagin Formalism for Training Nonconvex Nonsmooth Neural Networks

  • Vladimir I. Norkin
    • Categories Convex and Nonsmooth Optimization, Nonlinear Optimization, Stochastic Programming Tags , , , , ,

    The paper observes the similarity between the stochastic optimal control of discrete dynamical systems and the training multilayer neural networks. It focuses on contemporary deep networks with nonconvex nonsmooth loss and activation functions. In the paper, the machine learning problems are treated as nonconvex nonsmooth stochastic optimization problems. As a model of nonsmooth nonconvex dependences, … Read more

    Generalized Gradients in Problems of Dynamic Optimization, Optimal Control, and Machine Learning

  • Vladimir I. Norkin
    • Categories Nonsmooth Optimization, Stochastic Programming Tags , , , , , , ,

    In this work, nonconvex nonsmooth problems of dynamic optimization, optimal control in discrete time (including feedback control), and machine learning are considered from a common point of view. An analogy is observed between tasks of controlling discrete dynamic systems and training multilayer neural networks with nonsmooth target function and connections. Methods for calculating generalized gradients … Read more

    Distance geometry and data science

  • Leo Liberti
    • Categories Data-Mining, Global Optimization Applications, Graphs and Matroids Tags , , , , ,

    Data are often represented as graphs. Many common tasks in data science are based on distances between entities. While some data science methodologies natively take graphs as their input, there are many more that take their input in vectorial form. In this survey we discuss the fundamental problem of mapping graphs to vectors, and its … Read more

    A Survey of Recent Scalability Improvements for Semidefinite Programming with Applications in Machine Learning, Control, and Robotics

  • Amir Ali Ahmadi
  • Georgina Hall
  • Anirudha Majumdar
    • Categories Basic Sciences Applications, Control Applications, Semi-definite Programming Tags , , , ,

    Historically, scalability has been a major challenge to the successful application of semidefinite programming in fields such as machine learning, control, and robotics. In this paper, we survey recent approaches for addressing this challenge including (i) approaches for exploiting structure (e.g., sparsity and symmetry) in a problem, (ii) approaches that produce low-rank approximate solutions to … Read more

    General risk measures for robust machine learning

  • Émilie Chouzenoux
  • Jean-Christophe Pesquet
  • Henri Gérard
    • Categories Convex Optimization, Robust Optimization, Stochastic Programming Tags , , , , ,

    A wide array of machine learning problems are formulated as the minimization of the expectation of a convex loss function on some parameter space. Since the probability distribution of the data of interest is usually unknown, it is is often estimated from training sets, which may lead to poor out-of-sample performance. In this work, we … Read more

    Learning to Project in Multi-Objective Binary Linear Programming

  • Hadi Charkhgard
  • Iman Dayarian
  • Ali Eshragh
  • Sorna Javadi
  • Alvaro Sierra-Altamiranda
    • Categories (Mixed) Integer Linear Programming, Multi-Criteria Optimization, Other Topics Tags , , , ,

    In this paper, we investigate the possibility of improving the performance of multi-objective optimization solution approaches using machine learning techniques. Specifically, we focus on multi-objective binary linear programs and employ one of the most effective and recently developed criterion space search algorithms, the so-called KSA, during our study. This algorithm computes all nondominated points of … Read more

    Machine learning approach to chance-constrained problems: An algorithm based on the stochastic gradient descent

  • Lukáš Adam
  • Martin Branda
    • Categories Stochastic Programming Tags , , , , ,

    We consider chance-constrained problems with discrete random distribution. We aim for problems with a large number of scenarios. We propose a novel method based on the stochastic gradient descent method which performs updates of the decision variable based only on looking at a few scenarios. We modify it to handle the non-separable objective. A complexity … Read more

    Adaptive regularization algorithms with inexact evaluations for nonconvex optimization

  • Stefania Bellavia
  • Benedetta Morini
  • Philippe L. Toint
  • Gianmarco Gurioli
    • Categories Constrained Nonlinear Optimization, Data-Mining, Unconstrained Optimization Tags , , , ,

    A regularization algorithm using inexact function values and inexact derivatives is proposed and its evaluation complexity analyzed. This algorithm is applicable to unconstrained problems and to problems with inexpensive constraints (that is constraints whose evaluation and enforcement has negligible cost) under the assumption that the derivative of highest degree is beta-H\”{o}lder continuous. It features a … Read more

    Variational analysis perspective on linear convergence of some first order methods for nonsmooth convex optimization problems

  • Jane J. Ye
  • Xiao-Ming Yuan
  • Shangzhi Zeng
  • Jin Zhang
    • Categories Convex and Nonsmooth Optimization Tags , , , , , , , ,

    We understand linear convergence of some first-order methods such as the proximal gradient method (PGM), the proximal alternating linearized minimization (PALM) algorithm and the randomized block coordinate proximal gradient method (R-BCPGM) for minimizing the sum of a smooth convex function and a nonsmooth convex function from a variational analysis perspective. We introduce a new analytic … Read more

    POLO: a POLicy-based Optimization library

  • Arda Aytekin
  • Martin Biel
  • Mikael Johansson
    • Categories Convex and Nonsmooth Optimization, Optimization Software Design Principles, Parallel Algorithms Tags , , , ,

    We present POLO — a C++ library for large-scale parallel optimization research that emphasizes ease-of-use, flexibility and efficiency in algorithm design. It uses multiple inheritance and template programming to decompose algorithms into essential policies and facilitate code reuse. With its clear separation between algorithm and execution policies, it provides researchers with a simple and powerful … Read more