machine learning

Second-order optimality and beyond: characterization and evaluation complexity in convexly-constrained nonlinear optimization

  • Coralia Cartis
  • Nicholas I. M. Gould
  • Philippe L. Toint
    • Categories Bound-constrained Optimization, Constrained Nonlinear Optimization, Unconstrained Optimization Tags , , ,

    High-order optimality conditions for convexly-constrained nonlinear optimization problems are analyzed. A corresponding (expensive) measure of criticality for arbitrary order is proposed and extended to define high-order $\epsilon$-approximate critical points. This new measure is then used within a conceptual trust-region algorithm to show that, if derivatives of the objective function up to order $q \geq 1$ … Read more

    Convex Variational Formulations for Learning Problems

  • Pedro Borges de Melo
    • Categories Data-Mining Tags , , , , , , ,

    Abstract—In this article, we introduce new techniques to solve the nonlinear regression problem and the nonlinear classification problem. Our benchmarks suggest that our method for regression is significantly more effective when compared to classical methods and our method for classification is competitive. Our list of classical methods includes least squares, random forests, decision trees, boosted … Read more

    Optimization Methods for Large-Scale Machine Learning

  • Léon Bottou
  • Frank E. Curtis
  • Jorge Nocedal
    • Categories Convex and Nonsmooth Optimization, Nonlinear Optimization, Stochastic Programming Tags , , , , ,

    This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of … Read more

    A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimation

  • Émilie Chouzenoux
  • Jean-Christophe Pesquet
    • Categories Convex Optimization, Unconstrained Optimization Tags , , , , , , , , , , ,

    Stochastic approximation techniques play an important role in solving many problems encountered in machine learning or adaptive signal processing. In these contexts, the statistics of the data are often unknown a priori or their direct computation is too intensive, and they have thus to be estimated online from the observed signals. For batch optimization of … Read more

    On the Convergence of Multi-Block Alternating Direction Method of Multipliers and Block Coordinate Descent Method

  • Caihua Chen
  • Xin Liu
  • Yinyu Ye
  • Min Li
    • Categories Convex Optimization, Nonsmooth Optimization Tags , , , , ,

    The paper answers several open questions of the alternating direction method of multipliers (ADMM) and the block coordinate descent (BCD) method that are now wildly used to solve large scale convex optimization problems in many fields. For ADMM, it is still lack of theoretical understanding of the algorithm when the objective function is not separable … Read more

    From Predictive to Prescriptive Analytics

  • Dimitris Bertsimas
  • Nathan Kallus
    • Categories Applications - OR and Management Sciences Tags , , , , ,

    In this paper, we combine ideas from machine learning (ML) and operations research and management science (OR/MS) in developing a framework, along with specific methods, for using data to prescribe optimal decisions in OR/MS problems. In a departure from other work on data-driven optimization and reflecting our practical experience with the data available in applications … Read more

    Machine Learning and Portfolio Optimization

  • Gah-Yi Ban
  • Noureddine El Karoui
  • Andrew E.B. Lim
    • Categories Applications - OR and Management Sciences, Finance and Economics Tags , , , ,

    The portfolio optimization model has limited impact in practice due to estimation issues when applied with real data. To address this, we adapt two machine learning methods, regularization and cross-validation, for portfolio optimization. First, we introduce performance-based regularization (PBR), where the idea is to constrain the sample variances of the estimated portfolio risk and return, … Read more

    Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems

  • Nikos Komodakis
  • Jean-Christophe Pesquet
    • Categories Applications - Science and Engineering, Convex Optimization, Integer Programming Tags , , , , , , , ,

    Optimization methods are at the core of many problems in signal/image processing, computer vision, and machine learning. For a long time, it has been recognized that looking at the dual of an optimization problem may drastically simplify its solution. Deriving efficient strategies which jointly brings into play the primal and the dual problems is however … Read more

    A Stochastic Quasi-Newton Method for Large-Scale Optimization

  • Richard H. Byrd
  • Samantha Hansen
  • Jorge Nocedal
  • Yoram Singer
    • Categories Data-Mining, Stochastic Programming, Unconstrained Optimization Tags , , ,

    Abstract The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper, we propose a stochastic quasi-Newton method that is efficient, robust … Read more

    Incremental Accelerated Gradient Methods for SVM Classification: Study of the Constrained Approach

  • Nicolas Couellan
  • Sophie Jan
    • Categories Constrained Nonlinear Optimization, Data-Mining Tags , , , , , ,

    We investigate constrained first order techniques for training Support Vector Machines (SVM) for online classification tasks. The methods exploit the structure of the SVM training problem and combine ideas of incremental gradient technique, gradient acceleration and successive simple calculations of Lagrange multipliers. Both primal and dual formulations are studied and compared. Experiments show that the … Read more