hyperparameter optimization

Mathematical programs with complementarity constraints and application to hyperparameter tuning for nonlinear support vector machines

  • Samuel Ward
  • Alain Zemkoho
  • Selin Damla Ahipasaoglu
    • Categories Bilevel Optimization, Complementarity and Variational Inequalities, Optimization in Data Science Tags , , , , ,

    We consider the Mathematical Program with Complementarity Constraints (MPCC). One of the main challenges in solving this problem is the systematic failure of standard Constraint Qualifications (CQs). Carefully accounting for the combinatorial nature of the complementarity constraints, tractable versions of the Mangasarian Fromovitz Constraint Qualification (MFCQ) have been designed and widely studied in the literature. … Read more

    Analyzing Inexact Hypergradients for Bilevel Learning

  • Matthias J. Ehrhardt
  • Lindon Roberts
    • Categories Nonlinear Optimization, Optimization in Data Science Tags , ,

    Estimating hyperparameters has been a long-standing problem in machine learning. We consider the case where the task at hand is modeled as the solution to an optimization problem. Here the exact gradient with respect to the hyperparameters cannot be feasibly computed and approximate strategies are required. We introduce a unified framework for computing hypergradients that … Read more

    HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search

  • Dounia Lakhmiri
  • Sébastien Le Digabel
  • Christophe Tribes
    • Categories Nonlinear Optimization, Optimization Software and Modeling Systems Tags , , , , , ,

    The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time consuming process that is often described as a “dark art”. This explains the necessity … Read more