hyperparameter optimization

Analyzing Inexact Hypergradients for Bilevel Learning

  • Matthias J. Ehrhardt
  • Lindon Roberts
    • Categories Nonlinear Optimization, Optimization in Data Science Tags , ,

    Estimating hyperparameters has been a long-standing problem in machine learning. We consider the case where the task at hand is modeled as the solution to an optimization problem. Here the exact gradient with respect to the hyperparameters cannot be feasibly computed and approximate strategies are required. We introduce a unified framework for computing hypergradients that … Read more

    HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search

  • Dounia Lakhmiri
  • Sébastien Le Digabel
  • Christophe Tribes
    • Categories Nonlinear Optimization, Optimization Software and Modeling Systems Tags , , , , , ,

    The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time consuming process that is often described as a “dark art”. This explains the necessity … Read more