Andrea Walther – Optimization Online

Stochastic Aspects of Dynamical Low-Rank Approximation in the Context of Machine Learning

Published: 2024/03/23, Updated: 2024/05/15

Data Science Theory, Nonlinear Optimization, Optimization in Data Science deep neural networks, Dynamical Low-Rank Approximation (DLRA), Dynamical Low-Rank Training13 (DLRT), machine learning, stochastic gradient descent

The central challenges of today’s neural network architectures are the prohibitive memory footprint and the training costs associated with determining optimal weights and biases. A large portion of research in machine learning is therefore dedicated to constructing memory-efficient training methods. One promising approach is dynamical low-rank training (DLRT) which represents and trains parameters as a … Read more

On a Frank-Wolfe Approach for Abs-smooth Functions

Published: 2023/03/20, Updated: 2023/07/05

Constrained Nonlinear Optimization, Nonsmooth Optimization, Other Topics abs-smooth functions, Active Signature Method, convergence rate, Frank-Wolfe algorithm, nonsmooth optimization

We propose an algorithm which appears to be the first bridge between the fields of conditional gradient methods and abs-smooth optimization. Our problem setting is motivated by various applications that lead to nonsmoothness, such as $\ell_1$ regularization, phase retrieval problems, or ReLU activation in machine learning. To handle the nonsmoothness in our problem, we propose … Read more

A Semismooth Conjugate Gradients Method — Theoretical Analysis

Published: 2023/01/17

Franz Bethke

Andreas Griewank

Andrea Walther

Nonsmooth Optimization conjugate gradient method, generalized gradient, semismoothness, Shortest residual update

In large scale applications, deterministic and stochastic variants of Cauchy’s steepest descent method are widely used for the minimization of objectives that are only piecewise smooth. In this paper we analyse a deterministic descent method based on the generalization of rescaled conjugate gradients proposed by Philip Wolfe in 1975 for objectives that are convex. Without … Read more

An active signature method for constrained abs-linear minimization

Published: 2021/12/02, Updated: 2023/05/02

Timo Kreimeier

Andrea Walther

Andreas Griewank

Nonsmooth Optimization abs-linear form, active signature method (ASM), constrained optimization, linear independence kink qualification (likq), optimality conditions, piecewise linear

In this paper we consider the solution of optimization tasks with a piecewise linear objective function and piecewise linear constraints. First, we state optimality conditions for that class of problems using the abs-linearization approach and prove that they can be verified in polynomial time. Subsequently, we propose an algorithm called Constrained Active Signature Method that … Read more

A Structure Exploiting Algorithm for Non-Smooth Semi-Linear Elliptic Optimal Control Problems

Published: 2020/12/14, Updated: 2021/09/30

Stephan Schmidt

Olga Weiß

Andrea Walther

Infinite Dimensional Optimization, Nonsmooth Optimization

We investigate optimization problems with a non-smooth partial differential equation as constraint, where the non-smoothness is assumed to be caused by Nemytzkii operators generated by the functions abs, min and max. For the efficient as well as robust solution of such problems, we propose a new optimization method based on abs-linearization, i.e., a special handling … Read more

An Algorithm for Piecewise Linear Optimization of Objective Functions in Abs-normal Form

Published: 2017/12/22

Andreas Griewank

Andrea Walther

Nonsmooth Optimization abs-normal form, active set and signature, karush kuhn tucker (kkt), linear independence kink qualification (likq), normal growth, quadratic regularization, successive piecewise linear optimization (splop), tangential stationarity

In the paper [11] we derived first order (KKT) and second order (SSC) optimality conditions for functions defined by evaluation programs involving smooth elementals and absolute values. For this class of problems we showed in [12] that the natural algorithm of successive piecewise linear optimization with a proximal term (SPLOP) achieves a linear or even … Read more

Relaxing kink qualifications and proving convergence rates in piecewise smooth optimization

Published: 2017/11/17

Andreas Griewank

Andrea Walther

Nonsmooth Optimization abs-normal form, karush kuhn tucker (kkt), linear convergence, linear independence kink qualification (likq), quadratic convergence, second order sufficiency condition (ssc), sharp minimizer, strict complementary

Abstract. In the paper [9] we derived first order (KKT) and second order (SSC) optimality conditions for functions defined by evaluation programs involving smooth elementals and absolute values. In that analysis, a key assumption on the local piecewise linearization was the Linear Independence Kink Qualification (LIKQ), a generalization of the Linear Independence Constraint Qualification (LICQ) … Read more

Characterizing and testing subdifferential regularity for piecewise smooth objective functions

Published: 2017/11/05

Andreas Griewank

Andrea Walther

Nonsmooth Optimization abs-normal form, clarke gradient, first-order-convexity, linear-independence-kink-qualification, mangasarin-fromovitz-kink-qualification, mordukhovich gradient, subdifferential regularity

Functions defined by evaluation programs involving smooth elementals and absolute values as well as the max- and min-operator are piecewise smooth. Using piecewise linearization we derived in [7] for this class of nonsmooth functions first and second order conditions for local optimality (MIN). They are necessary and sufficient, eespectively. These generalizations of the classical KKT … Read more

Automatic Differentiation of the Open CASCADE Technology CAD System and its coupling with an Adjoint CFD Solver

Published: 2017/03/31

Optimization of Systems modeled by PDEs, Unconstrained Optimization adjoint cfd method, automatic differentiation, cad system, gradient-based optimisation

Automatic Differentiation (AD) is applied to the open-source CAD system Open CASCADE Technology using the AD software tool ADOL-C (Automatic Differentiation by OverLoading in C++). The differentiated CAD system is coupled with a discrete adjoint CFD solver, thus providing the first example of a complete differentiated design chain built from generic, multi-purpose tools. The design … Read more

Algorithmic Differentiation for Piecewise Smooth Functions: A Case Study for Robust Optimization

Published: 2017/02/28, Updated: 2017/03/07

Nonsmooth Optimization algorithmic differentiation, nonsmooth optimization, piecewise linearization, robust optimization

This paper presents a minimization method for Lipschitz continuous, piecewise smooth objective functions based on algorithmic differentiation (AD). We assume that all nondifferentiabilities are caused by abs(), min(), and max(). The optimization method generates successively piecewise linearizations in abs-normal form and solves these local subproblems by exploiting the resulting kink structure. Both, the generation of … Read more