Betty Shea – Optimization Online

Why Line Search when you can Plane Search?

Published: 2024/06/25

Network Optimization, Optimization in Data Science

We introduce the class of SO-friendly neural networks, which include several models used in practice including networks with 2 layers of hidden weights where the number of inputs islarger than the number of outputs. SO-friendly networks have the property that performing a precise line search to set the step size on each iteration has the … Read more

Greedy Newton: Newton’s Method with Exact Line Search

Published: 2024/01/10

Betty Shea

Mark Schmidt

Convex Optimization

A defining characteristic of Newton’s method is local superlinear convergence within a neighbourhood of a strict local minimum. However, outside this neighborhood Newton’s method can converge slowly or even diverge. A common approach to dealing with non-convergence is using a step size that is set by an Armijo backtracking line search. With suitable initialization the … Read more