Deep Neural Network Structures Solving Variational Inequalities

Published: 2018/09/05

Complementarity and Variational Inequalities, Convex Optimization averaged operators, fixed point algorithms, neural networks, non linear systems, proximity operator Short URL: https://optimization-online.org/?p=15377

We propose a novel theoretical framework to investigate deep neural networks using the formalism of proximal fixed point methods for solving variational inequalities. We first show that almost all activation functions used in neural networks are actually proximity operators. This leads to an algorithmic model alternating firmly nonexpansive and linear operators. We derive new results on averaged operator iterations to establish the convergence of this model, and show that the limit of the resulting algorithm is a solution to a variational inequality. In general, this limiting output does not solve any minimization problem.

Deep Neural Network Structures Solving Variational Inequalities

Citation

Article