LEArning neural networks with FLExible nonlinearities by Tensor methods – LeaFleT

Neural networks are a fundamental tool for solving various artificial intelligence tasks, such as supervised and unsupervised classification.
Recent progress is linked to deep neural networks with extremely large number of layers, which helped to achieve remarkable results in the context of many applied tasks for analysis and interpretation of natural images, audio signals and textual data. Despite this success, they still have a number of drawbacks, including lack of interpretability and large number of parameters. In this project, we propose to simplify neural network architectures by allowing flexible nonlinear activation functions, contrary to fixed activation functions typically used. The proposed pathway is based on an original tensor-based technique for decomposition of multivariate maps, developed in the context of nonlinear system identification. The fundamental property of such decompositions is identifiability, which we hope would be transferrable in the deep learning setup. We believe that the identifiability property, by enforcing stability of the representation, could be one of possible ways to define interpretability of neural networks. Also, neural networks will be simplified by reducing the number of layers, and recent advances in tensor computations can be used. The main goal of this project is to develop effective learning tools for the proposed model, together with their publicly available software implementation.