Linear Algebra for Neural Networks

Enhancing neural networks for broader applications

Deep learning methods have been transformative in some areas but faced limitations in others. This project, titled “Scalable Linear Algebra and Neural Network Theory,” aims to extend neural networks to a broader range of applications by more effectively leveraging linear algebra.

Among machine learning applications, deep learning methods have been most impactful for computer vision and natural language processing (NLP). In these contexts, researchers have generally started with large quantities of data, trained a neural network model using stochastic first order methods, and then implemented and applied the model in a user-facing industrial application. However, this approach requires extremely large quantities of data and compute resources, and can also raise issues around interpretability and robustness.

With this project, we aim to overcome these limitations with approaches that provide stronger control over the complementary algorithmic and statistical basis of linear algebraic methods.  While linear algebraic techniques are central to the design and use of modern neural network models, current methodology uses linear algebra in relatively superficial ways, e.g., matrix multiplication for finding stochastic gradient directions and backpropagating errors in a scalable manner. By addressing this central technical issue that underlies many of the limitations of current neural networks, we can close the gap between theory and practice to extend neural network applications beyond computer vision and NLP.

Project Team

About