Automating Algorithm Optimization

Creating new classes of self-optimizing algorithms

Fine-tuning the hyperparameters for machine learning algorithms currently requires extensive experimentation and expertise. This project, titled “Reinventing Numerical Algorithms for Linear Algebra and Optimization in Modern Compute Environments,” aims to automate this process and create new classes of self-optimizing algorithms.

With a focus on algorithms that are valuable for applications of interest to the U.S. Department of Defense, the project will lay the foundations to revolutionize the fields of randomized numerical linear algebra (RandNLA) and large-scale numerical optimization. To accomplish this, we are applying automatic algorithmic discovery through reinforcement learning and automatic differentiation to automate the process of randomized sketch design, including the fine-tuning of hyperparameters such as sketch size. This will allow the automatic design of novel numerical optimization algorithms capable of adapting to the most challenging optimization landscapes, while having the ability to self-tune in terms of learning rates, acceleration parameters, and batch sizes.

Through this work, we aim to pioneer new classes of adaptive numerical algorithms that self-optimize based on the computational resources and compute characteristics they encounter such as memory structure, GPU versus CPU, and others. Our approach focuses on developing robust numerical algorithms that can dynamically adjust their precision levels during computation to optimize performance without sacrificing accuracy or stability, not only streamlining the discovery and tuning process for modern numerical algorithms but also prioritizing interpretability.

Project Team

About