Machine learning (ML) offers powerful tools to tackle complex tasks, and it is becoming ever more integral to a wide range of industries and scientific applications, from medical diagnostics to weather forecasting. As the ML ecosystem matures, the end users of ML models are growing increasingly distanced from the developers of those models. In an emerging marketplace of pre-trained “off-the-shelf” ML models, how can buyers identify high-quality models and avoid lemons?
To answer this question, ICSI’s Michael Mahoney, in collaboration with Charles Martin of Calculation Consulting, developed a theoretical basis for evaluating model quality, and introduced a set of metrics that can be used to assess the performance of neural networks—even without access to the data used to train them. In contrast to the traditional trial-and-error approach to gauging the quality of ML algorithms—which relies on access to training data and other information about a model’s development—the team’s solution is based on analyzing the eigenvalues of weight matrices of the final model.
The work, funded in part by the National Science Foundation and U.S. Department of Defense, aims to help improve the utility of ML models, while minimizing pitfalls such as overly compute-hungry processing or hidden racial discrimination. The approach has found particular traction among researchers seeking to refine complex ML models influential in high-stakes areas of urgent societal importance, such as climate prediction and battery design. The researchers also applied the method in a study that exposed key shortcomings of a contest to develop generalization metrics for deep learning algorithms, underscoring the need to go beyond one-size-fits-all metrics for assessing the quality of neural network models.
What made ICSI a good place to pursue these projects?
“One of the real strengths of ICSI is its support for interdisciplinary projects that really break through common assumptions and push the bounds on our knowledge and capabilities. It also enables people to work across sectors and bridge the interests of industry and academia.”
Michael Mahoney
Vice President, Principal Scientist, and Group Lead for AI and Big Data, ICSI
This story was published in January 2026 as part of a retrospective series highlighting ICSI’s accomplishments and impacts over the years. To learn about our ongoing work, explore our Core Research Themes.