Most humans begin to make sense of the words we hear from the first months of life. Today, any smartphone can translate speech to text in real-time as we rattle off a shopping list. This capability may seem effortless, but teaching computers to recognize human speech has been an enormous computational undertaking decades in the making.
In the early 1990s, ICSI researchers Nelson Morgan and Hervé Bourlard were in the thick of it. Computer scientists had ambitious hopes for teaching computers to process speech, but the technology was far from ready for prime time. The researchers knew that a statistical method known as Hidden Markov Models (HMMs) were good for modeling sequences such as speech, and that Artificial Neural Networks (ANNs), a machine learning technique, were good for learning and expressing nonlinear relationships between variables, such as between speech sounds and speech sound categories. Could the two approaches be combined?
Morgan and Bourlard demonstrated their successful hybrid HMM/ANN technique in a paper recognized with Signal Processing Magazine’s 1996 Best Paper Award. Their solution used ANNs to generate probabilities representing the likelihoods of various speech sounds; this information was then fed into the HMMs to model speech sequences. The team also developed a new learning regimen that improved the ability to generalize from training data to testing data. At the time, their system performed roughly on par with other speech recognition approaches. But years later, as computational capacity caught up to the demands of their approach, their methods were shown to perform significantly better than peer systems of the time. In 2022, the researchers were jointly honored with the IEEE James L. Flanagan Speech and Audio Processing Award for contributions to neural networks for statistical speech recognition.
This story was published in January 2026 as part of a retrospective series highlighting ICSI’s accomplishments and impacts over the years. To learn about our ongoing work, explore our Core Research Themes.