Building Connections With ASR Models and How we Can do Better

Faculty Sponsor: Pavel Oleinikov

Image of the Author, Jeramiah Trout
Jeramiah Trout

I am from Houston, Texas and a rising Junior in the class of 2026. My degree field is Computer Science with a focus on programming and optimization. Outside of work this summer, I watched every episode of every series of Star Trek at double speed and played . Working with the QAC has allowed me entry into a world in which I typically don’t associate with, and it has opened my eyes to the wonders of statistics and collaboration between my department and the QAC. My summer project fits neatly into my field of study enabling me to understand the fallibility of NER, the strengths of knowledge graphs, and difficulties in using Automatic Speech Recognition.

Abstract: My summer project with the QAC involved the use of 2,500 political ads run in New York State through Meta’s Facebook Advertising Network. The videos were run through 2 ASR models, allowing me to see what each program excels and fails at when compared to each other. All resulting data was run through a name and entity recognition program to flag any organizations or people picked up by the ASR models. Once completed, we wanted to see how accurate the ASR was by checking them against each other and manually going though some of the advertisements to find potential errors. A part of this process was validating the names and entities that were given by the NER program to test for accuracy, something we discovered as a large issue in the ASR software. The results are clear, the ASR software has gotten better over time, best shown through the more recent model performing better than the older model, but neither is perfect and they don’t need to be. The imperfections show what the biggest weaknesses are and the problems we need to overcome to reach the end goal. The main focus of this project was to act as a launchpad for further research into potential solutions or ways to minimize the effect of these errors.