Our lip-reading technology promises to make hearing aids more human

Hearing aids can be lifelines for people with hearing loss. But their limitations can mean that, in particularly noisy environments, users cannot exploit the best of the existing technology. Most new hearing aid designs just make small improvements to microphones, power efficiency and noise filtering. We propose an entirely new approach.

My colleagues and I are working as part of a multi-disciplinary team led by Stirling University, which includes a psychologist and a clinical scientist and is supported by a hearing aid manufacturer. Our aim is to develop an audio-visual hearing aid for the 21st century, taking inspiration from the way that the human body naturally deals with noisy situations, something often known as the cocktail party effect.

Imagine a scenario such as a very busy party with lots of noise, music and people talking. Despite this overwhelming environment, a person with full hearing is often able to pick out and listen to the voice of someone next to them. This is something that people with hearing aids often find extremely challenging. In fact, in really busy environments many deaf people may prefer to remove their hearing aids altogether.

Dealing with noisy environments

The answer to why it is so difficult for hearing aids to deal with these situations is complicated. It’s partly down to the limitations of directional microphones, of inadequate noise cancelling, and of the loss of information about where sound is coming from. But the reason why deaf people can often “hear” better in overwhelming environments like this can be partly explained by lip-reading.

Lip-reading is known to enable individuals with hearing-loss to better understand speech. We all lip-read to a greater or lesser extent, but in people with hearing loss it can become a vital skill. Yet it’s a component of communication that existing hearing aids simply ignore.

No more smiling and nodding.Shutterstock

Our vision is for an ear or body-worn hearing aid linked to a small wearable camera, which could be mounted in a pair of ordinary glasses, jewellery or perhaps even worn as a discreet badge. The device would process the camera’s video stream to isolate relevant information about lip movement.

This data can be used by the hearing aid in several ways. On a simple level, if it knows someone is speaking it could apply some general background noise-reduction filtering. It could identify the direction the voice is coming from and focus a directional microphone accordingly.

Significantly, it could also use the lip movement information to apply an appropriate filter for further noise reduction, just as our brains do naturally. Specifically, if the device can estimate what the speech is likely to be, then it can remove sound elements that don’t match this. For example, if loud music is playing, “reading” the lips of the target speaker would indicate to the device that it should remove this music because it does not match the expected sound.

What are the challenges?

There are multiple challenges to ensuring a hearing aid like this can work practically in the real world, involving the same problems that human lip-readers face. It has to be able to deal with multiple speakers at once and sound that isn’t in front of it. And, generally, people do not simply stand motionless in front of the listener, but instead tend to move, turn their heads, cover their faces or show their emotions visually. They may also be interrupted or have someone else walk in front of them.

To overcome this, our solution will be to again consider how humans function. How much lip-reading we do depends on the circumstances. The more noisy it is, the more we tend to look at people’s lips. So a system that exclusively lip-reads would not be very useful when it comes to real conversations in real environments. We plan to integrate our approach with other non-camera approaches that hearing aids presently use, including noise cancelling and directional microphones.

Our aim is to produce an aesthetically designed system that improves users' ability to understand what someone is saying in a range of environments, potentially with less listening effort. This would help solve the real problems faced by those with hearing loss, including their low uptake of available technology, by delivering a freely available, next-generation hearing device prototype, inspired by the way we naturally think, hear and see.

Amir Hussain receives funding from the EPSRC and Royal Society of Edinburgh.