Smart hearing system reads lips of masked speakers
26 Sep 2022
A hearing aid system under development employs radio frequency sensing paired with artificial intelligence to read lips even when speakers are wearing facemasks.
The new system’s learning algorithms can interpret wifi and radio data from masked sparks with 80 and 83% accuracy respectively, says researchers. In the case of unmasked speakers, the percentages climb to 95% and 91% respectively.
Teams from the University of Glasgow and Edinburgh Napier University the work, alongside academics from Pakistan’s University of Engineering and Technology in Lahore and China’s Southeast University in Nanjing.
Their research, supported by Engineering and Physical Sciences Research Council (EPSRC) funding and titled Pushing the Limits of Remote RF Sensing by Reading Lips Under the Face Mask, appears in Nature Communications.
Traditional hearing aids are limited by the fact they enhance all ambient sounds which make it harder often for users to follow conversations in large groups.
Smart technology can combine hearing aids with a device to collect additional data, such as cameras for better lip reading. However, cameras create issues around privacy and consent and cannot cope with masks or face coverings.
The new system however focuses on radio-frequency data, without video. It used 3,600 samples to inform machine learning and deep learning algorithms to recognise lip and mouth movements associated with vowel sounds developed using male and female volunteers to repeat vowel sounds unmasked and then masked.
Volunteers’ facial movements were scanned radio-frequency signals from a radar sensor and a wifi transmitter, as well as being scanned with lips still.
Then, the 3,600 samples of data collected during the scans was used to ‘teach’ machine learning and deep learning algorithms how to recognise the characteristic lip and mouth movements associated with each vowel sound.
Lead author Dr Qammer Abbasi (pictured), from Glasgow’s James Watt School of Engineering pointed out that an estimated 430 million people, 5% of the global population had a of hearing impairment. New generation technology could play a key role in giving them an enhanced quality of life, he added.
“Given the ubiquity and affordability of wifi technologies, the results are highly encouraging which suggests that this technique has value both as a standalone technology and as a component in future multimodal hearing aids,” stated Abbasi.
