Journal paper on audio-visual speech enhancement with posterior sampling methods in recurrent variational autoencoders

Journal paper in Computer Speech & Language:

Robust Audio–Visual Speech Enhancement under Visual Occlusion using TITR–CNN and Silence-Aware MALA–EM.

Zohre Foroushi and Prof. Richard Dansereau present a new approach to improving speech quality in noisy environments using both audio and video information. Their work focuses on situations where the speaker’s mouth may be partially blocked or difficult to see, and where background noise is strong.

The proposed method reconstructs missing visual information from video, makes better use of quiet segments in the audio to reduce unwanted artifacts, and applies improved statistical techniques to produce clearer speech. Experiments show strong improvements in speech quality and intelligibility, even under challenging conditions.