Bimodal Speech Recognition
The goal of bimodal speech recognition is to
combine audio and visual information to enhance speech recognition rate under
poor audio conditions (noise or acoustically confusing words). A lipreading
system recognizes a spoken word based on the input lip motion. To handle this
problem, we proposed a space-time delay neural network that can automatically
discover the features embedded in spatiotemporal domain in the training process
and use these features to classify different lip motions. Our experimental
results indicated that, using only lip motion video, the lipreading system can
achieve a 77.8%~90% recognition rate for Chinese digits, and 44.7%~48.9%
recognition rate for nineteen Chinese confusing words.
We also implemented an on-line bimodal speech
recognition system to test how lipreading can improve the audio speech
recognition. The recognition system consisting of three DSP processors and one
Pentium processor concurrently processes lip motion video and speech signals.
The whole recognition process, including mouth region centering, 2D-FFT, speech
feature extraction, neural network computation, HMM computation, and decision
fusion, can be executed in real time.
Chin-Teng Lin, Hsi-Wen Nein and Wen-Chieh Lin, “A
Space-Time Delay Neural Network for Motion Recognition and Its Application to
Lipreading, ” International Journal of Neural Systems, Vol. 9, No. 4, Aug
1999, pp. 311-334.
Wen-Chieh Lin, A Space-Time Delay Neural
Network for Motion Recognition and Its Application to Lipreading in Bimodal
Speech Recognition, Master thesis, National Chiao-Tung University, Taiwan,
Wen-Chieh Lin, Hsi-Wen Nein, and Shin-Hui Liang, A DSP-based On-line Bimodal Speech Recognition System,
First Prize of the Graduate Student Team in the Texas Instrument DSP Design
Challenge, Taiwan, 1996.