research area Methods and Algorithms, research area Speech and Natural Language Processingconference ECCV
content type paperpublished August 2024

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition

AuthorsAndrew Rouditchenko, Ronan Collobert, Tatiana Likhomanenko

View publication

Audio-visual speech contains synchronized audio and visual information that provides cross-modal supervision to learn representations for both automatic speech recognition (ASR) and visual speech recognition (VSR). We introduce continuous pseudo-labeling for audio-visual speech recognition (AV-CPL), a semi-supervised method to train an audio-visual speech recognition (AVSR) model on a combination of labeled and unlabeled videos with continuously regenerated pseudo-labels. Our models are trained for speech recognition from audio-visual inputs and can perform speech recognition using both audio and visual modalities, or only one modality. Our method uses the same audio-visual model for both supervised training and pseudo-label generation, mitigating the need for external speech recognition models to generate pseudo-labels. AV-CPL obtains significant improvements in VSR performance on the LRS3 dataset while maintaining practical ASR and AVSR performance. Finally, using visual-only speech data, our method is able to leverage unlabeled visual speech to improve VSR.

Figure 1: AV-CPL trains jointly on labeled and unlabeled videos while continuously generating pseudo-labels (PL)s on unlabeled videos. The parameters of the model generating PLs, θT−Δ\theta_{T-\Delta}, are controlled through exponential moving averaging (EMA). Audio-visual inputs are used during PL generation. Modality dropout is used so that the model is trained on audio-visual, video-only, or audio-only inputs.

Related readings and updates.

Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data

May 12, 2023research area Speech and Natural Language Processingconference ACL

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient parallel data resources. We...

Read more

Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Models

May 19, 2019research area Speech and Natural Language Processingconference ICML

Speech-driven visual speech synthesis involves mapping features extracted from acoustic speech to the corresponding lip animation controls for a face model. This mapping can take many forms, but a powerful approach is to use deep neural networks (DNNs). However, a limitation is the lack of synchronized audio, video, and depth data required to reliably train the DNNs, especially for speaker-independent models. In this paper, we investigate...

Read more