Fine-tuning pre-trained Automatic Speech Recognition (ASR) models with synthesized datasets leads to suboptimal performance in low-resource ASR, due to domain mismatches between true and synthesized speech data. To address this issue, we propose a novel score-distribution-matching method for data selection, which filters synthesized data by explicitly aligning its score distribution with the prior distribution of the true data score. We apply our proposed method on the ASR of Te Reo Māori (the Māori langauge), an official yet endangered language of New Zealand. We legitimately collect and label 27 hours of Te Reo Māori speech data and synthesize an additional 520 hours using our recently developed Zero-Voice model, a zero-shot text-to-speech (TTS) system. By finetuning the pre-trained ASR model Whisper-large-v3 using both true and filtered synthesized data selected by our proposed approach, we achieve a remarkable improvement in accuracy for Te Reo Māori ASR that outperforms baseline models on the FLEURS Māori (mi nz) benchmark.
Personalized speech synthesis has attracted increasing attention in the field of robotics. Compared to traditional speech synthesis, it faces two primary challenges: the limited availability of adaptation data and the necessity for highly efficient adaptation method using compact parameters to reduce both training time and memory consumption. To address both the challenges, this paper proposes a personalized speech synthesis approach that incorporates an Empirical Mode Decomposition (EMD)-based speaker modeling method alongside a novel decoder structure with masked inputs, which improves the model’s ability to extract speaker-specific features accurately. Furthermore, we introduce a parameter-efficient fine-tuning technique, Attention-based Speaker-Text Scaling and Shifting Feature (AST-SSF), to enhance adaptation efficiency. We validate our approach using the MAGICDATA Corpus. The results indicate that our proposed approach outperforms the baseline in both naturalness and similarity, demonstrating its effectiveness. Moreover, although the proposed adaptation method substantially reduces the number of parameters, it exhibits only minimal performance degradation compared to full and partial fine-tuning strategies.
In this paper, we propose a self-training approach for automatic speech recognition (ASR) for low-resource settings. While self-training approaches have been extensively developed and evaluated for high-resource languages such as English, their applications to low-resource languages like Punjabi have been limited, despite the language being spoken by millions globally. The scarcity of annotated data has hindered the development of accurate ASR systems, especially for low-resource languages (e.g., Punjabi and Māori languages). To address this issue, we propose an effective self-training approach that generates highly accurate pseudo-labels for unlabeled low-resource speech. Our experimental analysis demonstrates that our approach significantly improves word error rate, achieving a relative improvement of 14.94% compared to a baseline model across four real speech datasets. Further, our proposed approach reports the best results on the Common Voice Punjabi dataset.
In this paper, we propose a novel Continual Learning approach, which is Randomly Layer-wise Tuning (CLRL-Tuning) of a pre-trained Automatic Speech Recognition (ASR) model. CLRL-Tuning tackles the randomness of subsequent datasets by updating the parameters of randomly selected encoder layers of the pre-trained model (such as wav2vec 2.0) for every training epoch. CLRL-Tuning is different from the previous approaches in that it neither uses previous datasets, nor expands/runs previous models. Furthermore, we perform experiments to evaluate our approach compared with four strong baselines, including Knowledge Distillation and Gradient Episodic Memory. Our approach achieves significant improvements over the baselines in average word error rate (WER) for the wav2vec 2.0 model. Additionally, we implement ablation studies for our approach by tuning one, three, six and full encoder layers of the model, and experimental results show only tuning one encoder layer of the model at each training epoch is the most effective way to mitigate catastrophic forgetting.
We propose a new meta learning based framework for low resource speech recognition that improves the previous model agnostic meta learning (MAML) approach. The MAML is a simple yet powerful meta learning approach. However, the MAML presents some core deficiencies such as training instabilities and slower convergence speed. To address these issues, we adopt multi-step loss (MSL). The MSL aims to calculate losses at every step of the inner loop of MAML and then combines them with a weighted importance vector. The importance vector ensures that the loss at the last step has more importance than the previous steps. Our empirical evaluation shows that MSL significantly improves the stability of the training procedure and it thus also improves the accuracy of the overall system. Our proposed system outperforms MAML based low resource ASR system on various languages in terms of character error rates and stable training behavior.
Recent speech data augmentation approaches use static augmentation operations or policies with consistency magnitude scaling. However, few work is done to explore the influence of the dynamic magnitude of augmentation policies. In this paper, we propose a novel speech data augmentation approach, CyclicAugment, to generate more diversified augmentation policies by dynamically configuring the magnitude of augmentation policies with a cosine annealing scheduler. We also propose additional augmentation operations to enlarge the diversity of augmentation policies. Motivated by learning rate warm restart and cyclical learning rates, we hypothesize that using dynamically configured magnitude for augmentation policies can also help escape local optima more efficiently than static augmentation policies with consistency magnitude scaling. Experimental results demonstrate that our approach is effective for escaping local optima. Our approach achieves 12%-35% relative improvement in word error rate (WER) over SpecAugment and RandAugment on the LibriSpeech 960h dataset, and achieves state-of-the-art result 7.1% in phoneme error rate (PER) on the TIMIT 5h dataset.
Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims
to identify speakers' smoking status from their speech. In the COVID-19 pandemic, speech-based
automatic smoker identification approaches have received more attention in smoking cessation
research due to low cost and contactless sample collection. This study focuses on determining the
best acoustic features for smoker identification. In this paper, we investigate the performance of
four acoustic feature sets/representations extracted using three feature extraction/learning
approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic
Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words
representations, (iii) the neural representations extracted from raw waveform signals by SincNet.
Experimental results show that: (i) SincNet feature representations are the most effective for
smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii)
the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on
the scale of the dimensions of feature vectors.
Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims
to identify speakers' smoking status from their speech. In the COVID-19 pandemic, speech-based
automatic smoker identification approaches have received more attention in smoking cessation
research due to low cost and contactless sample collection. This study focuses on determining the
best acoustic features for smoker identification. In this paper, we investigate the performance of
four acoustic feature sets/representations extracted using three feature extraction/learning
approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic
Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words
representations, (iii) the neural representations extracted from raw waveform signals by SincNet.
Experimental results show that: (i) SincNet feature representations are the most effective for
smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii)
the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on
the scale of the dimensions of feature vectors.
Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims
to identify speakers' smoking status from their speech. In the COVID-19 pandemic, speech-based
automatic smoker identification approaches have received more attention in smoking cessation
research due to low cost and contactless sample collection. This study focuses on determining the
best acoustic features for smoker identification. In this paper, we investigate the performance of
four acoustic feature sets/representations extracted using three feature extraction/learning
approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic
Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words
representations, (iii) the neural representations extracted from raw waveform signals by SincNet.
Experimental results show that: (i) SincNet feature representations are the most effective for
smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii)
the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on
the scale of the dimensions of feature vectors.
Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims
to identify speakers' smoking status from their speech. In the COVID-19 pandemic, speech-based
automatic smoker identification approaches have received more attention in smoking cessation
research due to low cost and contactless sample collection. This study focuses on determining the
best acoustic features for smoker identification. In this paper, we investigate the performance of
four acoustic feature sets/representations extracted using three feature extraction/learning
approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic
Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words
representations, (iii) the neural representations extracted from raw waveform signals by SincNet.
Experimental results show that: (i) SincNet feature representations are the most effective for
smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii)
the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on
the scale of the dimensions of feature vectors.
Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims
to identify speakers' smoking status from their speech. In the COVID-19 pandemic, speech-based
automatic smoker identification approaches have received more attention in smoking cessation
research due to low cost and contactless sample collection. This study focuses on determining the
best acoustic features for smoker identification. In this paper, we investigate the performance of
four acoustic feature sets/representations extracted using three feature extraction/learning
approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic
Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words
representations, (iii) the neural representations extracted from raw waveform signals by SincNet.
Experimental results show that: (i) SincNet feature representations are the most effective for
smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii)
the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on
the scale of the dimensions of feature vectors.
Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims
to identify speakers' smoking status from their speech. In the COVID-19 pandemic, speech-based
automatic smoker identification approaches have received more attention in smoking cessation
research due to low cost and contactless sample collection. This study focuses on determining the
best acoustic features for smoker identification. In this paper, we investigate the performance of
four acoustic feature sets/representations extracted using three feature extraction/learning
approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic
Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words
representations, (iii) the neural representations extracted from raw waveform signals by SincNet.
Experimental results show that: (i) SincNet feature representations are the most effective for
smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii)
the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on
the scale of the dimensions of feature vectors.
For speech enhancement, deep complex network based methods have shown promising performance due to their effectiveness in dealing with complex-valued spectrums. Recent speech enhancement methods focus on further optimization of network structures and hyperparameters, however, ignore inherent speech characteristics (e.g., phonetic characteristics), which are important for networks to learn and reconstruct speech information. In this paper, we propose a novel self-supervised learning based phone-fortified (SSPF) method for speech enhancement. Our method explicitly imports phonetic characteristics into a deep complex convolutional network via a Contrastive Predictive Coding (CPC) model pre-trained with self-supervised learning. This operation can greatly improve speech representation learning and speech enhancement performance. Moreover, we also apply the self-attention mechanism to our model for learning long-range dependencies of a speech sequence, which further improves the performance of speech enhancement. The experimental results demonstrate that our SSPF method outperforms existing methods and achieves state-of-the-art performance in terms of speech quality and intelligibility.
Authors
Discription
Type
Action
Yuan Gao, Feng Hou*, Huia Jahnke, Ruili Wang
Data Augmentation with Diversified Rephrasing for Low-Resource Neural Machine Translation