Meta-ASR: Automated Speech Recognition using Meta-Learning

Automatic Speech Recognition ASR is a significant technology that enables machines to transcript audio into text. It is found in various applications across many fields, such as security, healthcare, finance and etc. Also, it has been proven to be a valuable tool for saving time and resources when extracting and analyzing human speech data. To create an ASR system, users usually select ASR deep learning pre-trained models randomly from the available options without considering any specific requirements or characteristics of their data. To address this issue, we propose to integrate meta-learning into ASR systems.
This integration aims to automate the selection process of the most suitable pre-trained ASR model based on input audio features.

Automated Speech Recognition ASR

Automated speech recognition (ASR) is a field of AI that allows machines to recognize and convert human speech into text automatically [1].

Why ASR?

Today ASR technology has become a demanded tool for various applications such as virtual assistants and speech-to-text software. In other words, ASR helps humans to interact easily with machines.
This, in consequence, helps humans to share more information and data by saving a lot of time to transcribe the speech into text. Overall, the ASR supports efficiency, accessibility and improves user experience in a variety of applications [2].

ASR and Security

We can use ASR technology in the context of security. Hence, we can use ASR to detect abusive language [3] since it enables us to analyze the audio data in real-time.

ASR existing solutions

Several solutions are widely available in the market. These solutions are presented in cloud-based formats such as Microsoft Azure and Google Cloud [4] or in software-based formats like Facebook wav2letter [5], CMU Sphinxr, and Mozilla DeepSpeech [6].

ASR Pipeline

To create a general ASR pipeline [7], we need to follow some steps, as shown in Fig.1.

Fig.1: ASR General Pipeline

1. After converting the input audio into a digital signal, we start with audio preprocessing [8], which consists of removing noise, normalization, and resampling.
2. The second step is feature extraction from the preprocessed signal audio. Commonly used features [9] include Mel Frequency Cepstral Coefficients (MFCCs), filterbanks, and spectrograms. These features effectively describe and represent the input signal audio.
3. The third step is passing these features by an Acoustic [10] model such as HMM, or RNN. The main role of the Acoustic model is to map the extracted features into linguistic units. So, the Acoustic model predicts sequences of words, and to create a logical relationship between these words, it needs to use language models [11] such as 3N-gram models, CTC…
4. The latest step in this pipeline is adding Post-processing techniques that correct the output text (such as error correction, punctuation insertion, or language-specific adjustments) and then evaluate the ASR system with word error rate (WER) [12] to test the performance on a test data set of audios.

This pipeline has been enhanced through the utilization of powerful pre-trained deep learning models such as Wav2Vec [13], QuartzNet [14], Speech recognition [15], and others. These advanced models effectively integrate the Acoustic Model and the Language Model, enabling the conversion of resampled signal audio (resampled to 16,000 Hz) into an accurate and coherent text (Fig.2).

Fig.2: ASR with Deep Learning

Meta-ASR Pipeline

These pre-trained models have some limitations. One limitation is that pre-trained models may not perform well on data that is significantly different from the data they were trained on. As we know, these Pre-trained models are trained on specific datasets to define biases present, which can lead to errors and inaccuracies in the recognition of speech. In most cases, when we create an ASR system, we randomly choose one of these pre-trained models. Therefore, in this context, we decide to integrate the concept of the Meta-Learning [16]. Meta-Learning is a subfield of AI used to learn new skills or adapt to various environments by observing how a learning algorithm performs learning and then learns from metadata that describes previous learning experiences to do new tasks much faster. Thus, we apply \textit{Meta-Learning} on the head of these pre-trained models to automate and perform the task of selecting one of the pre-trained models based on the meta-features designed to describe the input audio data such as MFCC, Spectral Bandwidth, Spectral Centroid, Sample rate, etc.

As Fig.3 shows the Meta-ASR framework. Also, this framework can transcript the audio of the input videos and audio into text. This pipeline is composed of many steps. Initially, the input (video or audio.mp3) is converted to the audio.wav format. Following that, the audio is divided into smaller chunks, each spanning a duration of 10ms. These chunks undergo essential audio preprocessing, including noise removal and signal normalization. This crucial preprocessing step ensures the audio is appropriately prepared for the subsequent meta-feature extraction process. The meta-feature extraction process involves capturing the statistical distribution of the most significant features from the signal audio, as outlined in Fig.4. These extracted meta-features provide descriptive information about each audio chunk. Additionally, a meta-model is employed to predict the name of the pre-trained models used to transcribe the audio into text. This comprehensive pipeline, consisting of audio format conversion, chunk segmentation, audio preprocessing, meta-feature extraction, and meta-model utilization, facilitates accurate and efficient transcription of the audio data.

Fig.3: Meta-ASR Pipeline

Fig.4: Audio Features

The meta-model construction follows these three main phases:

1. Meta-data creation

2. Meta modeling

3. Meta-Model evaluation

Metadata Creation

To create the Meta-data, we followed the Processes which is presented in Fig.5:

Fig.5. Meta preprocessed data Creation

1. Audio Data Collection:

To collect audio data, We utilize YouTube to collect our audio data by extracting audio from videos and splitting them into smaller chunks, which are then saved in separate folders.

Get Ons Saadallah’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

2. Meta-feature Extraction

We carefully selected the most significant single audio features, which are presented in Fig.4. However, since these features are in array format and cannot be directly used as independent variables, we calculate distributional statistical features (such as minimum, maximum, median, mean, q1, and q3) from these features.

We generate two types of meta-data:

a. Raw Meta-data: This metadata involves extracting the meta-features from each audio chunk without applying any signal audio pre-processing techniques.
b. Pre-processed Meta-data: This metadata involves
extracting the meta-features from each audio chunk after applying signal audio pre-processing techniques, such as noise reduction and audio normalization.
By generating these two types of metadata, we gain insights into the impact of signal audio pre-processing on the extracted meta-features.
This allows us to assess the importance and effectiveness of the pre-processing techniques in enhancing the quality and relevance of the meta-features.


3. Clustering Process

We used the K-means to cluster the audio based on their meta-features.To determine the appropriate number of clusters, we employed two methods: the elbow method and the silhouette score.
In our case, we performed clustering separately on the Raw Meta-data and Pre-processed Meta-data. The results of these clustering processes are illustrated in Figures 6 and 7.

Fig.6. Result of the Clustering of the Raw Meta-data

Fig.7. Result of the Clustering of the Pre-Processing Meta-data

Based on our analysis of the data visualization clusters, we have observed that the pre-processed meta-data yields superior results. The utilization of pre-processing techniques on the meta-data has enhanced the quality of our clustering in which we can see the different audio clusters.

4. Labeling

To label the two meta-data, we first need manual annotation, which involves extracting reference audio from each cluster and manually transcribing these samples. This step enables us to create a ground truth dataset. We transcript these reference audios by the selected candidate pre-trained ASR models, namely, QuartzNet15x5BaseEn’, ’wav2vec2-large-960h-lv60-self’,wav2vec2-large-50-self, and the Speech Recognition of the Google engine. We compare the transcriptions generated by these ASR models and select the one with the lowest word error rate (WER) as the best pre-trained model for each audio sample.
Once we have obtained the ground truth data labeled with the selected ASR pre-trained models, we can proceed to merge this information with our existing meta-data.

Meta Modeling

1. Data Exploring
Before creating the meta-model, it is essential to analyze and understand each meta-data individually using data visualization techniques. By visualizing the different target classes within each meta-data, we can gain insights and define the classification learning task for each meta-data.

Fig.8. Raw Meta-data Exploring

Fig.9. Preprocessed Meta-data Exploring

As Fig.8. shows the raw metadata defines a multi-class classification. There are three classes to predict, which are QuartzNet15x5BaseEn, ’wav2vec2-large-960h-lv60-self’, and wav2vec2-large-50-self. Fig.9. shows the pre-processed metadata defines a binary classification in which the meta-model should predict two classes which are wav2vec2-large-960h-lv60-self’, and wav2vec2-large-50-self.

2. Modeling
To create the meta-ASR Model, we follow the following steps:

a. Preparing the Metadata: We start by selecting the X (input) and Y (output) variables to train the statistical machine-learning models. X represents the meta-features, which could include attributes such as audio duration, language, or audio quality. Y defines the selection of ASR pre-trained models, indicating which model from a set of options is best suited for each audio sample.
b. Handling Unbalanced Data: In order to address the issue of unbalanced data, we employ the RandomOverSampler method. This technique generates synthetic samples of the minority class, thereby balancing the dataset. This step ensures that each ASR pre-trained model has sufficient representation in the training data.

c. Training the Statistical ML Models: The training process involves applying commonly used statistical ML models such as Decision Trees, Random Forests, and K-Nearest Neighbors (KNN). These models are well-suited for classification problems, which is the nature of our task. By training them on the prepared metadata, the models learn to predict the appropriate ASR pre-trained model based on the given meta-features.

d. Selection of the Best Statistical ML Model: The final step in constructing the meta-model is to select the best statistical ML model based on the F1-score, which is a widely used metric that combines precision and recall, providing an overall measure of the model’s performance. The statistical ML model with the highest F1-score is considered the best performer among the models evaluated.

By following these steps, we can create a meta-ASR Model that leverages statistical machine-learning techniques to predict the optimal ASR pre-trained model for a given audio sample based on its meta-features. This meta-model construction process helps automate the selection of the most suitable ASR model, improving the overall accuracy and efficiency of the transcription process.
This process selects the K-Nearest Neighbors (KNN) model as the best meta-model for both the raw meta-data (F1-Score 0.93) and pre-processed metadata (F1-Score 0.95).

3. Evaluation and discussion
The evaluation process of the meta-model involves testing its performance on the test data composed of 15 audios where the ground truth is available by transcribing the text from each input audio and computing the Word Error Rate (WER).

Fig.10. Meta ASR Evaluation

We can highlight the following points based on the modeling F1-Score results of the KNN model and the clustering visualization:
Pre-processed Meta-data Performance: The modeling results indicated that the pre-processed meta-data yielded favorable outcomes. The F1-Score achieved by the KNN model demonstrated the effectiveness of the pre-processed meta-data in predicting the optimal ASR pre-trained model. This suggests that the preprocessing steps applied to the metadata resulted in improved performance and accuracy. The visualization of the clustering results further supported the effectiveness of the pre-processed meta-data. Also, the evaluation process of the meta ASR shown in Fig.10 proves that the meta-model successfully performed the transcription task by leveraging the selected ASR pre-trained model, which can reduce the WER error.

Conclusion

The development of the spoken data meta-classification system has shown highly promising results, significantly improving the accuracy and reliability of transcriptions compared to using a single ASR model. By leveraging a comprehensive analysis of spoken data features, the meta-classification system successfully determines the most suitable pre-trained ASR model for transcription. It has demonstrated its effectiveness in handling diverse spoken data types, including low-quality audio and multilingual conversations. These results highlight the system’s potential to enhance ASR performance and produce more accurate and unbiased transcriptions.

However, the system’s limitation lies in its current focus on the English language. It is specifically optimized and trained for English-spoken data, which limits its applicability to other languages. To overcome this limitation, future research should concentrate on expanding the system’s language capabilities by incorporating diverse linguistic datasets. By incorporating language diversity, the system can effectively handle various languages and improve its performance across different linguistic contexts. Furthermore, exploring techniques for cross-lingual meta-classification and considering the unique linguistic characteristics and challenges of each language can further enhance the system’s accuracy and performance. Real-time adaptation and learning for different languages can also be explored to enhance adaptability and effectiveness in dynamic multilingual environments.

In conclusion, the development of the spoken data meta-classification system represents a significant achievement in improving the accuracy and reliability of ASR systems. While the system has demonstrated promising results in handling English speech, addressing its limitation in handling other languages is crucial. Future research should focus on expanding the system’s language capabilities to ensure its applicability in diverse linguistic contexts. This advancement has the potential to revolutionize speech recognition by enabling more accurate and accessible communication across different languages and cultural backgrounds.

Code: https://github.com/onssaadallah/META-ASR/tree/main

[1] https://zaion.ai/en/resources/news/what-is automatic-speech-recognition
[2] https://cloud.google.com/speech-to-text
[3] https://ai.facebook.com/tools/wav2letter/
[4] https://deepspeech.readthedocs.io/en/r0.9/
[5] https://arxiv.org/pdf/2202.07991.pdf
[6] https://thelevel.ai/blog/automatic-speech-recognition
[7] https://developer.nvidia.com/blog/essential-guide-to-automatic-speech-recognition
[8] https://athina-b.medium.com/audio-signal-feature-extraction-for-analysis-507861717dc1
[9] https://towardsdatascience.com/preprocess-audio-data-with-the-signal-envelope-499e6072108
[10] https://en.speechocean.com/Cy/885.htm
[11] https://www.rev.com/blog/resources/what-is-a-language-model-in-speech
[12] https://www.rev.com/blog/resources/what-is-wer-what-does-word-error-rate-mean
[13] https://catalog.ngc.nvidia.com/orgs/nvidia/models/quartznet15x5
[14] https://arxiv.org/abs/2006.11477
[15] https://cloud.google.com/speech-to-text?hl=fr
[16] https://machinelearningmastery.com/meta-learning-in-machine-learning