Music Information Retrieval: Modeling Audio Data
Perhaps there is no other type of data with as much variety in modeling approaches as audio data. In this article, we will explore various ways to classify audio files, including classic machine learning, computer vision, and transformer models.
The goal of this study is to try out different modeling approaches to solve a genre classification task using the GTZAN data set. The GTZAN dataset, often called the MNIST of sound, includes 1,000 audio files across 10 genres. To ensure a fair comparison of models, the same train-test split was used in all experiments. However, this article does not aim to select the single best approach for modeling audio data but instead explores the possibilities.
This article continues a series of materials about music information retrieval:
- The first part discussed the specifics of processing audio data.
- The second focused on feature engineering.
I’ll skip the basics discussed in the previous parts, so please refer to them if needed. Check out the notebook in GitHub for the code.
From Audio to Table: Classic Machine Learning Approach
The first approach follows the classic machine learning logic for unstructured data: convert it into a tabular format through feature engineering.
Music Information Retrieval: Feature Engineering covered lots of possible features like Mel frequency cepstral coefficients, time and frequency domain features.
Preparation and Results
For this task, I calculated the mean and variance for these features:
- Tempo,
- Amplitude envelope,
- Root mean square energy,
- Zero crossing rate,
- Spectral centroid, bandwidth and roll-off,
- Harmonic and percussive,
- Chromagram,
- 20 Mel frequency cepstral coefficients (MFCCs).
This resulted in 57 features for 1,000 audio files across 10 genres.
The data was split into training (70%) and testing (30%) sets and scaled using a standard scaler. I trained a few models from KNN and Logistic Regression to ensemble models:
Result: Without extensive tuning, LightGBM outperformed other classic models, achieving an accuracy of 0.79 on the test set. Not bad!
Artificial Neural Network
The same feature engineering process can be used to train an artificial neural network (ANN).
Result: My four-level neural network reached accuracy of 0.77 at the same test set.
From Audio to Image: Computer Vision Approach
An intriguing idea is to convert audio into images and apply computer vision techniques. This involves generating a mel spectrogram for each audio file, which represents the frequency content of the audio signal over time.
Get Katsiaryna Ruksha’s stories in your inbox
Join Medium for free to get updates from this writer.
Unlike traditional image processing, data augmentation techniques like rotation, zooming, or flipping are not applicable to mel spectrograms since these distortions make the spectrograms meaningless.
Experiments and Results
I experimented with:
- A convolutional neural network (CNN),
- Transfer learning approaches using EfficientNet (inspired by this Kaggle post claiming 0.78 accuracy).
Result: Transfer learning with EfficientNet achieved an accuracy of 0.74 on the same test set.
From Audio to Embeddings: Transformer Approach
Transformers have taken 2024 by storm, and audio data is no exception. For this experiment, I used DistilHuBERT and parameters from WasuratS/distilhubert-finetuned-gtzan, which claims 0.9 accuracy on GTZAN.
Experiments and Results
The data preparation for this model is straightforward and involves using a pre-trained FeatureExtractor to perform the following steps:
- Normalization
- Truncation/Padding to a fixed length (3 seconds in this case due to resource constraints).
- Returning additional model-specific features, such as attention masks.
Results: Despite the shortened data length, the transformer model achieved an accuracy of 0.70 after 15 epochs.
Conclusion
The goal of this study was not to identify the best model for audio data but to explore a range of possibilities. In my experiments, the clear winner was LightGBM, although with proper hyperparameter tuning, this could change.
There is no one-size-fits-all solution for modeling audio data. Instead, a variety of approaches — ranging from feature engineering to computer vision and transformer models — should be considered depending on the task. It’s still advisable to explore different options to find the most suitable model for your specific application.
