Day 2: Understanding Audio Data Before Using Librosa

This is Day 2 of my Multimodal AI from Scratch series.

Yesterday, I learned that multimodal AI combines different types of information, such as text, audio, and video.

Today, I started with the first modality: audio.

Before using Librosa, MFCCs, mel-spectrograms, or wav2vec, we need to understand what audio data actually is.

What is sound?

Sound is created by vibrations in the air.

When someone speaks, their vocal cords create vibrations. These vibrations travel through the air and reach a microphone. The microphone converts them into an electrical signal.

A computer stores this signal as a sequence of numbers.

Sound in the air
↓
Microphone
↓
Electrical signal
↓
Numbers stored by a computer

This sequence of numbers is called a waveform.

What is a waveform?

A waveform shows how the amplitude of sound changes over time.

• The horizontal axis represents time.
• The vertical axis represents amplitude.
• Higher amplitude usually means louder sound.
• Lower amplitude usually means quieter sound.

A simple waveform can look like this:

Amplitude
   ↑
   |      /\        /\
   |     /  \      /  \
   |____/    \____/    \____ Time

In real speech, the waveform is much more complex because speech contains many frequencies changing very quickly.

What is a sampling rate?

A computer cannot record sound continuously. Instead, it measures the amplitude of the sound many times every second.

The number of measurements per second is called the sampling rate.

For example:

• 8,000 Hz means 8,000 samples per second.
• 16,000 Hz means 16,000 samples per second.
• 44,100 Hz means 44,100 samples per second.

If the sampling rate is 16,000 Hz and the audio lasts 3 seconds, the audio contains:

16,000 × 3 = 48,000 samples

For speech analysis, 16,000 Hz is a common choice. It is often used in speech-processing models, including wav2vec-based systems.

What is amplitude?

Amplitude represents the strength of the sound signal.

A larger amplitude generally means a louder sound. A smaller amplitude generally means a quieter sound.

However, amplitude is not the same as emotion. A person may speak loudly because they are excited, angry, emphasizing a point, or simply because they are close to the microphone.

This is why amplitude should be interpreted carefully in sentiment analysis.

What is frequency?

Frequency describes how quickly a waveform repeats.

Get Sunayana Parikh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It is measured in Hertz, or Hz.

• Low-frequency sounds have slower vibrations.
• High-frequency sounds have faster vibrations.

In speech, frequency is related to pitch. A higher fundamental frequency is often perceived as a higher-pitched voice.

Frequency is important because speech contains many frequencies at the same time. This is why we later use tools such as spectrograms and MFCCs.

Mono and stereo audio

Audio can be mono or stereo.

• Mono audio has one channel.
• Stereo audio has two channels, usually left and right.

For speech analysis, we often convert stereo audio to mono because we usually want one clean speech signal.

audio, sr = librosa.load(
    "speech.wav",
    sr=16000,
    mono=True
)

Loading audio with Librosa

Librosa is a Python library for analyzing audio and music signals.

A basic example is:

import librosa

audio, sr = librosa.load(
    "speech.wav",
    sr=16000,
    mono=True
)
print("Sample rate:", sr)
print("Number of samples:", len(audio))
print("Duration in seconds:", len(audio) / sr)

Here:

• audio is a NumPy array containing amplitude values.
• sr is the sampling rate.
• len(audio) gives the total number of samples.
• len(audio) / sr gives the duration in seconds.

Visualizing a waveform

We can visualize the waveform using Librosa and Matplotlib:

import librosa
import librosa.display
import matplotlib.pyplot as plt

audio, sr = librosa.load(
    "speech.wav",
    sr=16000,
    mono=True
)
plt.figure(figsize=(12, 4))
librosa.display.waveshow(audio, sr=sr)
plt.title("Audio Waveform")
plt.xlabel("Time (seconds)")
plt.ylabel("Amplitude")
plt.show()

This plot helps us see:

• When speech starts and stops.
• Periods of silence.
• Loud and quiet parts.
• Sudden changes in the signal.

Why is this important for multimodal AI?

In multimodal sentiment analysis, audio is one of the main modalities.

Before extracting features such as MFCCs, pitch, or spectral flux, we need to understand the raw audio.

The audio waveform is the starting point:

Raw audio waveform
        ↓
Audio features
        ↓
Machine-learning model
        ↓
Sentiment prediction

If the audio is noisy, incorrectly sampled, or not aligned with the video, the later multimodal model may learn the wrong patterns.

What I learned today

Today’s main lesson is:

Audio is not just a sound file. It is a sequence of numerical samples that represent changes in amplitude over time.

#MultimodalAI #Librosa #AudioProcessing #MachineLearning #DataScience #SentimentAnalysis #LearningInPublic