<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andreas Stergioulas on Medium]]></title>
        <description><![CDATA[Stories by Andreas Stergioulas on Medium]]></description>
        <link>https://medium.com/@andreas.stergioulas?source=rss-8cf6c6e88328------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*34hTGsDX4FvX5QkD</url>
            <title>Stories by Andreas Stergioulas on Medium</title>
            <link>https://medium.com/@andreas.stergioulas?source=rss-8cf6c6e88328------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:45:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@andreas.stergioulas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Neural Audio Embeddings for Content-Based Song Retrieval]]></title>
            <link>https://medium.com/wpp-ai-research-labs/neural-audio-embeddings-for-content-based-song-retrieval-a358c7733b99?source=rss-8cf6c6e88328------2</link>
            <guid isPermaLink="false">https://medium.com/p/a358c7733b99</guid>
            <dc:creator><![CDATA[Andreas Stergioulas]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 12:05:07 GMT</pubDate>
            <atom:updated>2026-01-26T12:05:07.441Z</atom:updated>
            <content:encoded><![CDATA[<p>The domain of Music Information Retrieval (MIR) is evolving from the rigid determinism of signal processing to more fluid, semantic capabilities of deep representation learning. The industry standard for identifying audio has been “audio fingerprinting” — a technology typified by services like Shazam, which excels at Known-Item Search. These systems rely on extracting low-level features from audio signals, typically identifying local energy maxima (peaks) within a spectrogram and hashing their time-frequency coordinates into a constellation map. This approach is essentially a high-speed, noise-resilient exact match system. It is computationally efficient and exceptionally precise at identifying a specific digital recording, even in the presence of significant background noise or distortion.</p><p>However, the definition of “identity” in music is multifaceted. While a fingerprint can identify a specific master recording of “Let It Be” by The Beatles, it possesses no inherent understanding of the song’s musical content. It cannot identify a cover version performed on a solo piano, nor can it discern the mood, genre, or instrumentation of the track. It fails to answer user queries such as “find me sad piano music similar to Ludovico Einaudi” or “retrieve songs with a driving 80s synth-pop rhythm.” This limitation arises because the “fingerprint” is tied to the specific waveform of the recording rather than the semantic essence of the composition.</p><p>The emergence of <strong>Neural Audio Embeddings</strong> addresses this semantic gap. Unlike fingerprints, which are hand-crafted features, neural embeddings are dense, high-dimensional vector representations learned by Deep Neural Networks (DNNs). These models — trained on massive datasets using self-supervision, contrastive learning, or generative objectives — map audio into a continuous vector space where geometric distance corresponds to semantic or acoustic similarity. In this latent space, a studio recording and its live acoustic cover are likely to cluster together, despite their waveforms looking completely different.</p><p>This article provides an analysis of the state-of-the-art in neural audio embeddings for song retrieval and their capabilities in song retrieval. In addition, we explore the implementation challenges of deploying these models in large-scale vector search systems.</p><h3>1. Theoretical Framework: From Spectrograms to Latent Spaces</h3><p>To understand the capabilities and limitations of modern embedding models, one must first deconstruct the underlying principles of how machines “listen.” The transition from classical signal processing to deep learning represents a move from handcrafted feature engineering to implicit feature learning.</p><h3>1.1 Classical Audio Fingerprinting</h3><p>Classical fingerprinting algorithms, such as the combinatorially hashed peak-pair method used by Shazam, operate on the principle of spectro-temporal landmarks. The process begins by converting the time-domain audio signal into a frequency-domain representation via the Short-Time Fourier Transform (STFT). From the resulting spectrogram, the algorithm selects “peaks” — points that have higher energy than their neighbors in both time and frequency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGdXOqdhvikOsxfVJrvJAw.jpeg" /><figcaption>Shazam’s algorithm</figcaption></figure><p>These peaks form a “constellation map.” To ensure specificity, the algorithm does not just index single peaks; it indexes pairs of peaks (an “anchor” point and a “target” point) separated by a specific time delta. This pair forms a hash. Retrieval is a rapid database lookup: if the query’s hashes match the database’s hashes, and the relative timing between matches is consistent, a hit is recorded.</p><ul><li><strong>Strengths:</strong> Extreme precision, scalability to billions of tracks, robustness to noise and compression artifacts.</li><li><strong>Weaknesses:</strong> Complete lack of semantic generalization. A change in tempo (time-stretching) or pitch (pitch-shifting) alters the coordinates of the peaks, breaking the hash matches. Consequently, traditional fingerprinting is not optimal for recommendation systems, genre classification, or finding stylistically similar music.</li></ul><h3>1.2 Neural Audio Fingerprinting</h3><p>Recent advancements have sought to bridge the gap between robustness and flexibility through “Neural Audio Fingerprinting.” Systems like NeuralFP [1] utilize deep learning to generate fingerprints that are invariant to severe distortions, such as speed changes or heavy background noise, which would break a classic Shazam-style algorithm. These models often employ contrastive learning . They are trained on pairs of audio snippets — an “anchor” and a “positive” (a distorted version of the anchor) — and “negatives” (unrelated audio). The network learns to minimize the distance between the anchor and positive embeddings while maximizing the distance to negatives.</p><h3>1.3 The Concept of the Embedding</h3><p>An “embedding” is a conversion of discrete, high-dimensional data (like a sequence of audio samples) into a continuous, lower-dimensional vector (e.g., a vector of 512 or 1024 floating-point numbers). The critical property of an embedding is <strong>topology preservation</strong>: items that are “similar” in the real world should be “close” in the vector space.</p><p>Different training objectives yield different definitions of “similarity”:</p><ul><li><strong>Discriminative Embeddings (e.g., Classification):</strong> If a model is trained to classify genres (Rock vs. Jazz), the embeddings will cluster by genre. A slow rock ballad and a fast rock anthem might be close together.</li><li><strong>Contrastive Embeddings (e.g., Audio-Text):</strong> If trained to match audio to text captions (CLAP [2]), the embedding space becomes semantic. A song described as “spooky” will be close to the word “spooky”.</li><li><strong>Generative Embeddings (e.g., Codecs/Synthesis):</strong> If trained to reconstruct the audio waveform, the embeddings capture acoustic fidelity — timbre, texture, and recording quality.</li></ul><h3>2. Self-Supervised Speech Models</h3><p>A surprising development in Music Information Retrieval is the performance of models originally designed for speech recognition. WavLM [3], developed by Microsoft, stands as a prime example of how large-scale self-supervised learning on speech can yield universal audio representations that excel at music tasks. WavLM is built upon the HuBERT framework, which itself draws inspiration from BERT in Natural Language Processing. The core architecture is a Transformer.</p><p>WavLM is pretrained on an enormous corpus of speech data — <strong>94,000 hours</strong>, combining datasets like Libri-Light, GigaSpeech, and VoxPopuli. The pretraining task is “Masked Speech Prediction and Denoising.”</p><ol><li><strong>Masking:</strong> Segments of the input audio are masked (hidden) from the model.</li><li><strong>Noising:</strong> The input is also corrupted with background noise or overlapping speech.</li><li><strong>Objective:</strong> The model must predict the discrete classification (pseudo-label) of the masked frames based on the unmasked, noisy context.</li></ol><p>This dual objective forces the model to learn robust features. To predict missing audio from context, it must understand syntax and prosody (which map to musical structure and phrasing). To handle noise, it must learn to separate signal from interference (which maps to identifying instruments in a dense mix).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*bPSBkXZTTdeYlvArznd6lw.png" /><figcaption>WavLM architecture, taken from [3]</figcaption></figure><h3>3. Multimodal Contrastive Learning</h3><p>While WavLM learns from audio structure, CLAP [2](Contrastive Language-Audio Pretraining) learns from the relationship between audio and language. CLAP employs a dual-encoder architecture, conceptually similar to OpenAI’s CLIP model for images.</p><ol><li><strong>Audio Encoder:</strong> Processes the audio waveform or spectrogram. CNN-based.</li><li><strong>Text Encoder:</strong> Processes the textual description. CLAP uses BERT.</li></ol><p>The model is trained on pairs of (Audio, Text). The training objective is a <strong>Contrastive Loss</strong>. The model projects both the audio and the text into a shared latent space and optimizes the weights so that the embedding of an audio clip and the embedding of its correct text caption are close, while unmatched pairs are pushed apart.</p><p>CLAP also allows for Zero-Shot Retrieval. Traditional classifiers can only identify classes they were trained on (e.g., if trained on 10 genres, they cannot identify “Dubstep”). CLAP, however, can retrieve any concept that can be described in language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G7dzB0E1xEDxxVVVZRPLnQ.png" /><figcaption>Clap model, taken from [2]</figcaption></figure><h3>4. Neural Audio Codecs</h3><p>A new frontier in audio embeddings has emerged from the field of <strong>Generative Audio</strong>. Models designed to <em>create</em> or <em>compress</em> audio (Codecs) are inadvertently producing some of the most powerful representations for retrieval, particularly for tasks involving structural and timbral similarity.</p><p>DAC [4] is a high-fidelity Neural Audio Codec designed to compress audio into discrete tokens at very low bitrates (e.g., 8 kbps) while maintaining near-perfect reconstruction quality. The model’s architecture is a convolutional encoder-decoder. It also uses Snake activation functions, which unlike standard ReLUs, introduce a periodic inductive bias, making the network inherently better at modeling the periodic nature of audio waveforms (frequencies).</p><p>The embeddings from DAC can be the continuous output of the encoder (before quantization) or the sequence of discrete codebook indices. Because DAC is trained with a reconstruction loss (MSE) and adversarial losses (Discriminators), it is penalized if the output doesn’t sound <em>exactly</em> like the input. Therefore, its latent embeddings must preserve fine-grained details like recording noise, room ambience, and precise instrumental timbre. This makes DAC embeddings superior for Sample Retrieval<strong>.</strong></p><h3>5. Generative Latents</h3><p>MusicGen [5] is an autoregressive language model for music. It uses a tokenizer (EnCodec, similar to DAC) to convert audio into a sequence of discrete codes, and then uses a Transformer to predict the next code in the sequence. The model learns to model the <em>probability distribution</em> of music. To predict the next bar of a song, it must understand the song’s key, tempo, chord progression, and genre.</p><p>MusicGen predicts multiple streams of codebooks in parallel using a specific interleaving pattern. This forces the model to capture both the “what” (melody/content) and the “how” (texture/fidelity) simultaneously.</p><p>For song Retrieval , the hidden states of the MusicGen Transformer can be extracted as embeddings. These representations are particularly rich in structural information. Recent experiments have combined CLAP and MusicGen (CLAP-MusicGen) [6] to leverage MusicGen’s deep musical understanding for retrieval tasks.</p><h3>6. Structural Representations</h3><p>A specific sub-problem in MIR is Cover Song Identification: recognizing that a heavy metal track and a piano ballad are the same underlying composition. Acoustic models (DAC) fail here because the timbre is different. Semantic models (CLAP) might work if the text description is similar, but often fail if the genre is radically changed.</p><p>Pop2Piano [7] offers a unique solution. It is a Transformer-based model (based on T5) trained to convert pop audio waveforms into piano MIDI files. By training the model to transcribe audio to MIDI, the encoder is forced to strip away “style” (timbre, instrumentation, production) and extract “content” (melody, harmony, rhythm). The latent representation produced by the Pop2Piano encoder is effectively a “style-agnostic” embedding of the song’s musical score. In this vector space, the heavy metal cover and the piano original become very similar, because their underlying sequence of notes (and thus their latent representation) is nearly identical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-yQG0yJ21g-5ITzTxanbA.png" /><figcaption>Pop2Piano model diagram, taken from [7]</figcaption></figure><h3>7. Comparative Performance Analysis</h3><p>To contextualize the models discussed, we performed an empirical evaluation of the aforementioned models to assess their efficacy in a real-world song retrieval task. A small dataset of 20 songs was compiled and the objective was to determine how well an embedding model can identify a parent track based solely on a short, 15-second audio segment.</p><h3>7.1 Evaluation Methodology</h3><p>The evaluation utilized a “segment-to-track” matching protocol designed to simulate common retrieval scenarios:</p><ol><li><strong>Segmentation</strong>: Each source audio track was divided into non-overlapping <strong>15-second chunks</strong>.</li><li><strong>Mean Embedding Generation</strong>: A “ground truth” representation for each full track was created by averaging the embeddings of all its constituent segments.</li><li><strong>Top-1 Retrieval</strong>: For every individual segment, the system attempted to retrieve the parent track by finding the closest mean embedding in the database.</li><li><strong>Scoring</strong>:</li></ol><ul><li><strong>Track-Level (Majority Vote)</strong>: A track is considered “correct” if 50% of its segments correctly identify it.</li><li><strong>Segment-Level</strong>: Standard metrics (Precision, Recall, F1) were calculated based on individual segment predictions.</li></ul><h3>7.2 Results Summary</h3><p>The following table details the performance of the models across these metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gvl4JIbwR3AyVKAD08f_kg.png" /><figcaption>Quantitative results of the discussed models</figcaption></figure><h3>7.3 Discussion of Model Performance</h3><p>The empirical results reveal several critical insights into the nature of these latent spaces:</p><ul><li><strong>The Power of Reconstruction (DAC)</strong>: <strong>DAC</strong> achieved near-perfect scores. Because DAC is trained to reconstruct the exact waveform, its intermediate representations capture high-fidelity acoustic details (timbre, texture, and noise) that act as a high-dimensional digital fingerprint. In a test set of 10 tracks, only a single segment was misclassified.</li><li><strong>Semantic Robustness (CLAP &amp; WavLM)</strong>: Despite being trained on general audio-text and speech tasks respectively, both <strong>CLAP</strong> and <strong>WavLM</strong> showed exceptional track identification. CLAP’s performance validates its ability to map musical segments to a stable semantic space. WavLM’s success suggests that the features it learns from speech (rhythm and phrasing) translate effectively to musical structure.</li><li><strong>Structural Focus (Pop2Piano)</strong>: <strong>Pop2Piano</strong> performed remarkably well. This confirms that by forcing the model to focus on MIDI-like melodic and harmonic information, it creates highly discriminative embeddings that are less sensitive to production variations.</li><li><strong>Generative vs. Discriminative (MusicGen)</strong>: The finetuned <strong>MusicGen</strong> model, while capable, showed the lowest performance. This suggests that its latent states, which are optimized for <em>predicting the next token</em> in a sequence, may contain more “temporal noise” than the static representations of purely discriminative or compression-focused models.</li></ul><h3>References</h3><p>[1] <a href="https://ieeexplore.ieee.org/document/9412489">https://ieeexplore.ieee.org/document/9412489</a></p><p>[2] <a href="https://arxiv.org/abs/2206.04769">https://arxiv.org/abs/2206.04769</a></p><p>[3] <a href="https://arxiv.org/abs/2110.13900">https://arxiv.org/abs/2110.13900</a></p><p>[4] <a href="https://arxiv.org/abs/2306.06546">https://arxiv.org/abs/2306.06546</a></p><p>[5] <a href="https://arxiv.org/abs/2306.05284">https://arxiv.org/abs/2306.05284</a></p><p>[6] <a href="https://huggingface.co/yuhuacheng/clap-musicgen">https://huggingface.co/yuhuacheng/clap-musicgen</a></p><p>[7] <a href="https://arxiv.org/abs/2211.00895">https://arxiv.org/abs/2211.00895</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a358c7733b99" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wpp-ai-research-labs/neural-audio-embeddings-for-content-based-song-retrieval-a358c7733b99">Neural Audio Embeddings for Content-Based Song Retrieval</a> was originally published in <a href="https://medium.com/wpp-ai-research-labs">Satalia AI Research Lab</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>