<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ben Willett on Medium]]></title>
        <description><![CDATA[Stories by Ben Willett on Medium]]></description>
        <link>https://medium.com/@ben_willett?source=rss-dad0701a9d2b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ebeNd8rfgrG_Nv5x.</url>
            <title>Stories by Ben Willett on Medium</title>
            <link>https://medium.com/@ben_willett?source=rss-dad0701a9d2b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:20:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ben_willett/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Music, Big Data, and OCD — part 01]]></title>
            <link>https://medium.com/@ben_willett/music-big-data-and-ocd-part-01-1d78ebe99a01?source=rss-dad0701a9d2b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d78ebe99a01</guid>
            <category><![CDATA[id3]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[ocd]]></category>
            <dc:creator><![CDATA[Ben Willett]]></dc:creator>
            <pubDate>Sat, 05 Oct 2019 23:50:25 GMT</pubDate>
            <atom:updated>2019-10-05T23:50:25.709Z</atom:updated>
            <content:encoded><![CDATA[<h3>Music, Big Data, and OCD — part 01</h3><p>The epic quest for analog data in a digital world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hcJjizGvNHqEKpmFru9qtg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@marksolarski?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Mark Solarski</a> on <a href="https://unsplash.com/search/photos/record-collection?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Like a lot of music collectors, I spend a lot of time thinking about how to organize my music collection, both my physical and digital media. Do I sort Thom Yorke under “T” or “Y” or just to the right of Radiohead? And do I put The B-52’s with all the other “The” bands or with the rest of the Bs? And what’s with that apostrophe? Having moderate <strong>OCD</strong> adds a whole new level of intensity to this process.</p><p>In simpler times, the <strong>vinyl LP</strong> cover was the single source of truth for artist and album data. In addition to beautiful cover artwork by artists like <strong>Gary Burden, Storm Thorgerson, Frank Olinsky</strong> and many others, you had the artist or band name, the album name, and an ordered list of songs with title and duration. Sometimes there were song lyrics and maybe a list of musicians and what they contributed to the album. Or maybe as is the case with <strong>The Beatles White Album</strong>, nothing but an embossed band name. At the time, lying on the floor listening with a pair of headphones this all seemed quite sufficient. If you wanted more info, you might catch some backstory from <strong>Dennis Elsas</strong> on <strong>WNEW FM</strong> or in a copy of <strong>Rolling Stone</strong>.</p><p>In mid-1980s, the advent of the Compact Disc added convenience — no more getting up to flip the record over, up to 80 minutes of continuous music and the ability to easily skip that awful track or repeat that awesome one. The artwork got a lot smaller and some bands made up for the smaller CD form factor by including a fold out insert or booklet containing art and album info. Was this convenience (plus the end of skips, scratches and pops), at the expense of sound quality? (<strong>INSERT ANALOG V. DIGITAL FLAME WAR HERE </strong><a href="https://emojipedia.org/imp/"><strong>👿</strong></a>)</p><p>In the 1990s, we entered the new age of <a href="https://en.wikipedia.org/wiki/Digital_audio"><strong>digital audio</strong></a>. <strong>Ripping</strong> CDs added another level of convenience. But, there was one problem... The <strong>Compact Disc Digital Audio</strong> standard a.k.a. <a href="https://en.wikipedia.org/wiki/Compact_Disc_Digital_Audio"><strong>Red Book Audio</strong></a> did not include album metadata on the disc. The spec includes a simple table of contents with track offsets and that’s it. No artist name, no track titles, no album art, nothing.</p><p>So, where do we get this metadata? Media library or CD ripping software like iTunes or <a href="https://jriver.com/"><strong>JRiver Media Center</strong></a> for example, based on user preferences would create media files of type <a href="https://en.wikipedia.org/wiki/MP3"><strong>MP3</strong></a><strong>, </strong><a href="https://en.wikipedia.org/wiki/MPEG-4_Part_14"><strong>M4A/MP4</strong></a><strong>, </strong><a href="https://xiph.org/flac/"><strong>FLAC</strong></a><strong>, </strong><a href="https://xiph.org/vorbis/"><strong>Vorbis</strong></a> and others. The software would then need to access an online database of artist and album data. One early example these databases was <a href="https://en.wikipedia.org/wiki/CDDB"><strong>CDDB</strong></a> which eventually became <a href="https://www.gracenote.com/"><strong>Gracenote</strong></a>. Once retrieved, software would store the album metadata in a ‘tag’ structure within each track file and in a database on disk to speed library searches and scrolling.</p><p>MP3 files typically use a structure called an <a href="http://id3.org/"><strong>ID3</strong></a> tag to store the metadata. Other media file formats like <strong>FLAC</strong> and <strong>Vorbis</strong> use <strong>Vorbis Comments</strong> or <a href="https://en.wikipedia.org/wiki/APE_tag"><strong>APEv2</strong></a> tags. The tags can be structured data or simple key value pairs for artist name, album name, year, track name, length, genre and others. ID3 tags have a formalized schema whereas APEv2 uses simple key value pairs. <a href="http://wiki.hydrogenaud.io/"><strong>Hydrogenaudio</strong></a> has a wealth of technical information on audio file formats, encoding and tagging specs.</p><p>So, where are these tags stored and what do they look like? Let’s look at an ID3 tag in an MP3 file created by JRiver Media Center. Using any hex editor, take <a href="https://ridiculousfish.com/hexfiend/"><strong>Hex Fiend</strong></a> for example, you can open the MP3 file and see the ID3 tags at the beginning of the file. Tag “TIT2” refers to the track title <strong><em>Thursday Afternoon</em></strong> which is also the album name as indicated by the “TALB” tag. “TPE1” indicates the lead performer, in this case <strong>Brian Eno</strong>. There are other tags indicating genre, dynamic range, the tool used to tag the file and many others. The ID3 link above has the complete list of tags available in the ID3 spec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZDEWLsT_ObhOWR4p4hVsbQ.png" /></figure><p>Then on October 23, 2001, Steve Jobs announced the iPod with a 5 GB mechanical hard drive that put “1,000 songs in your pocket.” No more carrying around that <a href="https://en.wikipedia.org/wiki/Discman"><strong>Sony Discman</strong></a><strong> </strong>and <strong>96 disc Case Logic. </strong>iPhones today have up to 512GB of storage, a 100x increase allowing us the freedom to store higher quality files using higher bitrates and/or lossless formats like FLAC. For comparison, The <strong><em>Thursday Afternoon</em></strong> track above is 60 minutes long. Encoded as a 320Kbps MP3 the file size is 106MB, while the FLAC (5) version is 250MB.</p><p>Fast forward to <strong>2019</strong> and in addition to digital downloads, I still buy physical CDs to get that <strong>Red Book</strong> audio quality (<strong>44.1khz x 16bit</strong>) that few of the online digital media giants provide. <a href="https://www.amazon.com/music/unlimited/hd"><strong>Amazon Music HD</strong></a>, <a href="https://www.hdtracks.com/"><strong>HDTracks</strong></a>, <a href="https://brio.orastream.com/"><strong>Brio by OraStream</strong></a><strong> and some artists, </strong>among other sources offer <strong>HD</strong> audio downloads at <strong>192khz x 24bit</strong> for a few bucks more than a standard CD. For the right album, IMHO, it’s definitely worth it. (<strong>INSERT NYQUIST FLAME WAR HERE </strong><a href="https://emojipedia.org/imp/"><strong>👿</strong></a>)</p><p>Requiring <strong>perfectly clean album metadata</strong> is another manifestation of my <strong>OCD</strong>. Whether it’s a digital download or a CD rip, often times the metadata is clean — each track with the proper spelling including uppercase/lowercase, title, artist, album, track number, year, genre, etc. But if it’s not, i.e. <strong>disk1, Disc 2, (disk 2/2)</strong>, etc., I’ll hand edit it using the venerable <a href="https://www.mp3tag.de/en/"><strong>Mp3Tag</strong></a> by Florian Heidenreich or <a href="https://jriver.com/"><strong>JRiver Media Center</strong></a><strong>.</strong></p><h3>Apps and Metadata Services</h3><p>I mentioned <strong>Gracenote</strong> earlier, but there are several free and paid metadata services out there as well as artists’ self-published discographies. Digital music library software will include a license to do metadata lookups against one or more of these databases, some offer it as a paid option and some provide access to free databases like <a href="http://www.freedb.org/"><strong>FreeDB</strong></a> and <a href="https://www.getdigitaldata.com/GD3.aspx"><strong>GD3</strong></a>. <strong>Apple </strong>and most of the digital media giants including <strong>Amazon Music</strong>, <strong>Pandora</strong>, <strong>Spotify</strong>, etc. use <strong>Gracenote. JRiver</strong> uses its own <strong>YADB</strong> database (+1 for the name).</p><h3>The Bigger Picture</h3><p>After about 1,000 CD rips, hand-editing metadata with various tools going back to the late ’90s including <strong>RealPlayer, Audacity, WinAmp, Windows Media Player, iTunes </strong>(Bye, Felicia!)<strong>, Exact Audio Copy, dBpoweramp, VLC, and now Mp3Tag and JRiver Media Center</strong> to name a few, I felt like I had this process down to a science. Ready to be peer-reviewed and submitted to the American Journal of Psychiatry for a whole new <strong>ICD10</strong> code. <strong>F42.9</strong> (Obsessive-compulsive disorder, unspecified) just wasn’t going to cut it anymore. Then it hit me, like that <strong>PJ Harvey</strong> album <strong><em>Stories from the City, Stories from the Sea </em></strong>that you have 3 copies of because it sounds so good when you’re drunk that you keep rebuying it (Thanks Amazon!),… there is always prior art. <strong>Always</strong>. CD look ups are great, but someone, somewhere must have all the metadata…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*4ocDeH0VfUdHpzGWSUL8zg.jpeg" /></figure><p>So, I did a little research…</p><h3>Metadata services</h3><p>As mentioned above, there are many apps that do a great job on lookups, but my brain needed access to the raw datasets. All the data. Searching the web, I found a few companies in this space including <a href="https://www.gracenote.com/"><strong>Gracenote</strong></a>, <a href="http://the.echonest.com/"><strong>The Echo Nest</strong></a>, <a href="https://us.7digital.com/"><strong>7digital</strong></a>, <a href="https://musicbrainz.org/"><strong>MusicBrainz</strong></a> and <a href="https://www.discogs.com/"><strong>Discogs.com</strong></a>. After a brief and <strong>rigorously unscientific analysis</strong> I decided to dive deep on <strong>Discogs</strong> for a few key reasons: they don’t require a commercial license, they have a <strong>RESTful API</strong> with client libraries available in various languages (including <strong>Python</strong>, <strong>NodeJS, PHP, Ruby</strong> and others), and they provide access to their entire database in ~7GB <strong>monthly XML dumps</strong>. Another service, <a href="https://musicbrainz.org/"><strong>MusicBrainz</strong></a><strong>, </strong>has a great set of apps, an <strong>API</strong> with a feature called <strong>AcoustID</strong> that enables tagging based on <strong>audio fingerprinting</strong> when no other data is available, and a large database.</p><h3>Discogs.com</h3><p>Discogs.com’s about page (<a href="https://www.discogs.com/about"><em>https://www.discogs.com/about</em></a>) says this:</p><blockquote>“We’re on a mission to build the biggest and most comprehensive music database and marketplace. Imagine a site with discographies of all labels, all artists, all cross-referenced, and an international marketplace built off of that database. It’s for the love of music, and we’re getting closer every day.”</blockquote><p>Take a look at <strong>Discogs </strong><a href="https://discogs.com/"><strong>Marketplace</strong></a><strong> </strong>site, their <a href="https://www.discogs.com/developers"><strong>developer porta</strong>l</a>, and their <a href="https://data.discogs.com/"><strong>database download</strong></a> pages.</p><p>I may cover their marketplace data in a future post, but for now we’ll cover the music metadata portion of their database which is centered around 4 main resource types: <strong>artist, label, release, and master release</strong>. The metadata for each object can be quite lengthy, so I won&#39;t list it out here, but clicking the links below will return the HTML page about a given resource and the JSON payload returned by the Discogs REST API.</p><h4><a href="https://www.discogs.com/developers/#page:database,header:database-artist">ARTIST</a></h4><p>The Artist resource represents a person in the Discogs database who contributed to a RELEASE in some capacity. See below for more info on the RELEASE resource.</p><p>Here are links for the HTML page and the JSON payload for artist “<strong>The Jimi Hendrix Experience”</strong> with<strong> Discogs ARTIST id 255672.</strong></p><pre><strong>HTML: </strong><a href="https://www.discogs.com/artist/255672-The-Jimi-Hendrix-Experience"><strong>https://www.discogs.com/artist/255672-The-Jimi-Hendrix-Experience</strong></a></pre><pre><strong>JSON: </strong><a href="https://api.discogs.com/artists/255672"><strong>https://api.discogs.com/artists/255672</strong></a></pre><blockquote>Note the HTML page URL uses the singular ‘<strong>artist</strong>’ while the REST API uses the plural ‘<strong>artists</strong>’. This quirk also applies to the label, release and master release resources.</blockquote><h4><a href="https://www.discogs.com/developers/#page:database,header:database-label">LABEL</a></h4><p>The Label resource represents a label, company, recording studio, location, or other entity involved with Artists and Releases. Labels were recently expanded in scope to include things that aren’t labels — the name is an artifact of this history.</p><p>Here are links for the HTML page and the JSON payload for label <strong>Reprise Records </strong>with <strong>Discogs LABEL id 157</strong>.</p><pre><strong>HTML: </strong><a href="https://www.discogs.com/label/157-Reprise-Records"><strong>https://www.discogs.com/label/157-Reprise-Records</strong></a></pre><pre><strong>JSON: </strong><a href="https://api.discogs.com/labels/157"><strong>https://api.discogs.com/labels/157</strong></a></pre><h4><a href="https://www.discogs.com/developers/#page:database,header:database-release">RELEASE</a></h4><p>The Release resource represents a particular physical or digital object released by one or more Artists.</p><p>Here are links for the HTML page and the JSON payload for release <strong><em>Songs In A Minor</em> by Alicia Keys</strong> with <strong>Discogs RELEASE id 946140</strong>.</p><pre><strong>HTML: </strong><a href="https://www.discogs.com/Alicia-Keys-Songs-In-A-Minor/release/946140"><strong>https://www.discogs.com/Alicia-Keys-Songs-In-A-Minor/release/946140</strong></a><strong><br> <br>JSON: </strong><a href="https://api.discogs.com/releases/946140"><strong>https://api.discogs.com/releases/946140</strong></a></pre><h4><a href="https://www.discogs.com/developers/#page:database,header:database-master-release">MASTER RELEASE</a></h4><p>The Master resource represents a set of similar Releases. Masters (also known as “master releases”) have a “main release” which is often the chronologically earliest. A set of similar releases might include the US release, European release, 180-gram vinyl deluxe edition, cassette re-issue (yes, that is a thing now…), etc.</p><p>Here are links for the HTML page and the JSON payload for <strong>MASTER RELEASE </strong>of <strong><em>Songs In A Minor</em> by Alicia Keys</strong> with <strong>Discogs MASTER RELEASES id 144290</strong>.</p><pre><strong>HTML: </strong><a href="https://www.discogs.com/Alicia-Keys-Songs-In-A-Minor/master/144290"><strong>https://www.discogs.com/Alicia-Keys-Songs-In-A-Minor/master/144290</strong></a></pre><pre><strong>JSON: </strong><a href="https://api.discogs.com/masters/144290"><strong>https://api.discogs.com/masters/144290</strong></a></pre><h4>Discogs Marketplace</h4><p>As mentioned above, there are also several metadata resources associated with the Discogs Marketplace including <strong>Users</strong>, <strong>Inventory</strong>, <strong>Wantlists</strong>, etc. The <strong>Discogs</strong> <strong>Marketplace</strong> is a fun place to spend a few hours unwinding on a Friday night browsing rare mono Beatles albums, for example. You can buy, sell, browse other Discogs user&#39;s collections and create Wantlists of albums you’re searching for. I may delve into the Marketplace data in a future post.</p><p>A few notes for those of you still reading…</p><ol><li>There is, as far as I know, no universal identifier for an artist other than their name. Artist identifiers will vary across data providers. Artist name, with all of its potential for ambiguity, is still your best bet although <a href="https://musicbrainz.org/"><strong>MusicBrainz</strong></a> has done some excellent work in linking IDs from various databases. <strong>Machine Learning</strong> is already helping with <strong>fuzzy matching</strong> and <strong>deduplication</strong> and may obviate the need for a universal identifier.</li><li>You may have noticed the <strong><em>data_quality</em></strong> attribute at the end of the JSON payloads linked above<em>. </em>This is a great feature of the Discogs database where <strong>crowdsourced</strong> data can be verified and <strong>upvoted</strong> increasing confidence that you’re looking at the highest quality (most accurate) metadata for a given resource. <strong>#OCD</strong></li><li>Some very interesting <strong>Blockchain</strong> projects are popping up with the goal of definitively identifying the copyright holder for a piece of music for the purpose of more accurately paying royalties to artists. Check out <strong>Imogen Heap</strong>’s <strong>Mycelia</strong> project (<a href="http://myceliaformusic.org/">http://myceliaformusic.org/</a>) among others.</li></ol><p>That’s it for now. Thank you for reading. I hope this backgrounder on digital music metadata was informative and useful.</p><p>In the next post, we’ll access the Discogs API using popular tools like <strong>curl</strong> and <strong>Postman </strong>to crack open the Discogs artist, label, release, and master release <strong>JSON</strong> and examine their data in detail.</p><p>After having had a thorough look at the <strong>Discogs API</strong>, future posts will dive into the raw ~7GB <strong>XML dumps</strong>. We’ll use <strong>Python</strong> and the <strong>ElementTree XML API</strong>, <strong>Apache Spark</strong> and <strong>Parquet files</strong>, and then some cloud-based tools from <strong>AWS</strong>. Once you’ve worked through the 7GB dataset on your laptop, you (and your laptop’s fans) will appreciate the power and flexibility of the cloud-based tools.</p><p>As this is the first part of a series, I’d love some feedback on the blog itself, areas of music metadata you’d like to see covered and which data processing tools you’re using for this type of work.</p><p>Part 2 should be out in a couple of weeks. Thanks for reading!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*gKxq3g91GTDVxu3BNxg9eg.jpeg" /></figure><p>“Is it wrong, wanting to be at home with your record collection? It’s not like collecting records is like collecting stamps, or beermats, or antique thimbles. There’s a whole world in here, a nicer, dirtier, more violent, more peaceful, more colorful, sleazier, more dangerous, more loving world than the world I live in; there is history, and geography, and poetry, and countless other things I should have studied at school, including music.” <br>― Nick Hornby, High Fidelity</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d78ebe99a01" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>