Discovering my own music tastes

A dear friend of mine (Talita Rossari), also diagnosed with ADHD, once said how strange it was to find out that other people don't have a soundtrack playing in their heads. I'd never thought about it before, and there it was: some music always going on behind my thoughts.

Despite my very little (honestly, none) musical aptitude, music has always been a big part of my life. My first attempts at English were translating songs off the radio. My first heartbreaks had Leoni songs playing underneath them. My teenage years sounded like The Police, Paralamas, Fito Páez and a lot of rock gaúcho. Early adulthood had, like almost everyone of my generation, Legião Urbana.

Over the years I noticed how choosing the soundtrack could change my mood. How many times I reached for "The Seeds of Love" to cheer myself up, or Republica's "Ready to Go" to face something hard. Or "Heroes", for, well, for everything.

And like most of my social media bubble, every December I join the wave of posting my musical year, compiled and analysed — in my case — by Spotify. It never told me much, though: a snapshot of a single year says nothing about how things changed, or about the life that changed around them.

Since I was a kid I've wanted access to the database of the simulation we live in, so I could ask "what's the song I've listened to the most — voluntarily — in my life?" or "which artist has always been with me?". That privileged access isn't available (yet), and it's the only thing that would ever count the records, tapes and CDs. So I settled for the data collected by the organisations of technofeudalism. Not perfect, but a start.

Three sources, three versions of the truth

I've been scrobbling since 2008 and using Spotify since 2015, and before either of those I had an iTunes library quietly counting plays since somewhere around 2003. That's a lot of listening history — and three completely different kinds of record. Getting them into one table turned out to be most of the project.

Here's what actually came out of each:

Every line in that table caused me a bug later on.

The Spotify API is nearly useless for history. The recently-played endpoint returns about 50 plays, tops. That's not a paging limit you can work around, it's the whole window. If you want your actual history, you have to ask for the Extended Streaming History export, wait a few days for an email, and parse the JSON they send you. Those 75 rows in the last line are just what the cron job has picked up since — they keep the dataset current, they don't build it.

Last.fm stopped in January 2016. Not because the sync broke, but because I stopped scrobbling when I moved to Spotify full-time. Which looks exactly like a stalled checkpoint, and yes, I checked twice.

The two sources overlap for about ten months. March 2015 to January 2016 is recorded by both, so the same listen exists twice, with slightly different timestamps and quite possibly different spellings of the artist's name.

And then there's that undated block. 11,390 plays imported from iTunes play counts, which aren't events at all. iTunes doesn't record when you played something, it records how many times. Turning "I played this 34 times" into 34 rows means inventing 34 timestamps, and whoever did that import — me, over a decade ago — invented them as a counter: 1, 2, 3, walking the library alphabetically by title. I was even able to recover a 2009 backup of iTunes to confirm my suspicions.

Hold that thought.

Deduplicating plays that two services both saw

If Last.fm and Spotify both recorded me listening to Under Pressure at 21:14 on a Tuesday, that's one listen, and it should count once. So the report collapses plays from different sources that land within 120 seconds of each other. Not exact-timestamp matching: the two services disagree by a few seconds about when a play "happened", because Last.fm logs when the scrobble arrived and Spotify logs when the track started. A window is the only thing that works — and it does work, collapsing 1,746 genuine duplicates today.

Now remember the undated block. Its timestamps are 1, 2, 3, 4, …, sorted alphabetically by title, so any song I played more than once occupies consecutive integers — and consecutive integers, read as unix timestamps, are one second apart. Every repeat fell inside the 120-second window. The deduplicator looked at eleven thousand plays, decided almost all of them were the same listen recorded twice, and threw them away. 11,390 plays were being counted as 670.

The fix is embarrassingly small — never collapse rows below the undated threshold, where the window means nothing anyway — but the reason it survived so long is the interesting bit. Nothing failed. No exception, no warning, no row count that looked off. My all-time total was 83,218, and 83,218 looks like a perfectly reasonable all-time total. It's 93,938. That's the theme of this whole project: when you merge sources that disagree, the failures are silent and they look like data. A crash would have been a gift.

Cleaning names when every source spells them differently

Scrobbles carry whatever text the client sent — not a canonical name, not an identifier, just a string typed or tagged by whoever ripped the CD in 2004. So one band shows up many times over: Echo & the Bunnymen and Echo And The Bunnymen, four capitalisations of Rocket Man, accented and unaccented spellings of the same Brazilian title that differ only in Unicode normalisation form and look identical on screen.

The first pass is a fold: lowercase, strip accents, drop punctuation, treat & and and as the same, ignore a leading article. Group by the folded form, keep the most-played spelling, rewrite the rest. That gets you a long way.

A perfect score can still be the wrong band

For anything the fold can't decide, you go ask an authority. MusicBrainz is the good one — open data, a real API, and unlike AllMusic it doesn't require scraping a site whose terms forbid it. But its search is fuzzy: helpful by design, and dangerous for exactly the same reason. Query a short name and it'll happily hand you a prefix match at full score. Which is how Pink ended up holding Pink Floyd's identifier.

That one's funny. Blitz isn't. 284 plays in my library, matched at score 100, name exact. Country: GB. Formed 1980, Derbyshire. Tags: punk, uk82 — except Blitz is a Brazilian band, from Rio, formed 1982. The UK punk band is perfectly real, and it scored exactly as well as the right one would have, because the score tells you whether the name matched, not whether it's the artist I listened to. The fold agreed, the identity check passed, and the row was written fully populated, looking impeccable, with 284 plays counted as British for months. So here's the rule I settled on: an artist whose name can't be confirmed is left unenriched on purpose. A null country is recoverable; a wrong one quietly poisons every aggregate downstream, and you'll never go looking for it, because it looks fine.

Those identifiers nearly cost me in the other direction, too. Artists have exactly one, so a fold whose members carry conflicting ids means one lookup is wrong — refuse the merge, flag it. That guard has caught real mistakes, and porting it to track titles looked like the obvious next step, until I counted how often it would fire: 21 of 285 perfectly good folds, like Would and Would?, or those four spellings of Rocket Man. A song gets a separate recording id for every master, remaster, live take and reissue, so conflicting ids are the normal case there and carry no signal at all. Track folding keeps the fold and drops the guard.

Artist or collaboration?

Queen & David Bowie is two artists. Simon & Garfunkel is one. So are Echo & the Bunnymen, Peter, Paul & Mary, Kool & The Gang, Daryl Hall & John Oates — all syntactically identical to the first one. I counted: of the credits in my library containing a separator, roughly half are a single band. A rule that split on syntax would be wrong about half the time, and wrong in the worst way available, moving hundreds of plays onto an artist who never played them, quietly, with the result still looking like a clean ranking. "No MusicBrainz match, so it must be a collaboration" doesn't rescue it either: Kleiton e Kledir is a duo with 294 plays and no identifier, and so are Gang 90 & Absurdetes and Fausto Fawcett e Os Robôs Efêmeros.

So the tool doesn't split anything. It proposes, and I decide — and it only proposes when there's actual evidence: either Spotify lists more than one artist id on the resolved track, which is the service's own authority rather than my string parsing, or every part of a separator split is already a known, enriched artist in my own library. That qualifies maybe a fifth of the candidates, which is the point. Half a split is worse than no split.

Then there's what this does to counting. A play by Queen & David Bowie should count for Queen and for Bowie — but do that everywhere and my country percentages stop adding up to 100%, because one play now belongs to two countries. So there are two models at once, on purpose: plays stay single-attribution, which keeps every share and total honest, while the ranking sorts on credited plays, with a separate column for guest appearances. The original plays.artist string is never rewritten — the mapping lives in its own table, so clearing it undoes the whole thing, unlike the name merges, which are permanent by design.

The parts that were just hard

Spotify's catalog API was supposed to give me genres and popularity. My app is in development mode, and moving it out is a bureaucratic process I have no appetite for — so the batch endpoints answer 403, and everything costs a request per track instead of per fifty. Annoying, survivable. The nasty part is what a successful response looks like: a 200 for a single artist comes back with the right name and the interesting fields missing from the payload entirely, not null, not empty, not there.

Parse that with the natural payload.get("genres") or [] and you get an empty list, which looks exactly like an artist who genuinely has no genres tagged. And these caches are write-once, because re-fetching everything on every cron run would be rude and slow — so that empty list would have been written down as fact, permanently, with nothing anywhere recording that the data had never been fetched. One probe printing sorted(payload) settled it. The code now spots a stripped payload and refuses to cache it, and genres come from MusicBrainz tags and the old iTunes library instead.

I also got MusicBrainz to block my IP for an entire evening, with a diagnostic that swept 25 candidate queries in a loop and a comparison script that retried on failure. A retry on a throttle response is indistinguishable from an attack, and they were right to block me. When you're diagnosing a rate-limited API: make one request, read all of it, stop.

The rest of it comes from running on cron, which imposes one requirement on everything above: every command has to be safe to run again. Inserts deduplicate on a unique constraint over source, timestamp, artist and track, rather than a check-then-insert, which races. Checkpoints only move forward, and only after the corresponding rows are committed. The Last.fm sync walks pages oldest-first, which looks backwards until you think about being interrupted — a run that dies halfway leaves a checkpoint behind reality rather than ahead of it, and behind is recoverable while ahead means silently skipped history. None of it is clever. It's just the difference between a script and something you can leave running for a year and forget about.

And the results are...

Not really a surprise.

Leading the track charts: Everybody Wants to Rule the World, Take on Me, Trac-Trac, Under Pressure and Head over Heels. And the top five artists: Os Paralamas do Sucesso, Tears for Fears, a-ha, The Police and Legião Urbana.

The interesting finds were further down. I've listened to over 600 distinct US artists, and the highest-ranked one only shows up at number 22 (Bob Dylan). In 2020, half of my top 20 artists were Brazilian. And in 2009, while preparing for a role in a play, I went through a lot of Yann Tiersen.