How Spotify’s audio recommendation system works
It’s that time of year again. On the first day of december, everyone unpacks their “Wrapped” lists from the Spotify music streaming app. Every person you follow seems to capture their best artists and songs, and float them on social media, providing a condensed auditory story of their year.
This Year’s Spotify Custom Recap of Your 2021 Soundtrack features your most played artists, songs, genres and even a musical mood board in a clickable interactive story. You can now also “to mix together“Your music year with that of a friend in a merged playlist.” (The mood board and mixed playlists are new this year.)
All of this begs the question: what is the idea behind showing users how they have consumed music? A clue could lie in a study by the Spotify research team written about last May. The researchers showed 10 users their personal data profiles based on their Spotify account which contained information about their top songs (from last month and all time), top genres, number of playlists they have. created and when they listened to Spotify. They found that presenting data from a user’s personal listening history actually allowed them to “think about who they were as listeners,” and it allowed them to see if they were only listening to music. music while working, or if they had periods of intense obsession with an artist.
We care about that kind of intuition even so intuitively, we know it’s just music, and besides, we all know we’re not the only fans of Taylor Swift or Lorde. It may be that seeing a narrative (now with a hint of emotional arc) created around the songs that shaped your year is always a little personal and at times revealing. (If you feel up to it, you can leave a outside of AI judge your Spotify). A FiveThirtyEight The writer once thought Spotify seemed to know him better than he knew himself.
So how exactly does Spotify do this? We know they have tons of data they’ve collected from listeners (381 million monthly active users at last count). Here’s the kind of analysis they’ve done behind the scenes to figure out what their users like to hear.
From music library to musical discovery
When Spotify was founded in 2006, its aspiration was to be a music library. The personalization came later, when the engineers of the app realized that letting people discover new music they might like could improve their experience. And that could be done by providing an algorithm with information about a user’s listening history, music choices, playing time of certain songs and how they respond to recommendations (do they like them, skip them, they replay, they record them).
“Personalization was a stimulating experience for listeners who didn’t have the time or knowledge to create unique and endless playlists for every dinner or road trip,” said Oskar Stål, vice president of the customization of Spotify, in a blog post from October 2021. “It opened up discovery to a broader level, allowing hundreds of artist discoveries per person per year.”
[Related: A brief history of shuffling your songs, from Apple to Adele]
Their approach to this type of personalization revolves around two main areas of research: user modeling and a complex musical analysis. Spotify attempts to model user behavior on the app by finding methods to project activities in the app into human traits and emotions, and by associating musical experiences with mood and situational contexts As time of the day, week or season. Knowing this could allow them to change what they recommend on a Friday night versus a Tuesday afternoon. Recommended playlists may appear on carousels on your home screen; there are custom playlists such as Discover Weekly, Daily Mix, and Radio playlists.
In addition, a new feature called “improve” lets you grab recommendations from a playlist you’ve already created, and this week only, said Stål in a presentation video that the Spotify team was considering an approach that mixes a human editor with the machine learning algorithm to create audio experiences that could possibly mix and match songs with podcasts and the like. Spotify even tested a neural network called CoSeRNN which evaluates certain characteristics such as past listening history and current context to suggest song recommendations suitable for the moment.
As for testing whether the music reflects certain human characteristics, they released the results of a small survey-based study last December to see how musical preference matched certain personality traits. In a blog post, the researchers noted that there appeared to be correlations between personality and genre preference. Unsurprisingly, people who identified themselves as “open to new experiences” viewed Discover Weekly more; those who identified them as extroverts listened more to the playlists that other people had created, while those who identified as introverts preferred to delve into the discography of a newly discovered artist.
Get to the heart of composing your perfect playlist
The Spotify team seems to be constantly thinking about new ways to sort and recommend different types of music to their users. To get there, they must first take the different types of data they collect and build models that can analyze, compare, contrast, sort, and aggregate the variety of information they get. The company’s researchers noted in an article from 2016 they scour the web for information about artists and words used in online reviews to describe specific songs. They build algorithms that can dissect the sound structures of songs and analyze how songs relate by analyzing the billions of user-generated playlists already on the platform. In addition, they approximate the musical tastes of a given user by analyzing their historical and real-time listening habits.
Discover Weekly, for example, is powered by an in-depth analysis of the songs a user has recently listened to and an analysis of all playlists that may contain those songs or similar songs. Spotify uses a machine learning tool called the approximate nearest neighbor search algorithm to group songs and users based on shared attributes or qualities.
[Related: This new AI tool from Google could change the way we search online]
“Imagine you and someone else have four of the same top performers, but your fifth performers are different. We would take those two quasi-matches and think, ‘Hmm, maybe each person would like the other’s fifth artist,’ and suggest it, ‘Stål explained in a blog post. “Now imagine this process happening at scale – not just one-on-one, but thousands, millions of connections and preferences instantly taken into account. “
Spotify, on the other hand, did a lot of math separate a song in its different layers of instruments, breaking down its rhythm and structure. In November, the music service released a study offering a new personalized recommendation model called MUSIG that learns “meaningful representations songs and users’ based on the individual characteristics of the songs (such as genre, acoustics, dance, verbosity of lyrics) and how they relate to each other (as if they appeared on the same list reading).
However, it’s not enough to find out what users are enjoying right now. Our musical tastes change over time and Spotify needs to come up with new songs that users like to keep them coming back.
This includes recommend a mix popular or similar content to music the person has listened to before, as well as exploratory content that is more eclectic and not aligned with usual user engagement.
[Related: MIT scientists taught robots how to sabotage each other]
“Maybe you’ve joined Spotify for dance music, but can we help you focus while you study. By balancing these polarities, can we help you achieve a more fulfilling diet, ”said Stål in the presentation video. “We have to think about your wants, the things that will keep you in your comfort zone and your needs, so things that may improve your listening ability down the line but may not be exactly what you expect right now. “
This type of formula “can make crawling easier by helping users discover new content or instill new tastes,” Spotify researchers wrote in a post in March. “It can help the platform distribute consumption among artists and make it easier to consume less popular content. “
Of course, Spotify has its own financial reasons for wanting to diversify the taste palette of its users. Internal studies showed that active users with more diverse listening habits were “25 percentage points more likely to switch from the free to premium version than those with less diverse music consumption.”
To stay ahead of our changing preferences and keep their recommendations up to date, Spotify must also be able to understand how what we like can change over time. Earlier this year, researchers built a model there based on a dataset of 100,000 Spotify users who were continuously active from 2016 to 2020. They looked at each user’s entire streaming history. , grouped their music into “micro-genres” and mapped them. Through time. They came up with a connected graphic that illustrated the transitions between different genres of music. For example, their model suggested that in order to move from a preference for “EDM” to “nu jazz” or “gospel”, users were likely to go through a phase of “tropical house” appreciation, which is cold but upbeat electronic music.
Using the paths laid out in this model, Spotify hopes that they can gradually acclimate users to different genres as they travel through microgreens that fall between what they already like and what they don’t yet know.
Outside of music, Spotify’s decade of personalization research has already spanned podcasts. “It turns out that we can even predict what types of podcasts a listener might enjoy based on their musical tastes,” Stål noted in a corporate blog post. But Spotify’s audio expansion plans don’t end there, it recently acquired the Findaway audiobook company.