How Spotify Tests Playlist Personalization

In this episode of Conversion Rate Optimization with Fexingo, Lucas and Luna dig into Spotify's approach to A/B testing for personalized playlists. They focus on a specific experiment from 2022 where Spotify tested two different ways to surface Discover Weekly tracks: one using a collaborative filtering algorithm and another using a neural network model. The test involved over 10 million users and measured not just click-through rate but also long-term retention and time spent listening. Lucas explains why most companies would have declared the neural network the winner after a few weeks, but Spotify waited three months to see if the novelty wore off. Luna pushes back on whether three months is enough for music fatigue. The episode reveals how Spotify balances short-term engagement metrics with true user satisfaction, and why their A/B testing framework explicitly includes a 'delight' metric alongside more traditional conversion goals. The hosts also briefly discuss how listeners can support the ad-free show via buy me a coffee dot com slash fexingo. #Spotify #DiscoverWeekly #Personalization #ABTesting #Algorithm #NeuralNetwork #UserRetention #EngagementMetrics #DelightMetric #MusicStreaming #ConversionRateOptimization #CRO #Marketing #DataDriven #LongTermTesting #NoveltyEffect #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo