This was especially important, since in 2019, we were working with about 5x the amount of data over 10 years. To reduce the amount of shuffles, we decided to use Bigtable as our final data store. The biggest change in design from 2018 involved storing the data in a way that would reduce the amount of shuffles needed to group all the data stories on a user level.
2020 SPOTIFY WRAPPED WINDOWS
We were able to use this listening history/analytics data source to get and summarize user stories for not only each year of the decade, but also for targeted windows of each season of 2019. This data lake is backed by Google Cloud Bigtable and is highly optimized for aggregating data over an arbitrary time range.
2020 SPOTIFY WRAPPED SERIES
Fortunately, we have a system developed internally to help us accomplish this in an efficient fashion by giving us access to time series data. We knew we wanted several separate statistics drawn from each year of the decade, and that required having access to the user listening analytics data for each individual year. Coupled with how we store user data, with a focus on protecting user privacy, there ends up being a potential of resource capacity issues. Spotify has user listening analytics data dating back to our first years as a streaming platform, but it is difficult to read this amount of data in a single job. In Wrapped 2019, the main challenges we faced with respect to the decade concept were getting the listening history and summarizing thousands of historical data points per user over 10 years in a cost-effective manner while ensuring data quality. Due to the scope of the work, we had to work closely with Google Cloud to implement the ability to handle the data at such a large scale. From the prior Wrapped Team’s experience, we took learnings, which heavily influenced our decisions in designing the architecture for the 2019 campaign.
The amount of data we tried to process for just 2018 with basic pipeline design struggled to complete at the scale we were attempting. In 2018, the Wrapped Campaign data pipeline had one of the largest Dataflow jobs to ever run on GCP (Google Cloud Platform), which resulted in limits around the amount of data we were able to shuffle. Individually, each of these data stories can be very computationally expensive. To bring you a Decade Wrapped, we had to process these data stories over 10 years’ worth of data for all of our monthly active users (MAU), of which we have over 248 million. Therefore, the biggest challenge we faced for Wrapped 2019 was scale. For example, Your Top Artists, Your Top Songs, and Your Top Podcasts are considered three separate data stories. In this post, we will refer to each of these separate statistics as a “data story”, because it is a statistic, or story, we are telling the user through their historical listening data. Scopeįor 2019’s Wrapped, there were multiple statistics that we wanted to highlight individually for our users. Many of these teams are spun up in the months’ prior launch, and are largely made up of volunteers who put aside their day-to-day work to focus entirely on launching Wrapped. In this post, we will focus on how our team–the User Wrapped data engineering team–managed to process ~5x the amount of data compared to that of 2018’s Wrapped campaign at a fraction of the cost. This was an ambitious goal and one that posed many engineering challenges.īecause Wrapped is such a massive effort, the Wrapped Team itself encompasses many sub-teams that are responsible for everything from marketing, legal, and design, to data, frontend and backend engineering. As one of the few streaming platforms that has existed since before 2010, Spotify had a unique opportunity to be able to provide users with a review of their listening habits over the entire decade. Since 2019 was the end of the decade, we wanted to do something special for our users. It enables our users to see a detailed breakdown of their listening habits over the past year. The Spotify Wrapped Campaign is one of Spotify’s largest marketing and social campaigns of the year.
0 kommentar(er)
