This guest post was written by Jakub Chour, a mobile growth freelancer and the former CEO of Surge, a dating app for gay men. Jakub is also a prolific member of the MDM Slack community.
Firebase predictions are a great dev-stack tool which helps to deliver better and faster apps with fewer bugs. It promises to deliver reliable churn and purchase predictions on a user-level. So, how do you figure out if it’s worth trying for your case? We (in Surge) tried, so you don’t have to.
The good parts
Predictions are available automatically when you send app events to Firebase. You only need to switch data-exports to BigQuery (Settings > Project settings > Integrations). It’s free when you don’t count BigQuery prediction storage; but even for a regular-sized app, it’s peanuts.
Google also claims that for a meaningful prediction, you need to have at least 500 daily active users, which makes it accessible to almost anyone who might need any prediction.
The bad parts
Most of our successful machine learning efforts started with cleaning and understanding data. For example, Surge is a swipe-based dating app, so if you swipe left or push the dislike button, it’s the same. Or, in the situation when a match_happened event happens simultaneously to both users, one event is dependent on the other. A human data-scientist knows that it’s the same. The machine might think it’s not. And this makes it potentially less precise.
There is also a set of recommended events that an app should have. This is to make Predictions and other Google services better adjusted to your user behavior. That means being able to see how many users upgraded their hero (unlock_achievement), how many of them made a purchase (in_app_purchase) and more. But this event list is only for a few types of apps, and it includes some of our personal favorites — like message_send (highly correlates with Day X Retention). What does this mean?
Google does not understand what is going on and probably only unleashes rough computing power to calculate churn based on events it recognizes. And even if it is predicting a purchase, you can’t really be sure what the purchase will be. If you have all of your purchases hidden under the in_app_purchase event, you can’t know if the user might want to buy a ?️ or a ?.
Moreover, when you tell Google who your purchasing users are, be aware that it uses the information to optimize Google UAC campaigns to deliver better users — for your competition. Why? Google knows that those are highly-relevant users and which apps they use.
Our results
Google offers two sets of tables. Old and “new” — which promises to get even better results. We used a data-set of approx. 200k users, which makes one month of predictions. We used Python Pandas and standard Python libraries to compare data from Firebase, then compared it to our real-world data downloaded from Branch and RevenueCat.
Spending firebase predictions
spend_confusion_matrix() True positive spend predictions: 0 True negative spend predictions: 63101 False positive spend predictions: 0 False negative spend predictions: 2360
As you can see, even if one of the automatic events from Firebase is a purchase, it couldn’t predict ANY of the purchases from approx. 2.5k a month.
Churning firebase predictions
churn_confusion_matrix() True positive churn predictions: 275 True negative churn predictions: 145130 False positive churn predictions: 26 False negative churn predictions: 48484
For churning users, it’s even worse. From approx. 200k users, it could only predict 275 out of 145k instances of churn (yes, we have a high user turnover — dating is hard).
Summary
As you can see, Firebase Predictions has no use for our specific case. Maybe we are just unlucky. My take is that it’s not common to have the majority of your users converting so early in the funnel.
Maybe for other segments — like games or travel, where algorithms are better and you have your events set up spick and span — this might be a great tool. But in our case, we’d end up giving precious information away to Google, and possibly our competitors, for free.
Recent