I see a lot of questions on Quora and elsewhere about prediction algorithms, many related to user conversion and retention (eg: how can I predict if a user will convert?). While modern big data and machine learning systems provide analysts with an ability to develop behavior profiles that were previously unaccessible, much of that capacity is wasted on useless predictions: predictions which are deterministic in nature and merely tell us things we would have learned had we waited.
I defined downstream marketing in a previous post as analytics-driven marketing directed at a user who has already been acquired. I believe most behavioral predictions in the downstream marketing phase are useless: knowing someone’s likelihood to convert doesn’t produce value if that user has already been acquired. A prediction about the end state (did convert / didn’t convert, retained for X days, etc.) of a user currently in the system can’t be used to optimize that user’s experience; that knowledge is only good for evaluating the accuracy of the prediction algorithm.
Predictions should be actionable: they should elucidate a clear path to improving the user experience. So what if user 581958 is predicted to retain for 59 days, based on his behavior in day 2? What can be done with that information? How is that knowledge useful? Some claim that predictions such as these can be used to inform the acquisition process by (eg users from channel Y are predicted to retain for 59 days), but to them I say: why not just wait and use real data to train that model? Wouldn’t a model trained on data produced by another model be less robust than a model trained on real behavioral data?
Downstream prediction algorithms should give the user more of what he likes and less of what he doesn’t; they should be useful in a very concrete, applied manner in improving the user’s experience. Knowledge of what the player may or may not do isn’t helpful unless that knowledge is applied in such a way so as to remove impediments to user delight. More important than the prediction when analyzing user behavior are the features that contribute to the prediction: the explanations for why something happens. These are actionable at the product level and can contribute the incremental metrics boosts needed to drive sustainable, appreciable growth in a service.