One of the most important data visualizations a Product Manager will use to inform product development during a soft launch and immediately after a hard launch is the new users drop-off chart, which relates the percentage of new users from a cohort present at each milestone in the onboarding process.
A typical new users drop-off chart looks like this:
Each bar on the chart represents the percentage of users that continued to that step in the onboarding process relative to all new users in the cohort. In the chart above, the value at the fourth milestone, Tutorial Start, refers to the percentage of the cohort of new users that reached the start of the tutorial (98%).
Designing a new users drop-off chart
The new users drop-off chart describes the onboarding process and should therefore not be left open ended: like every process, new user onboarding, as conceptualized by the Product Manager, starts and ends at discrete points. One common mistake in drop-off chart design is excessive breadth; tracking milestones in the new user drop-off chart that cannot reasonably be considered part of the onboarding process will produce muddied results.
Any individual drop-off chart should track user movement along a directed product process that doesn’t offer alternative choices at any of the milestones. In other words, each milestone should represent a point in the product that the user could not have avoided reaching aside from closing the product (because the purpose of the drop-off chart is to track drop-off). When the chart tracks product milestones late into the first session — or, worse yet, past the first session — the likelihood of those milestones being the only available option to the user at the point they were reached decreases.
Simply put, user drop-off values at milestones that are part of a number of options presented to the user – for example, tracking the number of users that reached Mission 2 when both Mission 2 and PvP play options are available after Mission 1 has been completed – are unreliable at best (and misleading at worst).
In the case that the new users drop-off chart cannot possibly be designed without including milestones that are part of a set of options to the user, the onboarding process should be conceptually separated into multiple processes: the onboarding process ends with the milestone where choices are presented, and each choice begets its own drop-off chart tracking user movement. Designed this way, the set of drop-off charts essentially tracks an A/B test. An outline for such a process is shown below:
Another common mistake in designing a drop-off chart is to blend cohorts over a period of time and use amalgamated percentages at each milestone. Again, this approach distorts the drop-off values: multi-cohort result blends, especially in the soft launch stage when user numbers may be low and product iterations are released frequently, are not valid. Ideally, the new users drop-off chart is capable of being segmented not only by daily cohort but also by various user characteristics such as location, device, and acquisition source.
Cohorts drop-off values should be calculated individually and compared – especially across releases – to track improvements to the process over time. Blending cohorts hides improvements and produces a single visualization that doesn’t inform the product development process.
Drop-off charts should likewise be oriented around discrete, specific events, not time markers (such as the number of seconds since product launch, eg. 30 seconds in-app, 60-seconds in-app, etc.). Players use products at different paces, and comparing time in product between users reveals nothing about drop-off. Milestones should only be constructed with trackable client events and not based on a running timer.
Interpreting a new users drop-off chart
When large drop-offs occur between milestones – that is, when the difference between the percentage of users at one milestone and the next is large – two options are available for improving the process. The first is to track a new intermediary milestone between the two milestones in an attempt to more accurately place the source of the drop-off. The second is to conduct an A/B test on the first milestone to understand why users are leaving after reaching it.
Both options have merit, but the A/B test produces actionable results faster. It’s rare that enough functional distance exists between two milestones in an onboarding process to engender ambiguity around why users are leaving the product. The purpose of the drop-off chart is to allow the Product Manager to prioritize the product improvement development schedule – in other words, the drop-off chart should alert the Product Manager to the points in the onboarding process where A/B tests would bear the most fruit.
Adding additional data to the drop-off chart doesn’t directly improve the product – and, if it leads to an overly complicated visualization, it can actually impede product progress. The need for additional data should be considered carefully and weighed against the product development cadence, especially when new tracking milestones must be implemented into the product (which can require time-intensive client work).
The drop-off chart must also not be confused for a measure of retention. Drop-off charts measure process completion, and retention measures fundamental product delight. Conflating the two sets of metrics in the use of a drop-off chart can lead to a conflict between local and global optimization.
For instance, simplifying the onboarding process will, in almost all cases, improve the values reported on the drop-off chart. But an overly simplistic onboarding experience may neglect to address fundamental product usage information to users, leading to decreased medium- and long-term retention (because users don’t fully understand the product).
Some level of drop-off in the onboarding process must be accepted; not every product is a perfect fit for every person. But the values on the new users drop-off chart shouldn’t dictate the degree to which the onboarding process helps the user to understand the product: that must be set as an immutable standard, the implementation of which the drop-off chart helps to inform.
Recent