Become a Data-Driven PM

As a PM who takes usage growth as my top priority, data plays a critical role in my daily work. In the past several years, I have run projects of growing usage for different products, it may worth sharing some methodology and personal learning on how to do data-driven product design and marketing.

Know your customers

Before talking about data, keep asking the 4Ws: 1) Who are your customers? What is their biggest pain points? 2) How will they use your product? 3) Why do they need your product? You should be crystal clear about the value add and competitive advantage of your product, which will help you define a good success metric in the next step.

Define success metrics

Having "One Metric that Matters" is the key to drive the success of your product. Depending on what business you are in and the stage of your product, you need to define different metrics as your product KPI. Lean Analytics has lots of real examples showing how to define a good metric in startup companies. The key thing to remember is - the metric should be simple enough to explain and actionable so that everyone can be guided to move towards the right direction. If you find it hard to define such a metric, How to Measure Anything might give you some inspiration.

Monthly Active Users (MAU) is commonly used as the KPI to measure startup companies in their early stage. Once you decided it's the right metric, you should document the detailed definition and calculation as there are many tricky things that need to be carefully thought through. For example:

• What do you mean by "monthly"? - Is it a calendar month or rolling 28/30/35 days? Calendar monthly usually shows up in month end reporting and rolling days are used frequently if you want to compare historical trend.
• What's the definition of "active"? - Users can interact with the product in various ways. You need to define a list of "intentional" or "high value" actions that can properly represent the value add of your product (e.g. sending/receiving/reading emails are core actions of an email app, you can't imagine a mail app user doesn't perform any of these actions to benefit from your product). If your app will be automatically logged in when system starts, then using the logon event to count active users is not a good choice, because users may do nothing valuable in your app or even not aware of the auto-logon.
• What ID should be used to represent a "user"? - It's quite prevailing today to build an app that a user can use in multiple devices. So whether you are counting a user ID or device ID should be clearly documented (especially for some apps that don't require user login). On the other hand, some test users or system accounts may be logged in your telemetry system, you need to consider excluding them to avoid over-counting.

Instrument & model data

Once you have the metrics defined, collect as much telemetry data as possible. You may need to make some trade-offs if telemetry collection will impact your product's user experience (e.g., pre-aggregate or sample the events before sending them back to server for better performance). Make sure you log consistent IDs for users and devices, which will save you plenty of time in the later stages of data cooking and analysis. Data granularity is usually collected at "action" level, i.e. WHO performs WHAT action against WHAT object at WHEN, WHERE and HOW (e.g. user agent with OS, device, app related info logged). Based on that, you can further aggregate to higher granularity to calculate active user, user retention, etc.

Understand your customers - usage analysis

Now it comes to the most fun part - analyzing data to understand your customers. Various approaches can be adopted like DAU/MAU historical trend, user funnel & retention, individuals' or segment of users' pattern analysis, etc. There are many tools today that can help you easily perform these analysis. Once you get some results, remember talking to your customers to validate them. It's also not surprising to get additional insights from your customers which data cannot tell you.

I like the picture below to explain how to breakdown the single number of MAU. Ask these questions and try to figure out the WHYs with the help of telemetry data as well as talking to your customers:

• Churned users - Why do they leave? Is it because of feature gap, usability or seasonality? What can I do to bring them back?
• Retained users - What are their usage patterns? Can I replicate their patterns to more users?
• New users - Where do they come from? How do they know my product?

From insight to action

Once you get all the above questions answered, you will probably know what to do next. You might need to ask for resources to be funded based on your analysis. Using a good chart with key messages delivered can help a lot to convince your leadership team. Story Telling with Data and Show Me the Numbers are 2 good books to learn all these Dos & Don'ts in data visualization communication.

To conclude..

• Data cannot tell everything, but it can help guide you towards the right direction.
• Intuition is still very important. Good data & analysis can help validate your intuition or assumption.
• Usage analysis is time consuming. You probably will spend 80% time in data cleansing & cooking. Try to leverage tools to automate the process as much as possible so that you can focus more on insight mining.