Product Metrics: To measure, or not to measure?
For the new ones here — 8,000+ smart, curious folks have subscribed to the growth catalyst newsletter so far. If you are new here, receive the newsletter in your email by subscribing 👇
You can read 50+ posts from the past here - https://www.growth-catalyst.in/archive
We often hear the adage: “What gets measured gets managed”. Here is another interesting and useful one I recently came across
“What gets mismeasured gets mismanaged”
A good example of the above is Cobra Effect. The story behind the term goes something like this. In colonial India, there was a cobra infestation in Delhi. So the British officials offered a bounty for every dead cobra brought to them. The policy initially appeared successful, snake catchers started claiming their bounties and fewer cobras being seen in the city. Yet, there was a steady increase in the number of dead cobras being presented for bounty payment each month. It was puzzling till they understood what was going on behind the scenes.
To get stable income, the snake catchers started breeding Cobras. Rather than hunting them in the city, it was much easier to kill captive cobras. This led to the discrepancy the British officials observed. When they discovered people breeding cobras, they abandoned the bounty policy.
You would think that this would be all. But the breeders were now stuck with worthless cobras. They released them, making the problem even worse than before! Hence the term ‘Cobra Effect’ was born!
‘What gets mismeasured gets mismanaged’. So when I came across this question that one of the Growth cohort members (checkout pmcurve.com if you haven’t, you may find it interesting) posted in the slack community —
"What are the metrics you will measure for WhatsApp Voice Calling?"
It looked like a good opportunity to discuss how to define metrics. Even if it appears to be a simple question that gets asked in product interviews, the response to this question can be much more evolved if you spend some time and energy in answering it well.
First, you have to strike a balance between covering all aspects of product metrics while not bloating the list by including everything. A lot of candidates and teams IRL become the victim of the bloating the metrics. They end up listing and tracking everything and lose sight of what's important and actionable. You also have to be careful to around which metrics can be gamed, and how to avoid that. You have be aware of your current product and org strategy, and incorporate them into metrics.
So how do we answer this question, not just for WhatsApp but for the products we are managing? In this post, we are going to take Whatsapp voice calling as an anchor and create a framework to track important metrics for any product. So let’s go!
If you are a PM who doesn’t come from a software background, you can checkout my book ‘Tech Simplified for PMs and Entrepreneurs’ which has been immensely useful (readers’ word, not mine) in getting them to understand tech well :) 240+ people have rated it 4.5+ on Amazon.
A two-step framework can help here. The first step is defining the north star metric, keeping company priorities and product value for the user in mind. The second step is defining metrics for various product areas to build a holistic set.
Step 1: Defining North Star Metric (NSM)
If we talk about Whatsapp, the company's priority is to ensure it stays the #1 communication app. Introducing voice calling is a way to ensure that.
Let’s talk about the user value. The value that users get out of Whatsapp is free and secure communication across the globe. Voice calling is another mode of communication.
Keeping the user value and company priority in mind, there are two candidates for north star metrics — daily calling users and daily call duration. For a social media app, time spent matters a lot because that is a proxy of value the user gets (posts read, videos consumed, etc.). You can't say the same about Whatsapp. Whatsapp falls in the productivity category. It has to ensure effective communication, and sometimes that means lesser time or lesser number of calls.
So we would pick daily calling users as the NSM for WhatsApp.
Let's move to step 2.
Step 2: Metrics for Various Product Areas
Here are 5 essential product areas that we need to cover:
Growth metrics
Acquisition
Retention and engagement
Monetization
Product experience
User satisfaction
Ecosystem/competition
Counter-metrics
Let's cover them one by one.
Growth Metrics
In the growth metrics, acquisition, retention+engagement and monetisation are important.
Acquisition metrics
Daily new users — measures the acquisition growth
Source-wise split of daily NU — Users can come from Whatsapp's old user set or can come from new users who downloaded the app and started using it.
Retention and engagement
Daily active users — NSM
Stickiness — provides a good measure of how loyal users are. DAU/MAU is a good measure which should be upward of 0.8-0.9 for Whatsapp.
NU retention — New user retention is important to measure over time. It tell you whether new users are sticking for longer periods. You can also monitor whether the retention for newer cohorts is improving.
Engagement — Total # of calls and total call duration are good to track metrics. Any change in product or market would reflect here first which makes them quite sensitive. We should note that engagement increasing isn’t necessarily a good thing. Sometimes a small uptick in call duration can also happen because people have to reconnect again and again due to some issues.
Share of engagement — Tells us how significant voice call is becoming over time for WhatsApp. Voice call DAU/ WA DAU is a good metric, along with time spent calling/ WA time spent.
Monetization — since WA doesn't focus on monetization, we would skip this one.
Product Experience Metrics
While talking about product experience metrics, you can go step by step through the user journey. In this case, the user connects the call, goes through the call, and then drops it off.
Call connect rate — Call funnel metrics would be call initiated to ringing % and call ringing to connected %
Call quality — During the call, call quality is important. The quality can be determined by latency (how much time it takes from the moment someone speaks to when it reaches the other person's ear). High call latency creates lag and confusion during a call, and you may experience it sometime. It's quite irritating. Another important quality metric is the data packet loss rate. Any data packet loss leads to the omission of a few words or syllables in between. This can also add to the confusion, causing users to repeat themselves, hence a bad experience. The third call quality metric can be disturbance duration as a percentage of overall call duration. This can be derived algorithmically from data loss events, reconnection events, etc.
Call drop analysis — It's important to look at the reasons of call drop. There could be three broad reasons — network issues, system errors, and the user disconnecting the call. The last one (user disconnect) is desirable, but the first two are not.
Data usage per minute of call — A lot of users are data-conscious so this one is another good product experience metric.
User Satisfaction Metrics
The user satisfaction metric can be the simple CSAT score that consumers submit at the end of the call. However, not every user rates the call after the call. So we have to track coverage as well.
Coverage — % calls rated
CSAT — can be calculated by (Number of 4 and 5 responses) ÷ (Number of total responses) x 100 = % of satisfied customers
Ecosystem Metrics
Ecosystem metrics track how the overall market is growing with respect to your product. This is important to measure because even if your growth rate is 100% MoM if the market growth rate is 200% MoM, you are growing sub-optimally. There are two important metrics to track when it comes to voice calling.
WA market share of voice calling market — This can include all forms of voice calling - telecom, voice calls over the internet, satellite calls, etc.
WA market share of the internet voice calling market — This would be a measure to get a sense of whether WA is a category leader in the internet voice calling market.
Counter-Metrics
Counter-metrics help us create guardrails so that WA overall experience doesn’t suffer because of voice calling. Two counter-metrics can help with that.
Effect of voice calling on overall CSAT — If users don't have a great experience in Whatsapp calling, they might move to other options which offer both chat and calling, like Messanger.
Chat usage reduction — The jobs to be done for both chat and call are the same in most cases - secure and effective communication. So voice calls can cannibalize chat usage for WhatsApp. We should track this metric as well.
We should be cautious while measuring the two counter-metrics as both are harder to measure. We can find a good correlation in both cases from the data. But causation would be harder to establish without an a/b test.
Summary
To summarise, thinking on how to answer this question well can make you a really good product manager. Just for fun, here is another one
"What are the metrics you will measure for Instagram reels?"
Hit the like button for the post if it was useful/actionable :)