Product Metrics That Matter
The power of metrics entered Product Management to change it forever.
Before product metrics, we were shooting blindfolded. We just understood the result of our shots at the end of the game or even the championship. Feedback and learning took too long.
We were relying on business metrics. Consider revenue, for example. That’s a lagging success indicator. It might take months for a decrease in user engagement to be reflected in a decrease in revenue. However, we could only be reactive about the lost revenue — that was the only visible indicator.
Since then, connected devices have enabled an unprecedented volume of data, and there were significant improvements in analytics tools. As a result, product metrics quickly got established. Now, we can understand user behavior at a grand scale and be proactive about a decline in engagement months sooner. We know the results of our shots right away, and we can adapt immediately.
Product metrics allow measuring product progress and creating alignment in an outcome-oriented way. They developed in us an appetite for experimenting and being data-informed/inspired. If we don’t use product metrics, we’re at a disadvantage. We’ll understand less about users, be prone to opinions, take unnecessary risks, work harder instead of smarter, and have less understanding of our work’s impact.
There’s no going back.
What to Measure
As now it’s so easy to know what our users are doing by tracking events (actions taken in our product) and properties (details attached to either users or events), our first intuition will be to measure all the things. So we’ll set dashboards all over the place with flashing numbers. I’ve done it, and I felt all important by being capable of dealing with such data density.
However, our job isn’t to measure things. Our job is to change the product for the better — to create value for the customers in a viable way for the business. Any product metrics dashboard is just an internal tool to help us do that. We have to make it as easy to navigate and valuable as possible, for us and for anyone we’re collaborating with. If we report on everything, everything will become noise.
That’s why we need to define key metrics — KPIs (key performance indicators) or KEIs (key experience indicators), as I’ve seen labeled as well. They’re a smaller data set that is good at helping us make decisions and take action, not just vanity numbers.
There are many product frameworks available to help us think about the right key things to track. I’ve summed up my favorites:
☠️ Pirate Metrics or, as a pirate would say, AARRR. This framework suggests that we follow the user through their product’s journey and track the success of the different phases.
❤️ HEART. This framework was created to address also how users interact with the product and how satisfied are they.
⭐️ North Star. This framework is a bit more radical and claims that we should distill all the product noise to the metric that matters most: the leading indicator that best captures the relationship between the core value delivered to users and the long-term business results. Still, this framework also includes 3 to 5 complementary inputs that most directly affect the North Star metric.
The key things to track will always depend on the product. Nonetheless, we may (and should) use insights from these three frameworks or any other we find appropriate.
A Closer Look
To have an example to look into, I’ve put together some standard metrics for a generic product. I went through the areas mentioned by ☠️ Pirate Metrics and ❤️ HEART and placed its key things to track in a tree-like architecture from ⭐️ North Star. Some metrics will be more visible than others, and that’s good. We’ll keep our analytics dashboard with the most impactful things first, yet we’ll still be able to deep-dive when needed.
Let’s then deep-dive and have a closer look at each of the metrics.
Acquisition
To track the success of the Acquisition phase, we need metrics that tell us about the new users and how they found the product. From the Product Management perspective, acquisition metrics indicate how well we communicate with the marketing team and whether our product resonates with our target audience.
- The Acquisition Rate is to know how representative are the new users in the whole cake of users.
- The Acquisition by Source(s) measure(s) the effectiveness of marketing initiatives designed to attract new users.
Activation / Adoption
To track the success of the Activation or Adoption phase, we need metrics that tell us how quickly users start interacting with the product or a particular feature. Here we should focus on new usage only (as opposed to Engagement).
These metrics will be extremely helpful in identifying whether or not what we built is promising. When people are quick to try something out for the first time, it means they care about the problem it’s set to solve.
- The Activation Rate measures how many of the users who signed up for the product started interacting with it and converted into active users.
⚠️ We must define what counts as starting to interact with it: e.g., performing the core action for the first time, or performing a queue of multiple actions. It’s up to us and what makes sense to our product. - The Adoption Rate measures how many users started interacting with a specific action/feature.
- The First Time Rate is to understand how many of the users who engaged with an action/feature are doing it for the first time.
- The Time to First Action is to know how quickly users start interacting with that action/feature.
Engagement
To track the success of Engagement, we need metrics that tell us how often users interact with a product or feature. These metrics should be unbiased and behavioral and therefore trustworthy, valid, and reliable.
Here, it’s essential to measure specific experiences instead of measuring the overall. And, if we don’t measure everything, we need to measure the most important things first. Then, when choosing those specific experiences, we should go for core ones — we need to think about why users use our product and the key behavior that defines that usage. For Amazon, it’s purchasing; for AirBnB, it’s booking nights; for What’s App, it’s sending messages; and so on.
⭐️ Engagement with the core action is so critical for businesses that most Product Management teams choose a related North Star metric. The more users interact with a product, the most likely they are to stay retained, and the easier it will be to turn that into revenue. It’s a leading metric, the score of every game. If users don’t do it, the whole thing falls apart.
- The Active Users is to know how many of the users were active on the product.
⚠️ We must define what counts as being active. In this case, it’s a user that performed the core action at least once. But it’s up to us and what makes sense to our product. - The Users per Action is to know how many users used a specific action/feature.
ℹ️ As, on the scheme above, I was counting active users as those who did a core action, the number of users of the core action will be the same as the active users of the overall product. - The Actions per User and Time Between Actions are to understand how often a user uses something and track the time between their usages.
Task Success
To track the success of a Task/Action, we need metrics that tell us how easy it’s for users to perform that action. It’s about usability.
- The Success Rate measures how many actions reached their end state. It measures effectiveness.
- The Time on Action and Lostness are to know how quickly the action can be performed and understand if users get lost while performing it. They measure efficiency.
⚠️ As we’ve no idea what the users’ motivation was while performing the action, it’s hard to make sense of these numbers alone. E.g. Time on Action: maybe they needed to go to the toilet, or they’re in the middle of a phone call. Nevertheless, we can use those numbers to know when to debug after they suddenly worsen or when improvements were successful after they got better.
💡 To get accuracy, we can make an environment where we create the motivation — give a task to a user. This way, we know what they wanted to do because we asked them to.
Happiness
To track the Happiness of users, we can look at Referral and Satisfaction. To track Referral, we need metrics that tell us how often users recommend the product. To track Satisfaction, we need metrics that tell us how satisfied users are (duh!).
These metrics are interesting to measure but tricky. As they’re self-reported, emotions and bias come to play. Yet, if combined with behavioral metrics, they can provide valuable insights.
- The Referral Rate is to know if users are happy enough with the product to recommend it.
⚠️ NPS (Net Promoter Score) is the most known way to measure Referral. “How likely are you to recommend to a friend or colleague? 1–10”. Yet, it measures intent rather than behavior and, for that reason, it doesn’t have many fans.
💡One of the rules of talking with users is to ask about the past. What we could ask them, instead of the NPS question, is: “In the past two weeks, have you recommended the product to anyone? Yes/No”. An even better way might be to give users an easy way to recommend the product and track how many do. - The Satisfaction Rate and Satisfaction Score are to measure how many users are satisfied enough to say it and understand the satisfaction level of most users.
⚠️ We should measure Satisfaction in context. Sending a survey a week after an experience won’t get us truthful answers — the user’s perception will have tampered.
💡If relevant, we can put a simple rating system at the end of an action, like those that appear at the end of a video call.
Retention
To track the success of Retention, we need metrics that tell us about how long users keep using the product or a particular feature. From there, we can identify what users value and why users leave. It’s crucial to retain users because, if nothing else, it’s much easier to sell to existing ones than to acquire new ones.
- The Retention Rate is to measure how many users keep using the product.
⚠️ We must define what counts as use: e.g., logging in, performing the core action, or remaining subscribed. It’s up to us and what makes sense to our product. - The Retained Time is to know how long users keep using the product. We’ll need to work to improve its extension.
- The Upgrade Rate is to understand when users choose to upgrade, in case the product has tiers.
ℹ️ Some teams measure Churn instead of Retention. It has the same goal, but it has half-empty glass lenses. I’m a half-full glass person.
Revenue
Revenue metrics tell us how much users are paying or any other way we might make money with our product. So they’re essential to test different pricing strategies or tactics to increase revenue from existing customers while ensuring the business remains healthy and sustainable.
- The Revenue per User is a way to measure that. There are many others — it’s a very business-dependent metric.
Selecting product metrics that matter is a journey, not a destination. Just as the product itself, product metrics need to be iterated. We should start with a small data set and then adapt according to the outcomes we need to measure.
Without product metrics, we can’t be outcome-oriented. Without being outcome-oriented, we can’t be agile in its proper meaning. Finally, without being outcome-oriented and agile, we can’t manage our products in an efficient and modern way.
The power of metrics entered Product Management to change it forever, and there’s no going back.
This was a long article. Thank you for reading this far. If you liked it, you’ll probably like the articles below too. They’re shorter, I promise!
