Design by Asif Jamal

I have repeatedly made this mistake of looking at the wrong metrics for the new products that we released. The realization of these mistakes made me seek mentoring from some of the accomplished product folks.

I wish I had reached out for their mentoring even earlier to grow faster.

Young product managers make the same mistakes all the time. So, here’s a very simple ABC framework that I use to decide the product metrics to look at, for measuring the success and failure of a product.

The ABC of Product Metrics

#1 Adoption

These metrics will tell you if your product/ feature is being used by the users or not. For non-interactive (communication changes / UI) features, you would want to put the immediate next action that the user does as the metric and observe the change in that metric.

These metrics would include what % of active users are interacting with the product, the product’s own funnel, etc.

#2 Business Impact

While being used by the users, your product should have a tangible impact on your key business metrics. Often, there’s a cannibalization effect- a new feature engages a set of users who would have otherwise engaged with the existing features as well.

An A/B test with every feature release and measuring those business metrics for both the control and test sets will tell you if your feature actually created an over-the-top impact.

These metrics would include overall platform conversion, engaged users, revenue, etc.

#3 Change in user behavior

Often, a bi-product (and sometimes the core strategy) of your most impactful feature will be a change in how users engage with your product. These changes are slow and hence you would want your A/B tests to run for a longer period of time to observe and measure them.

Mostly, these metrics would be around brand trust/ NPS, retention, and Lifetime Value (LTV).

Let’s look at some examples to understand this:

Social Product: Change in new user onboarding for a community platform to easily select my interests

Definition of an engaged user: Created content in the given time period.

Assume a four-step onboarding funnel:

1. First Time App Launched
2. Interest Selected
3. Home Screen Visited
4. Content Created

Let’s assume that we have made changes to the Interests selection screen. We have set up an A/B test to measure the impact of these changes. The goal of these changes would be to maximize the % of new users creating content.

Now let’s decide the metrics needed for this feature to measure its success or failure:

Adoption Metrics:

1. Increase in Interest Selected to Home Screen Visited conversion

Business Metrics:

1. Increase in First Time App Launched to Content Created Conversion
2. Increase in Weekly Engaged Users

The difference between the above two metrics of test and control sets in your A/B test will tell you the absolute difference, your changes have made, in the platform’s engagement

Change in user behavior:

1. Week 0 to Week 1 retention of new users
2. Long Term Retention (Flattening of retention curve) say Week 12 (After ~90 days)

If changes on interest selection screen set the right value and/ or experience expectations for the test set users, we should observe a higher retention for the test set.

Ecommerce Product: Giving users an option to switch to a higher quantity of the same product to get a better price per unit

Definition of an engaged user: Placed an order in the given time period

Assume a three-step switching funnel

1. Product Viewed
2. Higher Quantity Communication Clicked → Higher Quantity Shown
3. Higher Quantity Product Added To Cart
4. % Users in test set who added Higher quality product

The broader platform conversion funnel would have the following steps:

1. App Launched
2. Product Added To Cart
3. Address and Delivery Details Added
4. Order Checked Out

The primary business goal of this feature would be to increase the checkout amount (popularly known as Average Order Value- AOV)and hence increase the topline. Again, we have set up an A/B test to measure the impact of this feature.

Adoption Metrics:

1. Product Viewed to Higher Quantity Communication Clicked conversion
2. Higher Quantity Shown to Product Added conversion

Business Metrics:

Since the user will be switching from a lower quantity item to a higher quantity item, the amount for which the user would transact, should go up.

Interestingly, it is likely that the communication might attract more users which may lead to more users adding the product or might increase the information overload which may lead to users dropping off. Hence the overall platform conversion should also be observed in this case.

1. Increase in Average Order Value or Average Revenue Per User (ARPU)
2. Change in App Launched to Order Checked Out Conversion

Change in user behavior:

Realization of getting better prices may positively impact the users’ perception that the platform and can make them want to keep coming back and buy more. Hence the difference in retention between the test and control set should be looked at.

1. Increase in retention in 30 & 90 days of release
2. Increase in LTV of users in 90 days of release

Subscription Product: Adding a new benefit to attract more users to buy the subscription

Definition of an engaged user: Used a subscription benefit in the given time period

Assume your platform has free members as well as paid members. You have just decided to add a new benefit to your paid subscription product that would attract more free users to buy the subscription. Here’s what free to paid membership conversion funnel might look like:

1. App Launched (by free user)
2. Subscription Benefits Communication Clicked -> Benefits Details Screen Visited
3. Subscription Bought

You have exposed the new benefit to the test users in your test set

Adoption Metrics:

1. App Launched (by Free user) to Subscription Benefits Communication Clicked conversion
2. Benefits Details Screen Visited to Subscription Bought conversion
3. % of paid users using the benefit

Business Metrics:

1. App Launched (by free user) to Subscription Bought conversion
2. Increase in engaged users between test and control set
3. Increase in Average Revenue Per Active User (ARPU)

Change in user behavior:

The addition of a new benefit will also make the subscription more valuable for existing users which should drive up their retention and LTV (if the user needs to transact as a paid member e.g Amazon Prime)

1. Increase in paid member retention (and hence platform retention as well)
2. Increase in LTV/ paid member (in case of transactions after buying subscription)

These examples present a simpler version of the instances that PMs usually come across. Some metrics might be added to these depending on the different nuances in the implementation of the product or feature.

Takeaway

The one key takeaway from this piece would be:

We feel good at improving the local adoption metrics which may not improve the overall business metrics and may not impact user behavior.

Look at the overall performance of your product in the broader context of the entire user journey and behavior.

Awesome! Now you know the ABC of product metrics.