Measuring Product Health

A good product fulfills a deep-rooted, real human need. It is delightful, instills trust, and makes people want to be associated with it and become strong brand ambassadors. Healthy products exist on a wide spectrum: they can be enterprise or consumer, focus on specific verticals that provide a niche need, provide utility or improve efficiency, etc. This post explores multiple dimensions of healthy products with a particular focus on how consumer companies should measure aspects such as growth, retention, stickiness and engagement.

What is working in your product and what is not? Are you growing well? Are users being retained? Is your product sticky? Are your new cohorts doing as well as your old cohorts? Are people engaging with your product? Is there a core set of users who care about it deeply? Good products grow well as they penetrate their market, retain and engage their users, and bring users back regularly with a healthy content loop.

USER ADOPTION AND GROWTH

The total number of active users is the truest measure of your product’s impact. Driving adoption and growth is important for your product to achieve its full potential and create the most value for the most users.

It is important to understand product and user growth in the context of the overall market. How many people are using your product? What is the total addressable market (TAM) of your product? How does the number of people using the product compare with the TAM? How much of the market share does your competitor have, and how is this changing with time? How is the TAM likely to increase over time, and how much share is your product likely to have? (See Figure 1.)

It’s also important to understand specifics of growth for your product. How quickly is the product growing? Is it growing organically? Is growth primarily driven through new user acquisition? How is growth changing as market penetration increases? What is the balance between churn and resurrection within your product?

In a previous post, we discussed in detail the evolution of a product from a growth perspective. A healthy product has multiple phases of growth that resemble an S-curve (see Figure 2). Many products start with modest and shallow growth (Early phase), then growth accelerates to the point where the slope arches upward (Growth phase), followed by exponential growth (Hyper Growth phase) until it reaches maximum growth, after which growth tapers down and reaches maturity (Mature phase), where there is little to no growth.

If the market creating demand for the product is not large enough, it is very hard for the product to succeed. Furthermore, a healthy product requires a large market that is growing and not shrinking — otherwise, failure is inevitable. In a market that is still developing, timing the evolution of the product so you are growing along with demand is vital for success. This is often very hard to achieve, and many products fail because of timing.

How do you know if you are growing at the “right” rate? It’s valuable to benchmark your product with other products at the same stage. Knowing how fast similar products grew at the same level of market penetration can give you a sense of whether your product is special.

As your product starts to grow, more and more people will install it, and with time, installs will become a larger and larger percentage of your addressable market. Therefore, it’s very useful to understand how many installs or downloads your product has overall.

In a product’s early stages, the number of installs is typically very small, all the net growth is driven by new users, and there is no churn or resurrection. On the other end of the spectrum, when a healthy product reaches maturity and has a high degree of TAM penetration, churn cancels out resurrection and the number of active users stays steady with time. This steadiness implies that each of the old retained cohorts have more or less flattened out.

For anything in between these two extremes, growth depends on the balance between new users, resurrection and churn. By the time your product reaches maturity, it is too late for major changes, so you want to ensure it is growing sustainably while you are in the Growth phase.

MARKET METRICS

MAU/installs To monitor growth, it is useful to track both your monthly active users (MAU) as a percentage of overall installs and the total number of installs as a percentage of your market.

If the total number of downloads for your product is beginning to reach high penetration of the TAM, then your MAU/installs should be relatively high. As installs approach maximum TAM penetration, new user acquisition is no longer a tactic for incremental net growth, as there are fewer users left to try your product. As a result, net growth must come from resurrecting previously churned users. In other words, retention of your older cohorts needs to improve over time in order to produce net growth. This is generally difficult. If the ratio of active users to installs is getting smaller over time, your product is losing steam. For very strong products, MAU/installs grows slowly over time as older cohorts begin to resurrect.

Keep in mind, different genres of apps have very different behaviors. For example, all games have a shelf life, regardless of how popular they may be at their peak. In the case of the app Empires & Allies, a huge number of people installed (see Figure 3), and many became active users. However, the game also churned a large number of users. Even though more new users joined, the churn was too large, and MAU declined rapidly. Only two percent of the downloaded population remained active at steady state, and very few new users were incrementally added. As such, the game is now almost dead, used only by the most die-hard fans.

GROWTH METRICS

MAU/WAU/DAU The top growth metric you should track is the number of active users. There are three primary metrics, all correlated with one another: the total number of people who actively use the product on any given day is daily active users (DAU), the number of unique active users in a given week is weekly active users (WAU), and the number of unique active users in a given month is MAU. Choose your top metric based on how frequently you expect the product to be used.

D/D, W/W, M/M, Y/Y changes Understanding how the number of active users changes day-over-day (D/D), week-over-week (W/W), month-over-month (M/M) and year-over-year (Y/Y) is also extremely valuable. While a product’s DAU may be volatile, tracking its rolling seven-day average may be more meaningful. In general, it is best to use rolling metrics as opposed to calendar-based metrics. For example, comparing MAU between February and January is misleading, as the months are different lengths; a 28-day rolling MAU would be preferable. Comparing 28-day rolling windows has the additional advantage of avoiding a day-of-the-week effect that could skew results.

Quick Ratio The growth accounting framework helps us understand which factors contribute most to growth by splitting growth into new users, resurrected users, retained users and churned users. Between any two timestamps (t1 and t2), the change in growth = new users acquired within this time frame + users who were not active at time t1 but came back by time t2 — users who were present in time t2 but not there in time t1.

Δ growth = new users + resurrected users — churned users

When the Quick Ratio, which is the ratio of the sum of new and resurrected users over the churned users, is greater than one, there is positive net growth; when it is less than one, active users decrease month over month. For the most part, when a company has reached maturity, resurrection and churn roughly cancel each other out and much of the net growth happens through new user growth. For saturated markets, e.g., Facebook in the United States, new user acquisition is extremely small and growth remains relatively constant, with churn and resurrection balancing each other out.

New users/MAU For companies at a very early stage, all growth comes from new users, and there is no churn or resurrection. Soon after, new users continue to be the primary driver of net growth and contribute a large, but decreasing, percentage of overall MAU. As the company matures, new users’ contribution to net growth shrinks, and the retained population becomes a larger percentage of overall MAU. For unhealthy companies, new users sometimes remain the largest driver of growth even after the product reaches a high degree of market penetration; this is a death spiral. Once a product reaches a high percentage of the TAM, a smaller percentage of growth should come from new user acquisitions for the product to be sustainable.

Sign-ups/installs Acquiring new users and retaining existing users are the two most important product focuses for consumer companies. Early in a company’s life cycle, it’s particularly important to acquire, retain and engage new users. If the product is retaining well and users are deeply engaged, it makes sense to focus on increasing new user growth. This acquisition can happen through multiple channels and strategies: contact importers, social media, ads (banner, mobile app install, video, etc.), incentivized installs and referrals, SEO, ASO (app store optimization), email marketing, PR and press, app cross-promotion, etc. Knowing where your users are coming from and understanding the effectiveness of each channel is important to prioritizing your efforts. Be sure you understand every part of the acquisition funnel and identify what is working and what is broken, including invites sent and received, clicks, downloads, installs, sign-ups, etc.

GROWTH CONSIDERATIONS

Possible segmentations It’s useful to monitor growth by multiple dimensions depending on what is most meaningful to your business: country, device, age, gender, phone year class, connectivity class, platform, age in product, etc. Keeping track of the referrer source (paid, organic, SEO, App Store) is particularly important.

RETENTION

Retention is the best indicator of whether your product is valuable and whether you have product-market fit — because it tells you whether people who tried your product liked it enough to return and use it again. Generally, the first day or week is critical for a product with a new group of users. Do they have enough content to consume? Do they have a few close friends to chat with? Do they have a mental model of what the product is?

To enable healthy retention and build a successful product, you must have a core set of users who love and embrace and care deeply about the product. When you are starting out, always build your product for this set of users, focusing on the use case that matters most to them. This involves both creating a “magical moment” in which users first “get” the product and identifying the tipping point that establishes user retention.

For Facebook, the magical moment is when a new user sees the face of a friend for the first time. For WhatsApp, it’s the first message a person receives. For Amazon, it could be when a buyer receives their first purchase or has their first interaction with the customer service team.

Once you have created the magical moment, study how users are retained. The retention tipping point is different for different products; in Facebook’s case, it was connecting to seven friends in 10 days.

For your product to grow sustainably, you need to identify both its magical moment and its retention tipping point.

Retention is also the most important lever for your product’s growth. A poorly retentive product is not sustainable in the long term — you will churn through the entirety of the total addressable market with no long-term users to fall back on. Before you invest in acquiring users through marketing or paid channels, be sure you understand any and all variables that can influence retention and help stabilize it for early cohorts. Ultimately, you want long-term retention, but early retention can be a strong predictor of long-term retention. While the absolute value of retention depends on the type of product (social, gaming, messaging, etc.) and the time frame you choose to measure (daily, weekly, monthly, etc.), it’s always true that the higher the retention, the better.

There are three basic shapes of retention curves. In the first, worst-case scenario, long-term cohort retention drops to zero and the product eventually dies. In the second scenario, the retention curve flattens out at a number greater than zero. This ensures the product has active users — the higher it flattens out, the more users you have. The third scenario is best: when a product is very healthy and has reached strong product-market fit, there may be a Hyper Growth phase in which the older cohorts start to increase their retention. This is the case in Evernote’s smile graph (Figure 4). (Eventually, at a higher rate, this will flatten out).

RETENTION METRICS

Dn/Mn/Wn metrics Retention is assessed in terms of cohorts — that is, by tracking a set of users who install a product on a given day, week or month and seeing what percentage of them return Ultimately, the goal is for the product to retain at high levels over a long period of time. However, for newer products, it is difficult to project what the long-term retention will be. Therefore, you need to identify an early indicator for long-term retention. With games, for example, the percentage of people who retain on day one (D1 retention) is indicative of the long-term retention.

A simple way to express long-term retention is to break down the number of people retained from a cohort after roughly a year (say, 364 days) into multiple ratios, as seen in the equation below. The D1 retention rate is D1/D0, the fraction of your cohort retained for one day. (D0 is the number of installers in a cohort, and D1 is the number in that cohort who still use the product after one day.)

If D7/D1 stays relatively constant for all cohorts but D1 retention appears to decline, focus your attention on improving D1 retention, as this will very likely be the biggest lever for long-term retention. Likewise, if D1 retention is flat but D7/D1 is declining, focus on finding new ways to engage users in the first week rather than the first day. Pick the metric that best aligns with your product.

Cohort curves In addition to Dn retention metrics, analyzing cohort curves is extremely important. Cohort curves will give you a much better idea of whether your retention is flattening, experiencing hyper growth or dropping to zero.

Some additional tips for healthy retention:

• Compare new cohort retention rates to old cohort retention rates and make sure they remain healthy over time.
• Benchmark retention rates against similar products.
• Ensure that cohort retention flattens — preferably at a high rate.

STICKINESS

Even if your product is growing and retaining well, it may not be sticky. While retention is about getting people to come back to your product, stickiness is about getting people to come back of their own volition, thus helping reduce your dependency on tactics such as push notifications. When a product or service is sticky, the customer becomes tied to it and cannot easily leave. For example, Facebook’s stickiness is a direct result of users’ urge to share, and of their curiosity about other people’s lives.

STICKINESS METRICS

DAU/MAU This is the most widely used metric for stickiness. The ratio of daily active users to monthly active users tells you how sticky a product is. It’s valuable to understand benchmarks for these metrics for different products and to put your product in context; for example, SMS-type apps have a much higher DAU/MAU than social apps such as Facebook. A DAU/MAU ratio of 0.6 means 60 percent of people who come to the product on a monthly basis are also coming back on a daily basis. Generally, this means the product is very healthy. Social products in the early stages of growth tend to have a relatively small DAU/MAU, perhaps 30 percent. As more users join (and market penetration increases), DAU/MAU begins to grow. In addition, newer and older cohorts alike begin to retain at a better rate. The specific value of DAU/MAU is strongly dependent on the business or product and the expected usage. If your product has a low DAU/MAU, it does not necessarily mean it isn’t doing well. While a core set of sub-products might have high product-market fit and high DAU/MAU, it’s quite possible to find sub-products within that set that are less engaging. The ratio might be low due to a mix of casual and highly engaged users. In this scenario, you should investigate options to get more users highly engaged.

Open rate This is the percentage of people who have the product installed that use it in a given time frame. For a highly sticky product like Facebook, people open the app many times per day. For less-sticky products, the open rate may be 10 percent or less on a monthly basis. Stickier apps are generally correlated with higher open rates. (However, if the product is highly notification-driven, people may still open the app even if the product is not very sticky.)

Lness This is the number of days visited in a given time frame. For example, L5+/7 is the percentage of people who visit the product at least five times per week. Similarly, L21+/28 indicates how many people come back at least 21 times within a given month. The higher the percentage, the stickier the product is. Tracking Lness distribution over time will give you a very good sense of your product’s stickiness.

Sticky or DoD/WoW/MoM retention While cohort-based retention helps us understand how each group of installers behaves, it does not tell us about the behavior of the most-engaged users. Remember, the goal is to build a product for the core users who engage with it the most. Sticky retention helps us understand how well a product is doing for the most-engaged people. It is generally measured using DoD/WoW/MoM retention, which refers to the number of people who come back to the product the following day, week or month (regardless of when they joined the product). For example, to determine week-over-week (WoW) retention, we simply take the ratio of users active in a given week to the total users active the previous week. (In order to eliminate the “new user effect,” remove any user who first became active in the past 14 days.)

ENGAGEMENT

In most cases, if your app is growing well, users are retained at high levels and you have good indicators of stickiness, your product is in great shape. But it could also be the case that people are not returning often enough, aren’t producing or consuming enough content, or aren’t spending enough time using the product. These are indicators of low engagement. For social products (such as Facebook and Instagram) in particular, time spent in the product is the simplest indicator of whether your product is engaging. The higher the time spent, the higher the engagement. True product-market fit happens when a product is highly engaging (though it also manifests itself in retention).

What drives engagement? It is valuable to create a simple framework to understand what drives engagement in your product. Social products such as Facebook, Instagram, Reddit, Snapchat, etc. have a production/consumption framework to drive engagement. A simple version of this engagement model is seen in Figure 5. Users produce content which is then shown to others, creating delight. Consumers of the content provide feedback in terms of upvotes, comments, reactions, etc., which makes users want to produce content again. This production/consumption loop creates engagement, which can be measured by users coming back frequently (number of sessions), consuming more content (number of views) and spending more time on the site.

There are multiple approaches to increasing engagement: making it easier to create different types of content, helping connect users with the most relevant content, showing them the right content in the right order, making sure people are able to consume content easily on any device and any network condition, etc. In addition, as more individuals, influencers and businesses produce content, there is a greater likelihood that users will see engaging and relevant content they love. It’s also imperative that you design your product to make it easier for people to interact with its content, as this strengthens the positive feedback loop.

Another engagement model is the e-commerce model. For example, eBay connects sellers to buyers. In this model, you can increase engagement by focusing on the quality of the inventory available, the quantity of unique listings, the relevance of items shown to the buyer, the value of the listings, connecting the right content to the right buyer at the right time, simplifying the buying process, building trust at every step of the experience, etc. Optimizing each part of the funnel should increase overall engagement.

ENGAGEMENT METRICS

Note that many of these metrics are specific to companies that fit the production/consumption paradigm.

Time spent/DAU This metric is a strong indicator of whether your product is engaging. Again, this depends heavily on the specific product and expected engagement. Market share of time spent is also a good indicator of whether you are doing well from a market perspective.

Number of sessions Number of sessions in a given day is another good indicator of whether your product is engaging. An increase in this metric is also the earliest indicator of whether you are achieving product-market fit. (Similarly, a decrease in number of sessions is the earliest warning sign that something is going wrong.)

Time spent/session Both number of sessions and time spent/session are levers to grow overall time spent. It is valuable to understand whether you can more easily increase overall time spent by increasing the number of sessions or by increasing time spent/session.

Inventory available For companies that have news feeds (ByteDance, Instagram, Facebook, LinkedIn, etc.), users can consume content only if there is inventory for them to see. Knowing how much inventory is available and how it is distributed helps you understand how content can be monetized.

Content consumption (number of views) The metric of time spent is often skewed by types of content that are inherently time-consuming, such as videos. Therefore, it is helpful to understand how many pieces of content people consume relative to the inventory. Are some people inventory-constrained and thus unable to consume content?

Production of content How much content does each user produce every day, week and month? How many of them produce content on a regular basis? The total production of a product is based on the number of users who produce content and the amount that each produces. For ecosystems in the production/consumption paradigm, the production of high-quality content is the most important metric for an engaging ecosystem.

Feedback on content Key feedback metrics to track are likes, comments and reactions. Likes are a weaker form of feedback than comments. Regardless, all feedback is important to the ecosystem. Without it, production will decrease, which will ultimately lead to a drop in consumption.

ENGAGEMENT CONSIDERATIONS

Possible segmentations Consider segmenting your metrics by country, device, age, gender, phone year class, connectivity class, platform, age in product, content format (video, picture, text, etc.) and content type (social, entertainment, informational, educational, etc.).

Leading and lagging indicators Imagine you start out using a product multiple times per day (multiple sessions) and spending a lot of time with it. Over time, you start getting bored with the product and your use drops off — but you are still an engaged user in many ways, visiting the product many times per week and more than twenty times per month. At this point, DAU, WAU and MAU have not yet changed.

Eventually, however, as you use the product fewer times per week, you will continue to register as a WAU and MAU but less as a DAU. Over time, you will no longer be a WAU, and then not a MAU, either. And chances are, you are not alone; many other users are likely getting bored, too. At an aggregate level, these types of behavioral changes first manifest in engagement metrics such as the number of sessions. It’s likely that as your use of the product changes, first you will consume fewer items of content, then you will have fewer sessions and read fewer stories each time. A drop in number of sessions is the earliest leading indicator for a drop in DAU. Similarly, a drop in DAU is a leading indicator of a drop in WAU and ultimately, MAU.

Engaged/power users Your most-engaged users are the most important people to take care of. Keeping them happy is not really a growth concern, as they are not likely to churn. However, as mentioned in the discussion of retention above, it is ideal to target your product toward core users by making the experience easy for them and delighting them. Many of the metrics defined in this post can and should be segmented for the most-engaged users. Is the number of engaged users growing? Is their number of sessions and amount of time spent increasing? Are less-engaged users becoming more engaged? How many friends do these “power users” have? How much content do they produce? What features of the product do they use? Answers to questions like these can help build great experiences for your most-engaged group.

PRODUCT MYTHS

Good products don’t have troubled times The truth is nearly all products have problems with churn, new user acquisition and/or engagement. But if your core users are still engaged, and a reasonable number of them are deeply engaged, you have achieved product-market fit. And when you have product-market fit, there are ways to get your product back on track and grow your base of engaged users.

Growth tactics can solve anything While growth tactics can be very helpful if you already have product-market fit, you cannot use them to solve problems with product-market fit itself. The product must deliver value to users. For example, BranchOut’s meteoric rise was due to growth tactics, and their subsequent fall was due to not having product-market fit.

Grow quickly at all costs The truth is growing sustainably is far more important than growing quickly. Growing with low retention will eventually kill your product.

Build a product for everyone It’s a common misconception that you should try to satisfy all your users. But building a product for everyone will almost certainly lead to building a product for no one and will ultimately hurt the product. Build your product for your most-engaged users and ensure they have a great experience.

As long as the retention flattens, you are fine A high absolute retention is important for the long-term sustainability of your product. Yes, retention to a low number is better than nose diving to zero, but it still may not be sustainable.

TAKEAWAYS

Saturation of a market will slow down growth. Growth metrics should be considered in the context of the Total Addressable Market (TAM).

Retention is the best indicator of product-market fit and the most important lever for growth. “Magical moments” help your users understand the core value proposition of your product, and getting users to a “tipping point” will help you retain them.

Engagement is the most important driver of retention. Creating a framework for what drives engagement in your product will help you identify leading indicators for changes in user behavior and inform interdiction techniques.

This article, which was previously published on Medium, is a product of Sequoia Capital’s Data Science team. Jamie Cuffe, Avanika Narayan, Chandra Narayanan, Hem Wadhar and Jenny Wang contributed to this post. Please email data-science@siteadmin.peakxv.com with questions, comments and other feedback.