Over the past couple of months we’ve written about how we approach the quantitative part of doing diligence on potential investments at Social Capital. This covered several things including growth accounting, our approach to cohorts and LTV, and how we think about the depth of engagement in a given product. While these frameworks were introduced in the context of due diligence, we also described their utility in an operational context. Today we’ll do two things: 1) give some insight into what motivated us to write these pieces and 2) introduce a tool that we’ve built to help you conduct these analyses on your own business/product/feature.

The 8-Ball

The genesis of this entire project came about in late-2014 when we collectively came to realize that, in diligence, we were not seeing core product-market fit at a resolution that we were satisfied with. From our collective time at Facebook, we’d grown accustomed to a sophisticated collection of product metrics and we found ourselves wanting to use those same frameworks when evaluating startups. After doing these analyses ad-hoc a few times and significantly expanding the scope to include non-social network metrics we started referring to the whole analytical package as “the magic 8-ball” or more succinctly as “the 8-ball”. The 8-ball consists of the three pieces that have been described in these posts: 1) growth accounting, 2) cohort behavior and 3) distribution of product-market fit all measured for the core unit of value for the business (typically users and/or (possibly recurring) revenue).

In requesting data from potential investments to run the 8-ball analyses, it became clear to us that there would be value in packaging up a tool that would help perform the analyses. For the cohort-LTV analysis in particular, our heat map and LTV trend views were usually new for entrepreneurs and we typically ended up creating them ourselves and doing a lot of explaining along the way.

So we hired an intern from the Waterloo co-op program, Peter Zhang, and had him build version zero of the 8-ball tool to help us produce the analyses. I invite you to check it out here and give it a spin. If you’re coming in to pitch your company to us, feel free to use the tool to help generate the graphs that we’ll invariably ask for in our own diligence process. In it’s current iteration, the tool only addresses the cohort-LTV aspect of the 8-ball. In the future we’ll build out more features to provide the other standard analyses that have been described in this series of posts.

GAAP for Startups?

Beyond it’s usefulness in diligence, the 8-ball actually addresses a broader need in the startup ecosystem, namely, the lack of standardization in metrics and reporting. To elaborate on this, let’s first take a digression into financial metrics. In the world of financial reporting there is a set of standards that are used to communicate the financial health of a company. These standards are known as GAAP (Generally Accepted Accounting Principles) and their existence was driven by the desire from investors to be able to compare and contrast different companies in a standard fashion. There are other metrics beyond GAAP financials that have gained general usage through ecosystem dynamics. For instance, the notion of MAU (monthly active users) was popularized by the growth of Facebook. Prior to Facebook it was standard to measure pageviews or cumulative registrations. When Facebook launched the Platform it popularized the notion of MAU and DAU thus driving them to become standard language across the entire ecosystem of companies and investors. Similarly, MRR (monthly recurring revenue) is a relatively recent concept in the world of technology investing. It was the explosion of cloud driven software-as-a-service such as Salesforce (also in the mid-aughts) that made MRR a standard way of discussing subscription revenue for SaaS businesses.

For very early stage companies the investor is really looking for evidence of product-market fit and GAAP financial metrics don’t have much to say about the nature of that fit. As such, there is currently no standard way to compare the degree of product-market fit across these early companies.

We would propose that the 8-ball analyses that we have articulated in this series of posts comprise a reasonable GAAP-like standard for understanding product-market fit for early stage companies.

The actual emergence of GAAP is, of course, more complicated than what I mentioned above (in particular with regards to the involvement of regulatory bodies, see this paper for a short history on the topic) but the early stage startup equity market is clearly growing fast (especially with the recent enactment of Title III of the JOBS Act) and much more collective time is bound to be spent on assessing early stage product-market fit. We hope that by sharing our views on this topic we’ve helped inform that discussion and hopefully helped you think about your company, product, feature, investment, or portfolio in a slightly different light.

[Note from the author: See an update to the thinking presented in these articles here. I no longer work at Social Capital, if you want to reach me you can email me at jonathan@tribecap.co]

Appendix: One Big Query

In order to use the 8-ball tool, you will need as input a CSV of aggregated data that you produce from your own data store. Most companies store this data in a database of some sort and you’ll need a query to get the data out.

For the vast majority of companies that we work with the relevant data lives in either a transactional database or in a analytics data warehouse that has a SQL interface (most often Amazon Redshift). Piecing together the quantities in growth accounting or pulling together a complete cohort view of the usage can be tricky so we’re providing a complete SQL query below that captures both the growth accounting and the cohort-LTV data.

The query is rather long and is shown below. It’s written in Postgres and so does the heavy lifting in a series of WITH clauses. It also runs on Amazon Redshift. If your version of SQL doesn’t support the WITH clause (i.e. MySQL) you can separate each one out as a temp table. For the purposes of the example, it uses one of the tutorial data sets provided by Mode. You can run this query directly in Mode’s interactive SQL editor on the tutorial data sets that they provide (feel free to sign up for a free account at Mode).

To apply this to your own data, you should only have to modify the first temp table in the WITH clause and choose which of the final queries to use whether you want to output either variant of growth accounting or the cumulative LTV results. The current 8-ball tool only does visualization for the cohort LTV data.

Diligence at Social Capital, Epilogue: Introducing the 8-ball and “GAAP for Startups” was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the first two parts of this series we discussed growth accounting and applied it to both engagement and revenue. In the third and fourth parts we discussed lifetime value (LTV) in businesses that generate revenue and then applied the LTV framework to retention and other engagement related value. The frameworks covered so far give a good sense of the underlying mechanics of top-line growth. Today, we’ll cover a somewhat orthogonal question that we often ask in diligence, namely, “What is the distribution of engagement across the user-base?” and the related revenue question “How solid is the revenue stream?”

The Engagement Case

Consider the consumer app case. One area that isn’t covered by the previous approaches is the depth of engagement. Start with a monthly active user (MAU) growth accounting (as covered in Part 1 of this series) for some app that shows some solid month-on-month retention with reasonable churn. With the frameworks we’ve so far covered we don’t currently have a good way of separating out those MAU into highly engaged users vs. marginally engaged users.

To quantify depth of engagement in a consumer app, the most generic per-user metric would be “days active in the month”. Recall the discussion in the second part of this series where we introduced the concept of L28. A user of L28=10 was active 10 days of the last 28. If you sum up L28 across all users in a month (as measured on the last day of the month) you get the total sum of DAU across the month. Each user’s L28 is an example of value being created and thus you can do growth accounting on the total L28 from month-to-month. The previous post on non-revenue LTV is essentially a cumulative lifetime measure of L28. Let’s take a single 28-day MAU period (to avoid day-of-week effects) and look at an example L28 distribution:

For concreteness, let’s pretend that the app has 100k MAU. The L28 distribution is now telling us that fully 34k of them are active only one day in the month. It also says that that a small chunk, 3k, are super engaged using the app every single day of the 28-day month at L28=28. This distribution provides a natural segmentation of users into “not engaged” vs. “medium engaged” and “highly engaged” which can be used for other analyses. This view is good but it turns out to be easier to read as a cumulative distribution function (CDF). Here’s the CDF of the above.

The blue line shows the CDF of users. This tells us a few things. For instance, it says that 50% of MAU are L28<=2. It also says that the top quintile of users (the top 20%) are L28>=5. The top 10% most engaged users are L28>=15. If the app is really one that is supposed to be used once per week then it would be reasonable to say that a given user has achieved product-market fit if they are L28>3. In that case, 33% of the MAU have truly hit product-market fit and the other 67% are not engaged enough for us to really believe that the app is delivering the desired level of value.

The green line is a slightly different measure. This is the CDF of “active days”. To make sense of this, first note that if you add up the L28 variable across all MAU then you get the same number as if you had added up DAU across the whole 28 day period (this may not be obvious, feel free to work it out). This is basically the same as averaging DAU across the whole period.

∑ DAU(t) = 28*average_DAU = ∑ L28(u, t_end)

Where the first sum is over the 28 days ending on t_end and the second sum is over all users u in the MAU set measured exactly at t_end.

For concreteness, considering our 100k MAU example, this data would be showing a total of 500k summed DAU across the month which would be an average of 500k/28 = 18k DAU through the month. The CDF of days is telling you how much of this summed DAU figure is made up of each L28 bucket. For instance, it says that 7% of active days are made up of the L28=1 users. While these L28=1 users made up 34% of users they only make up 7% of average DAU because they don’t contribute to DAU as often as higher L28 users. Recall that we decided to declare L28>=4 as users who have hit product-market fit. In this example, such users comprise 25% of users but are contributing ~72% of total DAU (and hence average DAU). People often talk of the Pareto principle which roughly states that 80% of the value comes from 20% of the participants. Such an 80/20 for the visitation aspect of our example product is 72/25 which is not too far off. There are fancier ways to encapsulate this notion that don’t demarcate the 80% line as special (i.e. the Gini index). However, I find such measures harder to interpret and tend to prefer some combination of “80/20” measurements.

One common measure of engagement is the ratio DAU/MAU. This is actually related to the L28 framing. If you interpret “DAU/MAU” as average_DAU/MAU(t_end) then this is really 28 times the “average number of days active in the last month”. Since sum of DAU is the same as sum of L28 this is just the mean of the above L28 distribution. The proposal here is to replace a single measure of the distribution (the average) with a view of the entire distribution of contribution to DAU. In general, the mean provides a good description of a distribution when it is close to normal. In this case, the L28 distribution is not even close to normally distributed so the mean is not as indicative of a typical experience as, say, the median or other percentile measures.

The L28 approach is also useful because it gives you a sense of the relative value of different users. If we believe that a user who is L28=5 is 5x more “valuable” than a user who is L28=1 then the L28 distribution is really the distribution of “value” across the user-base. Whether an L28=5 user is actually 5x more valuable than an L28=1 user depends on your business. For instance, if your product is ads driven then the user is spending roughly 5x more time and is hence roughly 5x more valuable. If your definition of “active” means “spent money in a transaction” then this breakdown of value is clearly directly related to monetary value and the L28=5 user is clearly ~5x more valuable than the L28=1 user.

In terms of how we’d judge the above company at Social Capital, it would depend on the nature of the actual product. If it’s really a weekly use-case type of app (like, say, an app that sells tickets on the weekend) this would be pretty good. Fully 25% of users are using it every week. However, if it’s meant to be an ongoing content consumption app this is only so-so as 75% of users don’t use it once per week. This data isn’t useful without the product context. Conversely, having only product context is not very useful without objective metrics such as the ones described here.

From Depth of Engagement to Quality of Revenue

As in the discussion around growth accounting and LTV, we can take this L28 approach and abstract it to other forms of value that a customer generates. A customer may generate value by contributing DAU to your business. A customer may also generate value by spending money.

Let’s pretend that we have a B2B SaaS product that sells to companies and charges them $1 per seat per month and that may also offer some premium services on a per-customer basis. Each customer is a company so there is a distribution of how much each customer is paying. Shown below is the distribution of revenue (monthly recurring revenue aka MRR) per customer. This is analogous to the L28 distribution above.

This distribution is a bit hard to read. There are a bunch of users spending less than ~$100 and clearly a few customers who are spending a lot (either through up-sells or more seats). As in the L28 case, this data is easier to view in a CDF:

This view makes it easier to read off what’s going on. The red line is the CDF of customers and says, for instance, 50% of customers are spending less than $20. It also says that the top 20% of customers are spending are more than $63. The blue line is the CDF of total revenue and is the analogue of the CDF of total L28 from the engagement example. This says that 30% of revenue comes from customers spending less than $50. The top 20% of customers spending more than $63 are responsible for 66% of all revenue. So 80/20 for this company is 66/20 which is to say that the lower end customers make up more of the business than a typical Pareto principle would suggest.

You could imagine a couple of extreme versions of these distributions. For instance, consider the monthly revenue from Spotify. In that case the vast majority comes from users at a single price point in which case the red line would be zero until that price point and then abruptly go up to nearly 100% at the price point. The CDF of revenue would be very similar to the CDF of users and this would be a case of “20/20”. At another extreme, consider something like FarmVille or any of the other old social/Facebook games. In those cases there are a bunch of users who spend a little and a very small number that spend a whole lot (“whales”). As such, the blue line would be way below the red line as the top few percent would make up a very large portion of the revenue. This would be a case of “99/20”.You should be able to convince yourself that the blue line here is always below the red line.

For a B2B example, consider Google Cloud Services. It was rumored that Snapchat spends something like $25–30m on Google Cloud Services each year and that they are the far and away largest spender on the service. If you were to take the CDF of customers of Google Cloud and compute the CDF of customers by revenue most customers would be spending a reasonable amount each month but the blue line would be way below the red as the Snapchat outlier would be at a very high value single-handedly making up a large fraction of total revenue.

You might be wondering why I called this section “quality of revenue”. The idea is that, in the context of enterprise SaaS, customers who are spending a lot of money in a highly recurring fashion are generally higher quality customers because they are more likely to survive a downturn. The customers on the right end of the revenue distribution are the flagship customers who are getting a lot of value from the product and would likely continue being customers in a weaker macro environment. For enterprise startups it’s often the case that small customers with low contract values are other small VC backed startups. If/when funding dries up it will be these small customers that are at higher risk of going out of business or downsizing their spend and so that part of the revenue stream would be at risk. People often talk about “average contract value” (ACV) when talking about enterprise SaaS companies. For MRR driven businesses the above approach shows the entire distribution of contract values which gives us more nuanced information than just the average. We typically encapsulate such an analysis by stating that X% of the revenue stream is subject to a downturn in startup-driven spending vs. Y% that will survive because it’s coming from high quality customers.

Conclusion

Let’s review the three areas that we’ve discussed in this series:

• Growth accounting provides a framework for understanding the underlying components that drive net growth by unpacking top-line new customers from resurrection and churn. The framework applies not just to growing users but to growing anything of value including subscription revenue, visitation, posting behavior, etc.
• The textbook explanation of lifetime value (LTV) is not the most useful one for understanding early stage companies (or early stage features). Empirical N-week or N-month LTV is preferred. The idea of LTV can be generalized from revenue generation to other activities of value such as cumulative visitation, referrals, etc.
• Aggregate measures of engagement such as DAU/MAU provide a limited lens for understanding depth of engagement. A better practice is to observe the full distribution of depth of engagement. This is also true when considering the distribution of revenue generation where ACV is also a limited view compared with understanding the full distribution of contract values.

With these approaches we’ve given a pretty complete view of how we quantitatively think about product-market fit when conducting diligence on startups.

At Social Capital we also have an in-house growth team that spends most of it’s time embedded in portfolio companies helping them execute their growth, data and user acquisition strategies. The tools described in these posts are not just useful for a one-time determination of product-market fit but are even more powerful when integrated into the ongoing product development process. Hopefully you find them useful for understanding and driving your own business. In the future we hope to write more about our experience helping entrepreneurs grow their companies using these frameworks in specific contexts.

As always, feel free to comment or email me at jonathan@socialcapital.com if you have any questions.

Edit: For reference, here’s the full table of contents.

1. Accounting for user growth
2. Accounting for revenue growth
3. Empirically observed cohort lifetime value (revenue)
4. Empirically observed cohort lifetime value (engagement)
5. Depth of engagement and quality of revenue
6. Epilogue: The 8-Ball and GAAP for Startups

[Note from the author: See an update to the thinking presented in these articles here. I no longer work at Social Capital, if you want to reach me you can email me at jonathan@tribecap.co]

Diligence at Social Capital Part 5: Depth of Engagement and Quality of Revenue was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the first two parts of this series we discussed growth accounting and applied it to both usage and revenue. In the third part we discussed lifetime value (LTV) in businesses that generate revenue. In this post we will take the LTV approach from the last post and apply it to the engagement and retention aspects of a business.

In the revenue LTV case, the main lesson was to observe actual monetization throughout the lifetime of existing users and to view the trends of those LTVs. For businesses that intend to grow users and monetize them in the future the focus shifts from monetization to retention over long time periods. Imagine we have some consumer app that delivers some social/content style experience. This usually starts with a retention graph possibly like the following:

The parenthetical numbers are the sizes of the cohorts. This app shows that after four weeks weekly retention is between 15–40%. One of the cohorts (2014–03–03 in green) is exhibiting particularly poor retention and there are some odd dips in retention in some of the cohorts as the retention continues to decline over time. This is a useful view of the data as it shows us the shape of the retention falloff. As with many of these types of views it gets hard to read with too many lines so I only included roughly one weekly cohort per month. In this view it’s also hard to see the trends in the falloff so it’s often useful to look at this with a retention heatmap.

This says, for instance, that the cohort of customers who started on 2014–02–03 had 144 new users and was 41% weekly retained four weeks after their first visit. The oval at the top of the figure shows several cohorts all decaying at roughly the same rate. The second oval shows three cohorts that are doing worse than others. Looking at the cohort sizes on the left axis you can see that this coincided with a big spike in user acquisition. One of these cohorts was visible in the green line in the above retention curves. This is often what happens when a company significantly increases paid acquisition. They end up getting users with a lower propensity to engage and who thus retain much worse than other cohorts. This company stopped their paid acquisition experiments after a couple of weeks and subsequent cohorts went back to the same retention pattern as before the user acquisition bump. As time passes, data accrues on the diagonal edge thus events that occur on a fixed calendar date appear as diagonal features (see arrow). This one was probably due to some outage in the app that occurred on the week of 6/2/2014 which affected all cohorts (at different ages for each cohort). This fixed calendar date effect was the cause of the strange dips in the original retention curves. Luckily, after the outage, retention continued apace without any apparent long-term negative effects. Over multi-year timespans, seasonal effects (such as Christmas, summer break, Easter, etc) appear as repeating diagonal features.

The retention curve graph above is similar to the LTV graphs described in the last post. The main difference being that the LTV view computes a cumulative figure whereas the above retention curves compute an incremental figure. In the revenue case, incremental revenue accumulates up to an LTV figure. In the usage case, incremental retention accumulates up to a cumulative activity LTV figure. In this example we are accruing weekly retention up to a weekly active user (WAU) LTV figure. For example, say that a cohort starts out on week one and is all active and in week two 50% of them are active. This can be phrased in an LTV sense by saying that the cohort has accumulated an average of 1.5 active weeks within the first two weeks since first visit i.e. “the 2-week WAU LTV was 1.5”.

Here’s the cumulative active days LTV view of the sample retention data above.

This says that the 110 users who started in the first week of 2014 had an average cumulative 5.5 active weeks on the app after about six months. The outlier weak retention cohort appears here as having a much lower WAU LTV than the other cohorts. Similar to the revenue case, there are three qualitatively different cases of activity LTV curves here.

• Flat: No incremental visitation past a certain date.
• Sub-linear: Non-zero but decreasing visitation. This is what we’re seeing in the example. All users appear to be gradually losing interest.
• Linear: Consistent retention through the lifetime of the cohort. A core of users is consistently using the product indefinitely. The slope is determined by how much of the cohort is made up of core users vs. non-core users.
• Super-linear: Increasing retention as cohorts age. The propensity to use the product goes up as the user ages.

Since this quantity is non-decreasing, we can view it’s trends just like we did for revenue-based LTV.

As before, the bars show the cohort size for each week. Look at the 2014–04–07 cohort which consisted of ~205 users. In their first week they had a WAU LTV of 1 (i.e. all users were active in their first week of visitation). In the first month after first visit this cohort averaged 2.7 total active weeks. This cohort’s 12 week WAU LTV was just observed to be ~4.5. We don’t yet know the 12 week WAU LTV of later cohorts because they are not yet old enough. The effect of the paid acquisition experiments are clear in this view. The 2014–03–03 and 2014–03–10 cohorts were unusually large and the WAU LTVs subsequently suffered for those cohorts.

Needless to say, for engagement based consumer businesses, we are interested in situations with high retention that either holds steady or gets better as cohorts age and larger cohorts of new users are acquired. This is equivalent to saying that we want linear or super-linear cumulative activity LTVs (whether DAU, WAU or MAU).

As before, we can look at the WAU LTV trends in a heatmap. In contrast to the previous heatmap above, this one shows cumulative active weeks as opposed to weekly retention.

This says, for instance, that after 4 weeks cohorts are averaging ~2.3 total active weeks except for the cohorts that were unusually large and had lower accumulated active weeks. The colors show the cumulative WAU attained by each cohort after some specified age. The two heatmap views are complementary. The retention heatmap shown earlier is useful because it gives you a sense of what on-going retention looks like once users have aged a bit. The shortcoming is that the retention view makes it hard to detect small differences between cohorts. For instance, after a few months, one cohort might have weekly retention in the 9–10% range and another cohort might be more in the 8.5%-10.5% range. It would be hard to tell if one is retaining better than the other because of the noise in the weekly retention figures. By comparing the cumulative activity between the cohorts we can get a better sense of whether the slightly different week-to-week retention is amounting to materially different total activity with the product over long timespans. The downside to the cumulative views (whether the heatmap or the trends) is that it’s harder to see small effects such as the outage that caused a short-term dip in retention. Said another way, in this case, the one week of outage didn’t cause any lasting effects and so is not very meaningful to the cumulative activity in the lifetime of the app. This is a particular case of the general statement that integrating a time series of any sort smooths out the time series.

Lifetime Value of Everything Else

So the framework of cohorts and LTV is useful for understanding both incremental and cumulative revenue generation and is also easily generalized to incremental and cumulative retention (measured by active weeks/days/months). It should be clear at this point that this framework can be used to understand any sort of behavior that customers are exhibiting in your business. To give an example outside of retention and monetization, one can consider applying this framework to understand the dynamics of referrals. Say that you have an app that tries to get users to refer a friend. Each time a user sends a referral it is not that dissimilar from when a user decides to spend money — in both cases they are signaling some incremental bit of product-market fit. One can analyze the cumulative and incremental referral sending behavior as cohorts age and come up with an “N-week referral LTV”. Here’s some sample data showing referral LTV (really the same data as above but with different titles and meant to be interpreted differently).

The cohorts in this case are determined by the week in which they sent their first referral (although you could certainly create cohorts by week of first visit, registration, etc.). In this case, the 2014–04–07 cohort had 200 people sending their first referral who sent an average of 4.5 referrals per user within the 12 weeks after sending their first referral. There were a couple of weeks with a lot of users sending their first referral (possibly through some very aggressive referral up-selling) which caused those cohort’s referral LTV to suffer. And so on.

Such an approach is complementary to comparing bulk referral sending (whether before/after the change or measured purely as a daily difference in a controlled A/B test) because it allows the difference to accumulate over a longer time period which might be necessary if the changes are small. For instance, on a daily or weekly basis, referral sending may only change by a small amount but such a difference would be much more noticeable when comparing, say, one-month referral LTV between two groups. The overall referral LTV curve shape would be useful in understanding whether referral sending happens early in the life of a customer followed by a sharp drop-off vs. sustained referral sending through the life of a customer.

Together with the growth accounting framework, the LTV framework described here forms a big part of our quantitative diligence process. The last area that we focus on regards the distribution of value (either revenue or engagement) across customers. From the visitation point of view, over longer time spans, we don’t have a sense of whether or not it’s a small core of highly engaged users mixed with a large group of disengaged users vs. a large group of more moderate intensity. We will explore this in the next post.

Edit: For reference, here’s the full table of contents.

1. Accounting for user growth
2. Accounting for revenue growth
3. Empirically observed cohort lifetime value (revenue)
4. Empirically observed cohort lifetime value (engagement)
5. Depth of engagement and quality of revenue
6. Epilogue: The 8-Ball and GAAP for Startups

[Note from the author: See an update to the thinking presented in these articles here. I no longer work at Social Capital, if you want to reach me you can email me at jonathan@tribecap.co]

Published in #SWLH (Startups, Wanderlust, and Life Hacking)

Diligence at Social Capital Part 4: Cohorts and (engagement) LTV was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve gotten a ton of great feedback on these posts and are really excited that it’s providing a useful and valuable set of tools with which to understand your business. Feel free to email me at jonathan@socialcapital.com if you have questions. I can’t guarantee that I’ll respond, but I’ll certainly try!

In the first two parts of this series we described how growth accounting can be applied to understand the sub-components of both user growth and recurring revenue growth. We mentioned that growth accounting has a shortcoming in that it doesn’t give us a sense of the lifecycle of a customer. In particular it doesn’t help answer questions such as: “Do customers spend more early vs. later in life?” Or perhaps “Does churn occur abruptly at some point or does it steadily occur through the life of a customer?”

To get started, let’s pretend we have a business that sells something to customers. The following description is indifferent with regards to whether the revenue is subscription or transaction based. We are interested in the lifetime value of customers in our business i.e. the cumulative revenue realized per customer.

Most descriptions of lifetime value (LTV) use a model which ends up with a formula based on a combination of contribution margin (m), retention rate (r) and discount rate (d) which encapsulates the infinite time horizon LTV (e.g. the wikipedia article).

LTV = m * r / (1 + d - r)

This model turns out to be of limited usefulness for understanding early stage companies because of the following assumptions that are built into it’s derivation:

• Retention is constant both across cohorts and, perhaps more importantly, throughout the lifetime of a customer (i.e. it assumes that if you have a probability r of being retained from month 1 to 2 then you also have a the same probability r of being retained from month 20–21)
• Constant unit economics throughout cohorts and customer lifetime which leads to a constant contribution margin.
• It assumes that these quantities are sufficiently constant over long enough time spans such that including the discount rate is sensible.

When we look at a company, it’s usually the case that none of these assumptions hold. Early stage companies have only a few month- or week-sized cohorts that usually vary significantly in retention as the underlying product is changing across cohorts. Early stage companies also usually haven’t settled on unit economics. These uncertainties are large enough that it’s not sensible to forecast so far into the future where discounting would matter.

We strongly prefer to look at empirically realized cohort LTV as opposed to imputed LTV based on a formula.

Getting back to our fictional revenue generating company, let’s look at some sample LTV curves.

The parenthetical number is the size of the cohort in week zero. Note that younger cohorts appear as shorter lines because we don’t know much about their LTV yet. Also note that there is inherent ambiguity in a quantity such as “9 month (36 week) LTV”. In this data set it’s anywhere between $160-$280, which is quite a large range. The older cohorts appear to have been higher and the more recent ones lower. Also note that the 2014–03–24 cohort is unusually strong, possibly due to some unusually large customers in that cohort who are spending a lot. Overall, these LTV curves are linear which is to say that customers are consistently spending as they age. If customers were not getting value out of the product they would presumably slow their spend leading to a flattening of the LTV curve. Of course, any single LTV curve is always increasing. Also note that we decided to show only a few cohorts as showing them all would make the graph unreadable. We also indexed against the week of first spend rather than registration because different products encourage/require registration at different times relative to convincing customers to spend actual money.

Something to watch out for, the LTV for a given cohort at age T is computed as the total revenue realized by that cohort up to T divided by the total number of customers in the cohort including customers who may have churned out. Sometimes we see companies exclude churned customers and compute LTV at T based only on customers who are still paying at time T. This is not the way to go because you had to pay to acquire all those customers up-front (via marketing, etc.) and that expenditure doesn’t disappear when the customers churn out.

Also note that we typically only look at the top-line revenue LTV per cohort and leave out the contribution margin/unit economics discussion to a separate conversation. There’s the question of how customers are reacting to your proposed offering which is separate from your ability to deliver that offering with reasonable unit profitability. For the purposes of this discussion, we are only seeking to understand the first part of this question.

There are four types of behavior that any cohort can exhibit:

• Flat LTV: The cohort spent once up-front and never spent again. The cohort is generating no further incremental revenue. Not necessarily bad if the flat LTV is at a value high enough to be very profitable. For example, eBay Motors probably exhibits this behavior.
• Sub-linear LTV: The cohort continues to spend as time goes on although the spend decreases over time. Such cohorts approach Flat LTV after some time. Most businesses are in this category. Customers spend initially and then spend less and less as time goes on.
• Linear LTV: The cohort consistently spends the same amount per user in the cohort. This is probably what Spotify looks like. There is likely some fall-off in the first month but after that their cohorts probably evolve to linear or just under sub-linear assuming that the core Spotify user doesn’t intend to ever cancel their subscription. Truly linear LTV would be something like your relationship with a utility such as PG&E which is extremely high retention. Also note that there are different classes of linear LTV growth. A business can be linear with a large positive slope or linear with a smaller positive slope. If a business has a core of recurring customers in each cohort that continue to spend indefinitely then the LTV will be linear. The magnitude of the slope will be determined by how many non-core customers are in each cohort diluting the LTV of those core customers.
• Super-linear LTV: Customers spend more as they age. For example, this is almost surely what Amazon sees. In your first month on Amazon you spend some money and in later months you spend much more. For another example, consider Slack. Each paying customer is a company that pays for some number of seats. In successive months a customer may purchase more seats as more people adopt the service in the company. These cases are the most exciting. They suggest the possibility of almost limitless LTV per customer.

Needless to say, what we really want to see are businesses that exhibit some strong evidence of linear to super-linear LTV in at least some of the cohorts of customer.

The visualization above of the LTV curves is good and easy to understand but it doesn’t give a good sense of whether the LTVs are getting better or worse. If all the LTV curves looked the same and everything was stable then we’d just use the formula above to compute full LTVs. However, we’re usually looking at companies that are fluctuating a lot in their early days so we’re really interested in the trends of LTV. Here’s how we prefer to look at LTVs to get a sense of those trends.

This image is a bit tricky at first so we’ll describe it in detail. In this image, the x-axis is the calendar week of the cohort. The bars in the background are the sizes of the cohorts. They’re included for reference purposes. The lines show successive LTV points as the cohort ages. For instance, the 2014–11–03 cohort (arrow 1 in the above figure) had about 350 customers and spent an average of $44 per customer in the initial week. After a month, the 4-week LTV for that cohort was a bit higher at $55 (on the green line). This cohort just passed 6 months (24 weeks) of age and thus the 24 week LTV for this cohort was just determined at $125. We do not yet know the 36 week LTV for this cohort because the cohort is not yet 36 weeks old and the above image does not attempt to extrapolate it for us (recall, that’s essentially the problem with the formula based approach which mixes in extrapolation with actual observation). Note that the lines in this figure can never cross each other because for every cohort the N+1 week LTV is greater or equal to the N week LTV. Also note that any given line is made up of different customers on successive data points. The N week LTV line measures this quantity for each successive cohort which are made up of distinct customers.

This visualization is good because it shows us trends in the LTV. For instance, you can see that the 12 week LTV started trending down a bit as the cohorts got larger starting in late 2014 (arrow 2). The earliest cohorts in early 2014 had very high LTVs (arrow 3). This is pretty common as early adopters are usually more inclined to use a product. The 2014–03–24 cohort that appeared oddly in the original LTV graph appears here as a clear spike (arrow 4). Note that the spike didn’t occur until sometime between 24 and 36 weeks in the lifetime of that cohort (arrow 5). That is also apparent in the original LTV figure where that cohort started growing strongly at around 28 weeks.

In terms of what we want to see when we look at the LTV trends of a company:

We want to see increasing LTV both for later larger cohorts and later in the customer lifecycle.

As the business attracts larger cohorts of customers it’s often the case that LTV degrades because the larger cohorts are made up of later adopting customers who are less inclined to use the product. If the product has truly great product-market fit then the later larger cohorts will monetize at even higher rates both as the cohorts get larger and as the cohorts age.

Another view that is sometimes useful is the heatmap view.

For this figure, the cohort week of first spend is on the y-axis. The x-axis along the top is weeks since first spend and the color is the cumulative LTV of that cohort. So the topmost row is the 2013–12–30 cohort which has 110 customers and whose LTV goes up as one goes to the right. The bars on the left are the sizes of each successive cohort. More recent cohorts have not yet revealed their LTV. As time passes diagonal lines are added to this figure. This is showing only the first few cohorts as if time stopped in March 2014 for readability. The full heatmap for all the cohorts looks like this:

Note that you can see the outlier 2014–03–24 cohort here as the green horizontal stripe. Fixed calendar phenomena (such as holidays or sales) manifest themselves as diagonal features in this image. Phenomena that affect fixed cohorts (such as customers gained via a burst of paid acquisition expenditure) appear as horizontal features. If there was a significant drop in, say, N-week retention, it would make itself apparent as a change in the color where cohorts would be taking longer to reach the color yellow for instance.

Each of the above visualizations has a strength and a weakness. The LTV curves give you a good sense of the shape of the curves but it starts becoming unreadable after a small handful of cohorts and it doesn’t show trends well. The LTV trends gives a good sense of the trend and shows all cohorts, but only gives a hint of the shape of the LTV curve via the spacing between the lines for each cohort (it’s essentially a contour plot). Also, the LTV trends doesn’t show the LTV at all points in the cohort age but rather at selected age milestones. The heatmap allows you to see all the cohorts at all points in their lifecycle, but does not give a good sense of how the values are increasing as it’s encoded in the color change. The heatmap also does a good job of showing seasonal effects. When we do diligence we typically prefer to see the curves themselves and the LTV trends and only occasionally use the heatmap view. If you’re coming to pitch your company to us you’d do well to make these graphs ahead of time.

So that’s the story for revenue lifetime value. Next time we’ll take this framework and use it to understand cohort level engagement and retention.

Edit: For reference, here’s the full table of contents.

1. Accounting for user growth
2. Accounting for revenue growth
3. Empirically observed cohort lifetime value (revenue)
4. Empirically observed cohort lifetime value (engagement)
5. Depth of engagement and quality of revenue
6. Epilogue: The 8-Ball and GAAP for Startups

[Note from the author: See an update to the thinking presented in these articles here. I no longer work at Social Capital, if you want to reach me you can email me at jonathan@tribecap.co]

Published in #SWLH (Startups, Wanderlust, and Life Hacking)

Diligence at Social Capital Part 3: Cohorts and (revenue) LTV was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the previous post of this series we looked at growth accounting for users by showing monthly active user (MAU) growth accounting and how it could uncover different underlying dynamics for a growing user base. Today we will take that framework and apply it to revenue growth. This will be particularly useful for thinking about recurring revenue as would be exhibited in an enterprise software-as-a-service (SaaS) business or in a consumer subscription business.

Consider a company that makes revenue via monthly subscriptions. For the sake of concreteness, let’s consider a B2B SaaS company. For such a company, in addition to understanding MAU growth we will also want to understand the growth of monthly recurring revenue (MRR). Note, we’d still likely be interested in MAU accounting. If a customer falls out of MAU it is likely that they will also fall out of MRR. For now, let’s focus on the MRR growth. In this case, the most common figure we see is a cumulative MRR chart going up and to the right.

Similar to the MAU case we will now break down the components of MRR growth. There is some added nuance in this case. In MAU accounting a user either comes back or does not (i.e. is retained or churned). When counting dollars, a single customer may be retained or churned but they may also be retained as a customer by spending more or less in the second period relative to the first. As such, we separate out expansion and contraction into the growth accounting. The explicit identity is:

MRR(t) = new(t) + retained(t) + resurrected(t) + expansion(t)

MRR(t - 1 month) = retained(t) + churned(t) + contraction(t)

Note that if a customer spends $10 in the first month and $12 in the second month we are counting $10 as retained spending and $2 as expansion spending and similarly for contraction. We are counting dollars as churned only if the customer went to zero dollars spent and resurrection only when the user comes back from a churned state. The above can be rearranged as follows:

MRR(t) - MRR(t - 1 month) = new(t) + resurrected(t) + expansion(t) - churned(t) - contraction(t)

Which are the five components illustrated here:

Once again, we compute the dollar based “quick ratio” which comes out to between 1 and 1.5 depending on the month and the dollar based retention rate which is here ~40%.

Recall that in the consumer MAU case, a quick ratio of 1.5 was about average for a consumer business. For recurring revenue however, this is not so good.

Recurring subscription revenue is default-retained in contrast with recurring visitation which is default-not-retained. As such subscription revenue tends to have a much lower churn rate and higher quick ratio.

To give more examples, if you think about consumer subscription businesses (such as Spotify or Netflix), they generally shouldn’t churn much and this should be reflected in a high quick ratio. Comparatively, pure consumer transactional retail businesses (such as, say, Nordstrom online) tend to churn a lot on a monthly basis as customers don’t have a strong pull to purchase on successive months. If you consider subscription businesses that rely on landing-and-expanding (such as Slack) you would hope to see a lot of expansion revenue as each customer expands the number of paid seats.

For enterprise SaaS companies, we prefer to invest in companies with a quick ratio greater than 4.

If you have a quick ratio of <2 then your churn is probably too high and you have something to fix. For more information about how this fits into our overall approach to enterprise saas investing, see Mamoon Hamid’s deck on the topic from early 2015. Here’s the key slide from that deck illustrating some actual SaaS companies.

The two companies on the right are in our portfolio and we passed on the two on the left. Note that the top left Company A has a pretty good story. Company A would say that they have a ton of expansion revenue which definitely suggests some really good aspects of product-market fit. However, it’s largely eaten up by contraction revenue leading to a situation where they have to generate expansion MRR just to outrun contraction MRR to generate net growth.

I also recommend Bobby Pinero’s post on this topic discussing SaaS metrics metrics that they used in Intercom’s fundraising process.

Accounting for Everything Else

Now that we’ve shown growth accounting for both MAU and MRR it should be clear that we can do this accounting for any quantity that is important for the business. For example, say you have a social network consumer app and perhaps you think that MAU isn’t stringent enough and you really want users to be in your app every day. In that case, merely measuring a user as active or not is not sufficient and you’d like to know if a user was really active vs. only slightly active in the month. This can be accomplished by measuring days active in the month which was dubbed L28 at Facebook. For instance, a user of L28=10 was active 10 days of the last 28. If you sum up L28 across all users in a month (as measured on the last day of the month) you get the total sum of DAU across the month. You could compare the total sum of L28 in one month and compare it to the next month and do growth accounting on it. A user expands/contracts if they have a higher/lower L28 this month relative to last month and so on. This would provide a “monthly growth accounting of DAU”.

If you’d like an example unrelated to active users or revenue, consider something like link sharing on Twitter. If you are working on link sharing at Twitter your goal is probably to grow link share events and you could use the above approach to account for the growth in link share events by users who are sharing more/less than they were in a prior period.

To review, growth accounting is a framework that can be used in just about any situation where a group of users/customers are accruing value to your business in some form or another (revenue, their attention via DAU, contributing content to the system, etc). One clear shortcoming of growth accounting is that the churn figures don’t specify whether the churning customers are recent additions to the business or if they are older customers who have been getting value for a while. That is to say, this approach doesn’t provide a very clear picture on the life-cycle of users. In order to clarify that we should look at cohorts and how lifetime value is realized. We’ll do that in the next post.

Edit: For reference, here’s the full table of contents.

1. Accounting for user growth
2. Accounting for revenue growth
3. Empirically observed cohort lifetime value (revenue)
4. Empirically observed cohort lifetime value (engagement)
5. Depth of engagement and quality of revenue
6. Epilogue: The 8-Ball and GAAP for Startups

[Note from the author: See an update to the thinking presented in these articles here. I no longer work at Social Capital, if you want to reach me you can email me at jonathan@tribecap.co]

Published in Startups, Wanderlust, and Life Hacking

-

Diligence at Social Capital Part 2: Accounting for Revenue Growth was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here at Social Capital we spend a lot of time conducting diligence on potential investments. In the next few weeks we’re going to discuss how we think about quantitatively assessing product-market fit. The approaches we discuss here are useful beyond diligence for investment. Many of our portfolio companies use these concepts as part of their operating process to keep tabs on their growth and evolving product-market fit. Hopefully you find it useful!

Diligence is designed to help us understand both what currently exists and what may come to be as well as to try and help the entrepreneur on their path. While we definitely talk to many entrepreneurs in the pre-product stage, most of our time ends up being spent with entrepreneurs who already have a product with users/customers. In this case, part of diligence involves developing an objective understanding of demonstrated product-market fit. While every company is different, we have a few standard ways of looking at core traction metrics and we’d like to share these with you in the hopes that it helps to give you insight into your own business. It should go without saying that there are many other aspects to diligence including, but not limited to, team, market, vision, competition, core technology, etc. and that we will only be discussing a particular subset of diligence in these posts.

A tentative outline for how we will present our approach is as follows:

1. Accounting for user growth
2. Accounting for revenue growth
3. Empirically observed cohort lifetime value (revenue)
4. Empirically observed cohort lifetime value (engagement)
5. Depth of engagement and quality of revenue
6. Epilogue: The 8-Ball and GAAP for Startups

These topics apply in slightly different ways for consumer businesses vs. enterprise SaaS businesses so we’ll generally treat them separately, but the framework applies to both types of business.

Today I’m going to focus on the first of these topics.

Accounting for User Growth

To get started, let’s pretend we have a consumer company that intends to get lots of users via some novel social/mobile/content product strategy. For these types of companies the most common graph that we see in pitches is a graph of users going up and to the right. Sometimes companies try to show us a graph of “cumulative registered users”, which is clearly a vanity metric. A user who has registered and is not active in your product is probably not getting much value and is probably not a good indication of product-market fit.

Given that cumulative users is rarely shown in serious pitches, in the current era we typically see a graph of monthly active users (MAU) going up and to the right.

This is showing 16 months of roughly 12% m/m growth which is quite impressive. However we always go one level deeper to understand the nature of the growth. For the purposes of illustration, let’s pretend that we have some sort of mobile application. For the definition of “active” you should use whatever definition best encapsulates active for your application. This can be as broad as “opened the app” or as specific as “carried out a particular action”. Consider the following two accounting identities.

MAU(t) = new(t) + retained(t) + resurrected(t)

MAU(t - 1 month) = retained(t) + churned(t)

The first one says that active users today (for the trailing 30 days) are either new users, retained from the previous month or resurrected from some prior time period. Note that this is a mutually exclusive and completely exhaustive classification of current users. The second identity says that the MAU from last month either came back and were retained or did not and thus churned.

Manipulating the above yields the following:

MAU(t) - MAU(t - 1 month) = new(t) + resurrected(t) - churned(t)

Which is to say that MAU growth receives positive contributions from new and resurrected users and receives a negative contribution from losing users to churn.

Here’s how we prefer to look at the MAU growth accounting quantities for the above fictional company. If you’re going to show us your MAU growth you’d do well to show it to us in this manner.

The bars show the three terms on the right of the equation above that contribute to MAU growth. I also overlaid two ratios that are useful. The retention rate is month-over-month so this says that our fictional company has a retention rate of ~40%. Note that because of the accounting identities above, this means that the churn rate is (100% - 40%) = 60%. The other ratio is (new + resurrected)/churned which is to say the ratio of the area above the line vs. below the line. This ratio needs to be greater than one if the app is to be growing, otherwise churn is overwhelming growth. We call this the “quick ratio”.

Quick Ratio = (new + resurrected)/churned

This term was coined in a deck from Mamoon Hamid from earlier this year. We’ll talk more about this concept in a later post when we discuss revenue. Also, this should not be confused with the normal finance quick ratio that measures the ability of cash and near cash assets to pay off liabilities . For this company the MAU quick ratio is fluctuating between 1 and 1.5. Which is to say that for every 3 new users the company adds it is also losing 2–3 users to churn.

Note that this gives much more information than the MAU graph above. This is telling us that there is some large month-to-month churn which is being overcome by large contributions from new and resurrected users. The retention rate has been stable and not showing any particular trend. On the plus side, it’s not going down as the app is growing, but it’s also not getting any better which you might assume if you only looked at the top-line number or at presumable feature/functionality changes in the past 12 months.

In terms of how the above looks, we’d classify this as a so-so situation for a consumer app. Most consumer applications don’t have a very strong mechanism to bring users back month after month and so the quick ratio tends to be just above 1. The dynamic for each month in a consumer app is typically to add a bunch of users and to simultaneously lose a bunch of users with a small additive piece on top from resurrection yielding overall small positive growth.

To show you the power of this view, there are other ways that these components could conspire to produce the same top-line growth. For instance..

Note, these numbers would produce exactly the same MAU graph shown above. I kept the axes the same to emphasize the difference. In this case, the new component isn’t growing as fast but it’s also retaining much better. The quick ratio for this company is more in the 1.5–2.0 range which would be very good for a consumer company (for every 3 customers that you are gaining you are losing between 1.5-2 customers, much better than the first company). Note the spikes in resurrected possibly due to some resurrection campaigns which were not accompanied with a corresponding spike in churn.

All else being equal, the second example would be a more attractive company to us because it is starting from a better base. With such high retention it would be worth trying to push harder on the top of funnel with new users to drive growth (more aggressive sharing/referral mechanisms, paid acquisition, etc.). For the first example it’s harder to justify pushing on new users as you would end up losing many of them. It’s easier to fill the top of funnel than it is to fix some underlying churn problem.

It should also be clear that this accounting can be done on time-frames other than calendar months. Indeed, to make this operational, several of our portfolio companies implement it on a rolling 28 day basis (to remove day-of-week effects). Also, this approach works just as well with weekly active users as it does with monthly. Typically early stage consumer products already have trouble generating m/m retention, much less w/w retention so the w/w views will show very high churn and may not be useful. But if your product is extremely sticky and is already retaining at a very high level on a monthly basis then it might be time to explore generating the next level of engagement at a weekly level. For something as sticky as FB it even makes sense to get it down to the daily or even sub-daily level.

In the next post we’ll apply this framework to subscription revenue which is useful both for enterprise SaaS as well as consumer subscription businesses.

[Note from the author: See an update to the thinking presented in these articles here. I no longer work at Social Capital, if you want to reach me you can email me at jonathan@tribecap.co]

Published in Startups, Wanderlust, and Life Hacking

-

Diligence at Social Capital Part 1: Accounting for User Growth was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.