As I mapped out the various flywheels that exist within Limbix’s product, I noticed that each and every one of them had a single step in common. It became clear that this step represented a high-impact metric that was foundational to our success. It eventually informed the north star that our entire company now rallies behind.

I traced the path I took to this insight and deconstructed it into three steps. Anyone can apply this process to reveal valuable insights about the metric that matters most to their business:

1. Identify the individual flywheels that exist in the business
2. Map out how the flywheels interact with one another
3. Identify the central node to inform the north star metric.

Let’s walk through each of these steps as they apply to Limbix.

Step One: Identify the individual flywheels

In my last post, I identified four types of flywheels that occur frequently in mental health tech:

1. Shared economies of scale. For example, when a mental health company invests in technology to lower the marginal cost of care. If the lower cost is passed on to patients in the form of lower prices, it attracts more patients, which drives down per patient costs even further.

2. Network effects. For example, when two-sided marketplaces are built to match therapists with patients. More patients attract more therapists, which attracts even more patients.

3. Brand habit. For example, when a digital therapeutic becomes associated with the treatment of a specific diagnosis, that therapeutic is more likely to be recommended by healthcare providers and used by patients, which strengthens the association even further.

4. Switching costs. For example, when a product gets integrated into a health system’s electronic health record (EHR), it becomes costly for those health systems to switch to another vendor. This reduces competition, making it easier for the existing vendor to upsell and cross-sell new products to the health system.

Let’s see which ones apply to Limbix.

Flywheel One: Brand Habit

From Flywheels in Mental Health Tech:

Brand habit refers to the positive association between a product and its value proposition.

When a customer has a positive experience using a product it strengthens that association, increasing the likelihood that they’ll use the product again. At the same time, prospective new customers vicariously learn the brand habit from existing users, making it more likely that prospective customers will convert.

At Limbix, we’re building a special type of product called a prescription digital therapeutic. It is standalone software (i.e. an app) that is FDA cleared for the treatment of a medical condition and is prescribed to patients by licensed healthcare professionals.

The more providers prescribe our digital therapeutics, the more patients are helped, which in turn makes it more likely that existing and prospective providers will associate our therapeutics with their standard of care. This, in turn, increases the likelihood that providers will prescribe our therapeutics in the future.

Flywheel Two: Switching Costs

From Flywheels in Mental Health Tech:

Switching costs refer to the loss in value that a customer incurs when switching to an alternative supplier.

New customers often invest considerable resources in integrating a product into their existing systems. If the customer then wants to switch to a different provider, they must incur that cost all over again.

Furthermore, if that customer wants to buy related products, there is a distinct benefit to purchasing from an existing vendor. This is especially so if the new offering seamlessly integrates into the vendor’s products. This dynamic lowers customer churn and makes it much easier for vendors to upsell and cross-sell products.

When providers consistently prescribe a Limbix therapeutic, it often gets formally integrated into their clinical workflows. While this occurs at the individual provider level, it also occurs at the health system level, whereby the administration incorporates Limbix’s therapeutics into default workflows and electronic medical record (EMR) configurations for all providers within the health system.

Once these integrations occur, providers are more likely to continue using Limbix therapeutics, rather than switching to those of a competitor. Additionally, providers become more likely to use our future therapeutics since they seamlessly integrate into providers existing workflows and systems.

Flywheel Three: Data network-effects

From Flywheels in Mental Health Tech:

Data network effects occur when new data makes the product more valuable, and increased usage of the product yields more data. A classic example is the navigation app, Waze. Users create traffic data, which produces better route calculations, which encourages more usage, which generates more traffic data.

At Limbix, we leverage data from our digital therapeutics to continually improve engagement and clinical outcomes over time, which in turn, leads to more usage and more data.

For example, we can incorporate algorithms that personalize our intervention to the patient, optimizing for clinical outcomes and engagement. The improved outcomes and engagement lead to more usage by existing patients and incentivize more providers to prescribe our therapeutic. This will produce even more data, which further optimizes the personalization algorithm.

Step Two: Map out how the flywheels interact

Flywheels do not exist in isolation. In fact, they have been called the “meta competitive advantage” because individual flywheels naturally combine to produce a force that is greater than the sum of their parts.

To map out Limbix’s flywheel meta-structure, we must identify the linkages that the flywheels have in common.

All of our flywheels include provider prescriptions as a step. I’ve visually represented the full system by putting this common step in the center.

Step Three: Identify the central node to inform the north star metric

Prescriptions from providers serve as the central node that joins the three flywheels together. That is, accelerating any given flywheel, increases the number of prescriptions, which, in turn, accelerates all the other flywheels in the system.

This central node has informed our north star metric.

As Amplitude writes:

A north star metric is the key measure of success for the product team in a company. It should consist of 2 parts:

1. A statement of your product vision, and

2. A metric that serves as a key measure of your current product strategy

Applying this to Limbix, we arrived at the following vision and key metric:

Furthermore, a north star metric should be composed of multiple inputs, each serving to drive the metric.

We can unfurl our flywheel diagram to inform the inputs of our north star metric.

We can then use these inputs to guide our organizational structure and OKRs. For example, at Limbix, we have dedicated teams for each input: patient engagement & clinical outcomes, health system integrations, and provider awareness.

Each team prioritizes the initiatives within their input that will maximize our north star metric: # of active prescribers. Doing so gives each team clarity on what they should focus on while maintaining alignment with a cohesive company vision and strategy.

Flywheels Matter

I didn’t know what I would find when I started my deep dive into flywheels. I was simply driven by a curiosity to learn about the strategic forces that shape mental health tech and Limbix’s products in particular.

What I found was a powerful framework that had rippling effects across our entire company. From the macro of vision and strategy to the micro of team structure and metrics, flywheels have forever changed the way I approach product management.

—

Thank you to Ben Lewis, Jonathan Sockell, Elise Vierra, Jake de Medeiros, Amanda McCawley, Eugene Hauptman, and Tom White for valuable input on drafts.

Flywheels: From Theory to Action was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, what is a flywheel?

The term flywheel refers to a positive feedback loop in a product or business that makes it progressively easier to increase revenue and fend off competition. Think of a heavy wheel that is difficult to spin at first, but as it gets going, starts to build on its own momentum, and becomes easier and easier to accelerate. If you’ve ever used a spin bike, you’ve experienced this for yourself. The bike’s internal wheel is literally called a flywheel.

Flywheels are powerful. Virtually every one of the world’s most valuable and enduring companies has leveraged them to achieve success.

So, it came as no surprise to me that flywheels can (and must) be leveraged to achieve success in mental health tech.

I’ve identified four types of flywheels that occur frequently in mental health tech:

1. Shared economies of scale. For example, when a mental health company invests in technology to lower the marginal cost of care. If the lower cost is passed on to patients in the form of lower prices, it attracts more patients, which drives down per patient costs even further.
2. Network effects. For example, when two-sided marketplaces are built to match therapists with patients. More patients attract more therapists, which attracts even more patients.
3. Brand habit. For example, when a digital therapeutic becomes associated with the treatment of a specific diagnosis, that therapeutic is more likely to be recommended by healthcare providers and used by patients, which strengthens the association even further.
4. Switching costs. For example, when a product gets integrated into a health system’s electronic health record (EHR), it becomes costly for those health systems to switch to another vendor. This reduces competition, making it easier for the existing vendor to upsell and cross-sell new products to the health system.

Shared economies of scale

Economies of scale occur when increased volume reduces the cost of production. They’re usually associated with the manufacturing of physical goods, which has high initial fixed costs (building the manufacturing plant), followed by low marginal costs to produce items within the plant.

The flywheel effect occurs when companies share their economies of scale with their customers in the form of lower cost or greater value. Think of Costco, which shares its economies of scale with customers in the form of lower prices. The lower prices attract more customers, which increases the economies of scale, allowing Costco to lower prices even further.

Mental health example: Tech-enabled mental health providers

Affordability and lack of access are among the greatest challenges in mental health care today. There is an overwhelming need for care combined with a severe shortage of mental health care providers. This classic supply-demand mismatch results in only a small percentage of those in need accessing care, and those lucky enough to do so end up paying extraordinarily high out-of-pocket prices.

This problem, in large part, stems from the economics of traditional face-to-face therapy. Therapy has high variable costs: regardless of volume, the hourly cost to deliver traditional therapy stays roughly the same.

Tech-enabled mental health providers offer therapy, but leverage technology to lower the cost of care and improve access. This includes:

• Automating administrative tasks. Technology can be used to automate time-intensive administrative tasks ranging from appointment scheduling and note-taking to billing and processing insurance claims.
• Delivering care through coaches. Many tech-enabled mental health providers are improving access by delivering care through coaches — many of whom receive on-the-job training — rather than Master’s and Doctoral level providers. The coaches deliver highly manualized therapies, and technology is used to facilitate strict adherence to protocols.
• Incorporating software-based treatments. There is a growing body of evidence showing that stand-alone and guided1 software-based treatments deliver clinically significant outcomes for patients. This is particularly so for patients with lower-severity symptoms. Many tech-enabled mental health companies are incorporating software-based treatments to lower the overall cost of care.

In all of the above cases, the technology has considerable initial fixed costs. However, it becomes minimal when spread out over a large patient base. Over time, these initiatives drive down the per-patient cost of care.

A flywheel effect can occur if tech-enabled mental health companies share these economies of scale with payors and patients. The most obvious way is to reduce the price of care. Doing so expands access to the services, increases usage, and reduces the cost even further.

Companies can also share their economies of scale by providing more value for the same price. Think of Netflix, which continually reinvests its profits into producing more and more content, and thereby delivers more value for the same price. So, too, tech-enabled mental health providers can reinvest profits into the creation of more value for patients, such as more self-guided content or on-demand access to coaches. Doing so attracts more users, which further lowers the per-user cost of delivering care.

Network effects

Network effects refer to a mechanism in a product in which every new user of the product makes it more valuable for everyone else. While there are many different types of network effects, the types I see most frequently in mental health tech are two-sided network effects and data network effects.

Two-sided network effects occur when there are two categories of users (e.g. riders and drivers on Uber). More supply-side users (e.g. drivers) attract more demand-side users (e.g. riders), which in turn attract more supply-side users.

Data network effects occur when new data makes the product more valuable, and increased usage of the product yields more data. A classic example is the navigation app, Waze. Users create traffic data, which produces better route calculations, which encourages more usage, which creates more traffic data.

Mental health tech examples

Two-sided network effects: Therapy marketplaces

In mental health tech, the two user groups are usually therapists and patients. Many of today’s most valuable mental health startups, such as Talkspace, Lyra, and Headway, operate as 2-sided marketplaces that match these two groups.

Therapists prefer the platform that offers them to build the largest caseload with the least amount of effort.2 Patients, in turn, prefer the platform that has the best selection of therapists. Thus, more therapists on the platform attract more patients, which in turn attract even more therapists.

While therapists and patients can switch between marketplaces, the two-sided network effects operate such that only a small number of marketplaces will achieve the critical mass of therapists and patients necessary to sustain themselves. Just as we are left with Uber & Lyft, DoorDash & Uber Eats, and Airbnb & Vrbo, we will be left with just a few dominant therapy marketplaces.

Therapists also prefer the platform that offers them the highest rates. Platforms that have more therapists can leverage their scale to negotiate higher reimbursement rates with payors. For example, Headway is a platform that connects patients with in-network therapists. As it increases the number of therapists on the platform, it has greater power to negotiate rates with insurers (e.g. Anthem or United Healthcare), which it can then leverage to attract more, higher-quality therapists. In this case, there is a combined effect of shared economies of scale (from the platforms negotiating power) and two-sided network effects.

Data network effects: Digital biomarkers

Mental health providers have traditionally relied on subjective measures of clinical effectiveness. For example, the primary measure of depressive symptoms is the PHQ, which requires patients to answer questions like “How often have you felt down, depressed, or hopeless in the last two weeks?”

Digital biomarkers refer to technology that measures mental health symptoms based on objective and passive user data. Companies like Ellipsis Health and Kintsugi quantify symptoms of anxiety and depression based on patient speech data. Others like Mindstrong and KeyWise are aiming to do so using smartphone interaction patterns like typing, swiping, and scrolling.

These products leverage algorithms trained on large amounts of proprietary data. Product usage leads to more training data, which improves the accuracy of the product. This leads to greater uptake of the product, which leads to even more data.

If a digital biomarker amasses proprietary training data that materially improves product quality relative to competitors, it can produce a flywheel effect that is very difficult for competitors to catch up to.

As of this writing, Apple just announced its entry into the digital biomarkers space. Apple’s vast access to data may prove to be an insurmountable advantage for the existing players to compete with.

Brand Habit

Brand habit refers to the positive association between a product and its value proposition. When a customer has a positive experience using a product it strengthens that association, increasing the likelihood that they’ll use the product again. At the same time, prospective new customers vicariously learn the brand habit from existing users, making it more likely that prospective customers will convert.

Mental health tech example: Employer-funded mental health benefits

As I’ve written about before, self-funded employers tend to be early adopters of mental health tech products, and they account for a significant proportion of the industry’s revenues.

Brand habit impacts the likelihood that a self-funded employer will choose to offer a specific mental health product to their employees. For example, benefits directors (the decision-makers) read case studies, go to benefits conferences, and network with their industry counterparts. The more employers that offer a specific product, the more likely it is for additional employers to follow suit due to the strong influence of social proof and bandwagon effects.

There is also a simultaneous brand habit impacting employees. In most cases, self-funded employers only pay for products when their employees use them. Therefore it is not enough to get the employer to offer the service (the first sale) — a secondary sale is needed to activate each employee and drive usage. As employees start to use the product and develop a brand habit, they are more likely to continue using it. At the same time, their colleagues will learn about the product from their peers, and be more likely to use the product as well.

Switching costs

Switching costs refer to the loss in value that a customer incurs when switching to an alternative supplier.

New customers often invest considerable resources in integrating a product into their existing systems. If the customer then wants to switch to a different provider, they must incur that cost all over again.

Furthermore, if that customer wants to buy related products, there is a distinct benefit to purchasing from an existing vendor. This is especially so if the new offering seamlessly integrates into the vendor’s products. This dynamic lowers customer churn and makes it much easier for vendors to upsell and cross-sell products.

Mental health tech example: Selling to health systems

When a health system purchases a new product, it must obtain signoff from a complex group of stakeholders, including clinical, legal, finance, and IT. Then, once the vendor is chosen, significant resources are invested into training staff and integrating the product into the health systems workflows.

Health systems are therefore reluctant to replace an existing product with a competitor unless it’s significantly better or less expensive. They’re also more likely to purchase new products from existing vendors because those new products will more easily integrate into their existing workflows.

This is one of the reasons why many large tech companies have recently acquired behavioral health products. Take the merger of Teladoc and Livongo. Teladoc already serves many of the nation’s largest health systems. The merged entity can now easily cross-sell Livongo’s behavioral health offerings to Teladoc’s customers. The recent acquisition of Silvercloud by Amwell is motivated by a similar strategy.

Understanding the theory is just the beginning

In this article, I’ve mostly explored the theory of flywheels. Using this information, I hope that you’ll be able to identify some of the flywheels that exist in your own businesses.

However, identifying flywheels is just the beginning. The real value comes from translating that knowledge into actionable insights to inform product strategy.

In a follow-up post, I’ll walk through a case study and map out multiple flywheels in a single product. I’ll then explore how the flywheel map indicates where to invest for maximum leverage and the selection of a ‘true north’ metric.

Additional reading

If you’d like to learn more about flywheels, I highly recommend the following resources:

• Advantage Flywheels by Max Olson. This is a great primer with excellent diagrams of well-known companies’ flywheels.
• Flywheel Effect: Why Positive Feedback Loops are a Meta-Competitive Advantage by Eric Jorgenson. Another great primer on flywheel effects.
• Scale Economies Shared by Jonny Cornish. A fantastic deep dive into shared economies of scale.
• The Network Effects Manual by James Currier. The best article I’ve read on network effects.
• 7 Powers: The Foundations of Business Strategy by Hamilton Helmer. While not specifically about flywheels, this covers many of the strategic forces that underlie flywheel effects, such as brand, switching costs, network effects, and economies of scale.
• Thinking in Systems by Donna Meadows. At their core, flywheels are systems of positive feedback loops. This classic on systems thinking provides excellent foundational knowledge about how positive feedback loops accelerate and then ultimately decline.

Thank you to the following people for valuable input on drafts: Ben Lewis, Jon Sockell, Elise Ogle, Mel Goetz, Amanda McCawley, Tina Quach, Kabir Daya, Meeta Sharma, Eugene Hauptmann, Nithya Krishnamoorthy, Marcus Whitney, David Burt, Jordan Jones, Soma Mandal, Kavir Kaycee, Melissa Trevino, Angelo Belardi, Chris Angelis, Kiki Schirr, and Christine Cauthen.

Flywheels in Mental Health Tech was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why did Ginger succeed while Lantern failed? While there were many factors at play, one, in particular, played a major role: Ginger was able to prove compelling value propositions to third-party payors while Lantern did not.

Third-party payors (often simply referred to as ‘payors’) are the entities that cover healthcare expenses on behalf of patients. These include commercial insurers (e.g., the ‘Blues’ or UnitedHealthcare), self-funded employers (e.g. Google or JPMorgan), and the government (e.g. Medicare or Medicaid).

Whether they are focused on exclusively, or included as a channel in a larger revenue mix, third-party payors form an essential pillar of most mental health startups’ long-term success. As a16z general partner Jorge Conde elegantly puts it: “In healthcare innovation, all roads lead back to the payor”.

In healthcare innovation, all roads lead back to the payor

However, it is not enough to simply decide to target third-party payors. To be successful, startups must have a compelling value proposition for them!

Many mental health startups struggle with payor value propositions because the US healthcare industry is riddled with complex dynamics and counterintuitive incentives. For example, while it would make sense for reimbursement to be primarily driven by better health outcomes or long-term cost reduction, that is often not the case.

I’ve spent the last several years at health-tech startups — and countless conversations with colleagues across the industry — deep diving into what third-party payors care about and how best to demonstrate value propositions to them. I’m excited to share what I’ve learned!

Specifically, I’ll cover 5 value propositions:

1. Cost reduction
2. Differentiated benefits offerings and increased member satisfaction
3. Improved health outcomes
4. Provider satisfaction and support
5. Mental health parity

Much of what I’m sharing applies to all health tech startups. However, I have included many details and examples that are specific to digital mental health.

Value prop 1: Cost reduction

Cost reduction is arguably the strongest value proposition for payors. If your product materially reduces health care spending, payors are highly motivated to cover it.¹

There are two types of costs that mental health startups can reduce: First-order costs and second-order costs.

First-order costs refer to the direct cost of treating a mental health diagnosis. If someone with depression generally requires 12 sessions at $100/session to address their symptoms, then the first-order cost is $1,200.

Second-order costs refer to the indirect costs of a mental health diagnosis. If that same person also has diabetes, their depression tends to exacerbate their diabetes symptoms, because they’re less likely to eat well, exercise, or take their medication. If this exacerbation doubles their overall medical expenses from $16,000 to $32,000, then the second-order cost of the mental health diagnosis is $16,000.

In this depression example, the first-order cost of a mental health diagnosis is relatively small. Many payors do not consider such costs large enough to invest significant efforts to decrease them.²

In contrast, the second-order cost of the mental health diagnosis when combined with other chronic illnesses is significant and deemed worthy of cost-reduction efforts. Other impactful second-order costs of mental health conditions include lower employee productivity and increased absenteeism.

AbleTo, Quartet, and Vida Health are examples of behavioral health startups whose primary value proposition is to reduce the second-order costs associated with mental health conditions. Many chronic condition management companies, such as Livongo, Omada, and Onduo, have also expanded into mental health care for exactly this reason.

Proving the value prop

Cost-reduction value propositions are proven through studies and pilots that measure the product’s impact on the first and second-order costs of mental health conditions.

Commercial insurers are primarily interested in a product’s impact on overall healthcare expenditure. For example, AbleTo demonstrated in a peer-reviewed study that the use of their program led to significantly fewer all-cause hospital admissions and fewer total hospital days.

Self-funded employers also care about the impact on overall healthcare expenditure. However, they are also interested in metrics like employee absenteeism, retention, and productivity. For example, Lyra Health claims that the use of their platform is associated with a 50% decrease in employee turnover and that 70% of members show improved productivity. These productivity and retention savings account for a significant proportion of their purported 4x ROI.

Commercial insurers generally require payback periods of around 12–18 months. Anecdotally, the average member stays with an employer-sponsored plan for around 36 months. A plan is willing to invest such that they are in the red for 12–18 months, and then earn a positive return for the final 18–24 months. If the payback period is any longer, the plan risks making the upfront investment, only for a different insurer to reap the reward.

Most self-funded employers require a positive ROI within 12 months, because healthcare expenses are usually approved by their finance departments, and must show a financial return within a single, annual budget cycle.

This emphasis of insurers and employers on short-term ROI is unfortunate, because mental health treatment often leads to very meaningful and significant savings, but only when measured on much longer time horizons.³

On the other hand, Medicare Advantage plans tend to accept longer payback periods because members are usually retired, and stay with their plan for a longer period of time. The longer payback period, combined with higher instances of chronic illness in older populations, makes Medicare Advantage an excellent segment for cost-saving value propositions.

Value prop 2: Differentiated benefits and member satisfaction

Commercial insurers care about differentiation and member satisfaction because they want to compete on quality over price. Self-funded employers — particularly those in high-paying industries — want to attract and retain the best employees. They often compete with one another by having the best portfolio of benefits.

Having excellent mental health benefits has become even more important during the COVID-19 pandemic, with a Teladoc survey showing that 90% of employees reported having mental health concerns during this period. A weak behavioral health offering is now a deal-breaker for many employers and individuals when choosing a plan.

Directly advertising to consumers and providers — a common strategy among pharma — may ‘incentivize’ the member satisfaction value proposition. When members demand a product, payors become motivated to cover it, lest they risk disappointing their members and driving them to a competitor that does cover it.

There are four common ways to differentiate and increase satisfaction through a mental health offering:

1. Access. Many plans and employers struggle to offer accessible mental health care. They offer it in theory (i.e., there is a mental health benefit) but in reality, it is often very difficult to find a provider that takes insurance. The first wave of mental health startups addressed this problem. These include Ginger, Lyra, and Talkspace. In such cases, payors and employers want turnkey solutions that guarantee immediate access to mental health care for their members and employees.
2. Quality. As more mental health startups offer turnkey solutions, they must compete with one another on the basis of quality. Quality is most often measured in terms of net promoter score (are members satisfied with the service), engagement (are members consistently using the service), and health outcomes (are members getting better after using the service). Many digital mental health solutions for employers, like Lyra, Ginger, and Modern Health, explicitly position themselves as higher-quality alternatives to ‘antiquated’ employee assistance programs (EAPs).
3. Breadth. Instead of only offering therapy, many services have expanded their offering to encompass both higher and lower acuity diagnoses and adjacent well-being solutions. For example, Ginger offers everything from coaching to psychiatry. Modern Health offers management and financial coaching in addition to mental health. BetterUp came from the other side, starting with performance and management coaching, and later adding in mental health offerings.
4. Specialization. There are an increasing number of startups that target specific demographics and diagnoses. For example, NOCD offers services specifically for obsessive-compulsive disorder (OCD), Big Health offers digital therapeutics for insomnia and generalized anxiety disorder (GAD), Cognoa offers digital therapeutics for developmental and autism-spectrum disorders, and Brightline offers services for children and their families.

Proving the value prop

Payors gauge differentiation and satisfaction through net-promoter score (NPS) and engagement.

Net promoter score is measured by asking users how likely they are to recommend the product to a friend or colleague. It is a well-established measure of user satisfaction in most industries.

Engagement is often measured in terms of utilization rates. I.e., the percentage of eligible employees that are actively using the product. Engagement is important to measure for two reasons. First, if you have a high NPS but low engagement, then the NPS is moot because most employees don’t find the offering attractive. Engagement is also important because many products charge on a per employee per month (PEPM) basis, regardless of usage, and employers do not want to be stuck paying for a service that few employees use.

Showing strong NPS and engagement from existing customers will be enough to get the conversation started with a new payor. However, they’ll ultimately want to run a pilot to directly measure the metrics with their own members before committing to a long-term contract.

Another important strategy is to ask benefits directors from large employers to participate in case studies that outline how your product increased employee satisfaction.

Value prop 3: Improved healthcare outcomes

There are several reasons why payors value improved clinical outcomes.

Improved outcomes for its own sake

Non-profit health plans, in particular, are motivated to improve healthcare outcomes for its own sake. The primary mission of these plans is to maximize the health of their covered population, and they are not beholden to achieving quarterly growth and profitability targets for shareholders.

For this reason, non-profit health plans (BlueCross BlueShield of North Carolina is a notable example) are great early adopters of new, pioneering mental health products. You can work with such early adopters to collect data, generate a groundswell of support, and then move onto for-profit insurers and employers that need a shorter-term ROI.

Differentiation and member satisfaction

Better clinical outcomes can be a means to differentiate a plan from the competition, and to increase member satisfaction. For example, many employer offerings point to superior clinical outcomes as a way to distinguish themselves from legacy EAPs. In this way, improved clinical outcomes are a means to achieving a different, more impactful value proposition.

Financial incentives through value-based care

There is an increasing trend within the US to pay for healthcare on the basis of quality rather than fee-for-service. In particular, this impacts commercial health insurers that administer Medicare Advantage (MA) plans.

CMS (the government body responsible for Medicare & Medicaid), assigns MA plans Star Ratings across various dimensions, including “improving or maintaining members’ mental health”. Plans are awarded approximately $500 per member in bonuses when they achieve an average of 4 or more stars across all dimensions. To put this in context, a plan with 1M members can expect to make an additional 500M per year when they hit the required quality targets.

Managed Medicaid plans are also measured based on quality ratings, but are not awarded any financial bonuses based on them. However, the ratings are prominently displayed to consumers to influence what plan they pick.

Proving the value prop

The gold standard is to measure clinical outcomes using randomized control trials (RCTs) with clinical endpoints.

However, RCTs are expensive and logistically complex, so single-arm or retrospective research designs are often used instead. These look at the average clinical outcomes before and after completion of the program. The most common clinical outcome measures for mental health studies are the PHQ for depression and the GAD for anxiety.

CMS measures MA plans’ behavioral health quality using items 4, 6 & 7 from the annual Medicare Health Outcomes Survey (HOS), rather than through clinical trials. Plans are rated based on the percentage of members whose scores are the same or better than expected after two years.

CMS measures Managed Medicaid plans’ behavioral health quality across the following 4 dimension (see exhibit 33):

• Antidepressant medication management
• Follow-up after hospitalization for mental health illness
• Follow-up care for children prescribed ADHD medication
• Initiation and engagement of alcohol and other drug dependence treatment.

Value prop 4: Provider satisfaction and support

Payors track broad measures of provider satisfaction and consider it an important factor in attracting high-quality providers to their network. While provider satisfaction is unlikely to be a primary driver of reimbursement, it is taken into account when making coverage decisions.

Examples of mental health solutions that improve provider satisfaction include tools for easy identification and referrals for mental health conditions (e.g. Quartet), or the ability to provide immediate access to behavioral healthcare through prescription digital therapeutics (e.g. Pear Therapeutics & Akili)

Provider support can also lead to new therapeutics being considered medically necessary, or even becoming a new ‘standard of care’. In such cases, health plans may be legally required to cover the therapeutic, as they would any other medically necessary drug or procedure.

Proving the value prop

Provider satisfaction is evaluated through provider NPS. Specifically, payors will ask providers how likely they are to use the product with a patient, or how likely they are to recommend the product to a colleague.

Clinical society support is also important because payors view such support as a proxy for both provider satisfaction and standard of care. For this reason, many digital health startups (particularly in digital therapeutics) hire medical liaisons and participate actively in clinical society conferences.

Value prop 5: Mental health parity

In 2008, the federal government enacted the Mental Health Parity and Addiction Equity Act. The law — further strengthened by the 2010 Affordable Care Act and various state laws — prevents health plans from providing mental health and substance abuse disorder benefits that are less favorable than their medical/surgical benefits.

Despite the law, many insurers have struggled to provide mental healthcare access to their members. Although plans provide an in-network therapy benefit, there are often far too few in-network therapists to meet member demand. This results in members facing long waitlists or receiving no care at all.

Insurers are experiencing regulatory pressure and negative PR to comply with mental health parity laws. For example, in 2015 Kaiser Permanente paid a $4M fine to regulators for inadequate access to mental health services, and a recent, widely circulated article in Fast Company, criticized its continued lack of access. Over the past 4 years, regulators have assessed $13m in fines for non-compliance with federal and state mental health parity laws.

While the fines are relatively minor (e.g. the $4M fine represented around 0.2% of Kaiser’s $1.8B 2015 operating income), payors are motivated to improve their compliance, even if primarily to avoid negative publicity.

Some startups work with payors to elastically expand their mental health network when demand exceeds the payor’s baseline supply. This is particularly relevant in California, where the government enacted state regulations in 2020 that require plans to arrange coverage for out-of-network mental health services when in-network services are not available.

Bringing it all together

Let’s finish by exploring how a hypothetical company might apply the different value propositions to third-party payors. Not all payors are alike, so we’ll need to segment the market and identify which value prepositions are most important to each type of payor.

Self-funded employers

We can launch our go-to-market strategy by targeting a cohort of self-funded employers known to be fast-moving, early adopters of digital health solutions. These companies — often in Big Tech — are looking for ways to increase employee satisfaction. They tend to be willing to pilot promising new products that do not yet have an established ROI.

As part of these early employer pilots, we’ll run studies looking at our product’s ability to reduce costs, including overall healthcare spend, as well as employee retention and absenteeism. Of course, we’ll also closely measure user satisfaction and engagement. These metrics, in particular, will be critical to converting our early adopter pilots into long-term contracts.

Our early adopter sales won’t be sufficient to get us much further than a Series A, so we’ll need to move quickly to the next wedge of third-party payors: ASOs and more financially conservative self-funded employers.

ASOs (administrative services only) are divisions of commercial insurers that administer the health claims of self-funded employers. They often compete with other ASOs by offering a superior suite of benefits that help employers lower costs and improve employee satisfaction.

Financially conservative self-funded employers (and the ASOs that sell to them) will require a strong 12-month ROI based on a combination of healthcare cost reduction and productivity improvements. We’ll leverage the data from our early adopter pilots to substantiate this value prop.

Fully-insured plans

Next, we’ll start targeting fully insured plans. Due to regulations limiting their profit margins (known as the 80/20 MLR rule), fully insured plans are often less motivated by cost reduction, and more motivated by the potential to attract and retain more members through differentiated benefits and greater member satisfaction.

We can start by targeting some of the non-profit plans that are known to be early adopters. Later on — once we are a well-known brand with a wealth of user satisfaction data — we’ll target for-profit plans.

We can also offer our product to fully insured plans as a way for them to meet mental health parity requirements and alleviate pressure from members and the press. They will value our ability to elastically scale their behavioral health capacity when needed.

Medicare Advantage

Around this time, we can also target Medicare Advantage plans. In addition to showing cost reduction and high member satisfaction, we’ll run pilots demonstrating that the use of our product materially improves the health outcomes tied to Medicare Star Ratings. We can also run studies showing that our service reduces the medical costs associated with high-cost comorbidities — like diabetes and heart disease — that are prevalent in Medicare populations.

CMS

Finally, we’ll target CMS, itself, which is responsible for non-managed Medicare and Medicaid. By this time, we will have established a strong case for cost reduction, improved health outcomes, and have a groundswell of provider satisfaction and support. The rest of the healthcare industry tends to take CMS’s lead on reimbursement, so once we gain CMS coverage, we will be well on our way to long-term commercial success.

Your path to commercial success

One silver lining of the COVID-19 pandemic is that mental health is finally getting the attention that it has always needed and deserved. Digital behavioral health is experiencing a deluge of funding and third-party payors are more motivated than ever to offer mental health solutions.

The time is ripe to partner with payors. However, proving compelling value propositions takes time and considerable effort. To be successful you’ll need to start early and learn from the successes and failures of those who came before you.

I hope the insights I’ve shared will help you along the way.

Thank you to Maitreyee Joshi, Ross Klosterman, Joe Connolly, Elise Ogle, Mel Goetz, Jon Sockell, Ben Lewis, Matias Sanchez, Ashley Bland, Arun Bhojwani, Ryan Lawler, Annie Li, Cady Macon, and Ziv Reichert for feedback on drafts!

This article was originally published on behavioralhealthpm.substack.com. Consider subscribing to get future articles like this direct to your inbox.

Footnotes

1. The economics of cost reduction appears simple: lower costs lead to more profits for the payor. However, they are more complex than they seem. Fully insured plans, in particular, are bound by the 80/20 medical loss ratio (MLR) requirement. A fascinating analysis in the American Economic Journal showed that the regulation inadvertently incentivized the plans to increase rather than reduce their costs. It’s therefore important to understand the specific value props that each type of payor (and plan) values.
2. Moreover, there is a serious shortage of mental health providers in most payor networks. Many members simply cannot find an available therapist that takes their insurance. This shortage of providers results in many members not accessing any care at all, which further lowers the first-order costs of mental health conditions for payors.
3. For example, a study commissioned by the UK’s NHS examined 15 forms of prevention and early intervention in mental health. It showed a strong ROI (up to 18x return), but only when measured on a 10-year time horizon. Unfortunately, such pay-back horizons are often only feasible when financed by a single-payer health system.

Value Propositions for Mental Health Tech: All Roads Lead Back to the Payor was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The COVID-19 pandemic has disrupted norms and created new stressors for people of all age groups. For many young people, it has stripped away outlets of joy and release — whether that’s the loss of high school classes, sports teams, or simply just time away from home to grow independently. In this shrinking world, technology allowed us to connect with others. But this connectivity can be both a lifeline and a strain on mental wellbeing.

To learn more about what this experience has been like for young people, we reached out to Lexi and Senti: two 18-year-olds who served as youth advisors through the Youth 2 Innovator (Y2I) program established by the Headstream Accelerator in 2020. They have been thinking critically about how the pandemic has shifted dialogue about mental health among teenagers — and how navigating digital spaces can be a piece of this puzzle.

Lexi, who uses she/her/hers pronouns, is a Bay Area local and advocate for purposeful tech usage for young people. Lexi finds her peace in the outdoors, whether surfing on the California coast or trail running in the Redwoods, and will be playing field hockey next year at Vassar College.

Senti, who uses ey/em/eirs pronouns, feeds eir soul through music and social justice work. Senti studies songwriting through the Berklee College of Music and works as the Tennessee state director for Generation Ratify, a youth-led feminist advocacy organization that focuses on causes including climate justice, criminal justice reform, and queer liberation.

For those like Lexi, quarantine invited a deep sense of self-reflection. Discovering new things about herself shifted many of her relationships, while talking through her experiences with friends helped boost her mood during the “high highs and low lows” of the past year . “At first, I didn’t really legitimize [those fluctuations],” says Lexi. “I thought that because it was never something that I had really gone through before, that it wasn’t real.”

Meanwhile, for Senti, the pandemic presented an opportunity to slow down and take steps to support eir mental health, away from the daily hustle and bustle of pre-quarantine life. “The pandemic has allowed me to gather all the resources I can, which maybe I wouldn’t have had time to do [otherwise],” says Senti, pointing to the utility of digital apps like Moody, Minddoc and Calm.

Digital tools like these make it possible for teens to explore mental health resources on their own time, on their own terms. Lexi points out that, though adults use technology and played a role in creating it, they can be quick to criticize young people for screen time. She feels it is crucial for adults to understand that teens can uplift their lives while navigating digital spaces. “There is a lot of miscommunication about what it means for us to ‘live’ on these digital spaces and what we use them for,” says Lexi. “It feels like there’s this idea from adults that if we aren’t checking our emails, or doing really important professional things, it shouldn’t matter — but I think that’s false.”

From Senti’s perspective, adults need to acknowledge the everyday necessity and ubiquity of tech with more nuance. “It’s not like we want to have a phone addiction,” Senti says. “Adults should have patience and understand that our phones are literally a world in a box, and not just a singular thing.”

Tech has also led to the growth of a vast digital landscape full of supportive spaces, such as community youth groups for young people to connect about mutual struggles and challenges. Both Lexi and Senti, who have facilitated group conversations in these digital spaces, have witnessed how such discussions can build a powerful sense of empathy. “Something I’ve found that young people always respond to, especially myself, is separating yourself from clutter and finding a group of people who really understand what you’re experiencing or maybe have experienced similar things,” Lexi says.

She’s hopeful that the pandemic has shifted dialogue around mental health, and that it will create new opportunities for understanding. “When it comes to people relating to other people, the pandemic gave everyone a baseline to connect on,” Lexi says. “Everyone was going through it. A lot of people were going through it a lot worse than others, but still, everyone had something that affected them.”

Senti hopes the pandemic will shift expectations around mental health and break down boundaries that preclude people from sharing how they’re doing with peers, teachers or other people in their life. “I hope that disorders and disabilities stop mattering so much,” says Senti. “When you ask how someone is doing, don’t let ‘good’ or ‘ok’ be the only acceptable — or expected — responses.”

Above all, this challenging year has reinforced the fact that people never fully know what others are going through. Whether those ups and downs are visible or not, it’s critical to “allow space for people having a bad day,” Senti says.

Teen Perspectives: Navigating Digital Spaces to Connect About Mental Health was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

As we’ve written about before, engagement is a primary challenge faced by the digital health industry today. Our industry has created interventions that are clinically effective and safe. Yet, these same interventions often struggle to sufficiently engage end-users, and consequently, fail to achieve their real-world potential.

Researchers and practitioners are increasingly turning to co-design as an essential strategy for improving engagement. Co-design (sometimes referred to as co-creation or participatory design) is a structured process for engaging end-users and stakeholders as creative partners in the design process in order to understand their needs and fine-tune product features to be as engaging and impactful as possible.

The reason for incorporating co-design is simple: We, as product developers, are not our users. Due to differences across age, culture, life experience, and cognition, what is engaging to us is often different than that which is engaging to our users. For example, at Limbix, we are building an intervention for adolescent depression. However, none of the core team are adolescents and few of us have lived experience with depression. It is therefore critical to involve prospective end-users as active participants in our design process.

Yet, while there is widespread agreement that co-design is essential for creating engaging digital health interventions, there is limited guidance on how it can optimally be undertaken. Unfortunately, we cannot simply draw upon generic guidance on co-design because of differences between digital health interventions and the products of other industries.

At Limbix, we have invested heavily in co-design as a core pillar of our engagement strategy. We have developed a five-step process for co-design that is grounded in the broader literature but fine-tuned to the unique demands of digital health intervention development. We apply this process iteratively throughout the product development lifecycle.

1. Recruit participants that are as similar as possible to your target users
2. Prioritize the problem based on impact and risk
3. Select the right tool based on the problem you are trying to solve and the phase of product development.
4. Run the session, taking into account the sensitive nature of the material.
5. Validate and iterate, building on past feedback for continual improvement.

Step 1: Recruit participants

Recruiting participants is an essential but time-consuming step of co-design. Here are a few principles to keep in mind to make this critical step as time-efficient and effective as possible.

1. Recruit participants that are representative of your target users. You cannot assume that what’s engaging for one group will have the same effect on another population. It’s critical to ensure that your participants adequately present the varied characteristics of your target users across dimensions such as diagnosis, symptom severity, age, sexual and gender identity, and cultural background, which can all impact engagement.
2. Build a council. While time-consuming upfront, building a stable cohort of participants to provide regular product input can be a tremendous time-saver over the long run. It will ultimately allow you to spend less time on recruitment and more time on designing. Plus, we’ve found that participants love the opportunity to play a long-term role in developing a product that will help others! At Limbix, we call this group our Teen Advisory Council, and it includes 12 adolescents with experience of depression.¹
3. Ensure safety and ethics up-front. You will often collect sensitive and personal information during co-design, so it’s essential to protect participant privacy by gathering informed consent, creating clear protocols for data storage and protection, and evaluating 3rd party vendors & suppliers for data security. When developing a regulated medical device, you may also need institutional review board (IRB) approval, which ensures that your research protocol meets high ethical standards for the protection of human subjects. At Limbix, we’re building a medical device so our design research is conducted under the purview of an IRB protocol and subject to Good Clinical Practice (GCP) guidelines.

Step 2: Prioritize the problem

You have limited time and resources, so it’s important to prioritize your co-design efforts. We recommend prioritizing co-design efforts based on two dimensions: impact and risk.

Impact refers to how likely a feature is to impact users’ clinical (health) outcomes. Digital health interventions are like medicine in that they have active ingredients (clinical content), and those active ingredients are only effective if users sufficiently engage with them. The more a new feature impacts users’ interactions with an active ingredient, the more important it is to incorporate co-design.²

For example, imagine that you’ve identified that in-app exercises are an ‘active ingredient’ because your data shows that completion of in-app exercises causally impacts users’ clinical outcomes. If you’re designing a feature that directly impacts exercise completion rate, then you should incorporate co-design into the feature’s development. In contrast, you wouldn’t spend much time — if any at all — involving users in developing a log-in screen, since the login experience is unlikely to impact engagement with your active ingredients.

Risk has two components: 1) How confident you are in your ability to design for your users’ needs without their direct input and 2) the level of effort required to build the feature.

Turning back to the example of the login experience, login flows are well-established, low-effort features that can be well-designed without user input. By researching established flows instead, we can save time and resources that can be redirected to seeking user input on other features that we have lower confidence in. In contrast, it would be very important to get user input on a complex, high development-effort feature like a reward system, which can be implemented in a variety of ways and has a well-known history of unintended negative effects when implemented poorly.

Let’s bring this all together with an example. Let’s say you’ve identified the completion of in-app exercises as your target engagement metric because it causally impacts clinical outcomes.

Based on your understanding of engagement techniques and prior user feedback, you believe that a reward system could be an effective way to drive exercise completion. You want to build a reward system into your next release and need to determine whether to incorporate co-design techniques into its development.

You score the reward system as high impact because it touches a clinically impactful area of your intervention. You also score it as high-risk, because it is a high effort feature and poorly implemented reward systems can have unintended negative effects. The reward system is, therefore, an ideal candidate for co-design.

Step 3: Select the right tool for the job

Co-design is an umbrella term for a range of tools used to engage end-users in the product development process. It’s critical to choose the right tool for the design problem, and the phase of product development.

Here are a few of the tools we use most:

• Problem interviews — sometimes referred to as scoping interviews — are often used at the start of product development to identify user needs and how users are currently addressing those needs. The outputs from these sessions feed the problem ‘backlog’ that forms the basis of prioritized roadmaps.
• Co-design is, somewhat confusingly, the name for a specific technique, as well as the name for the overarching category of techniques. The co-design technique, itself, involves working with participants to generate potential solutions to a design problem. This often involves a design studio with Crazy-8s, in which participants sketch out potential solutions, share them with each other, and then iterate. We usually start a feature’s development with a co-design session.
• Group critiques and brainstorming — sometimes referred to as a solution interview — is a lightweight version of co-design. We’ll brief participants on the design goal and show them early designs. We’ll then ask participants for feedback and ask them to brainstorm improvements together. We tend to use group critiques for lower-risk features, where we have a good sense of the solution but want to quickly calibrate it before moving forward.
• Usability tests involve users completing defined tasks within a prototype while they think aloud. The goal of these tests is to assess how easy the intervention is to use and understand. Comprehension is particularly important for many behavioral health interventions because clinical improvement often relies upon the learning of core psychological concepts.
• User interviews involve collecting feedback from users after they complete the intervention. At Limbix, as part of every release, we interview a sample of users, asking them a series of structured questions about their experience. This feedback is essential, and many of our best feature ideas have come directly from these interviews.
• Diary studies involve asking users to document their experiences and thoughts while completing the intervention. The problem with post-study user interviews is that memories are notoriously biased. Diary studies enable us to get in-the-moment feedback that more accurately represents users’ experiences. We tend to complete diary studies on minor releases that occur in between our larger clinical trial releases.

Let’s now turn back to the example and choose a technique for developing the reward system. You’ll want to involve users as early as possible in the process, so you might choose to start by running a co-design session, in which you brief users on the design objective and then generate some initial solutions together. By the end of the session, you’ll aim to have a range of ideas, some far-fetched, others more realistic, and have a good sense of what will be engaging to your users.

Step 4: Run the session

Everything leading up to this point has just been preparation. Now you get to the fun part! The specifics of each session will depend on the tool that you’re using. However, here are some overarching principles for running effective sessions when developing digital health interventions.

Keep participants engaged during sessions. You’ll need to keep participants engaged to get the most out of sessions. Here are a few tips that are especially important for co-designing digital health interventions:

• Use icebreakers. Oftentimes, sessions can dive into personal topics related to a participant’s health. It is important to help people feel comfortable and warmed up before starting to discuss sensitive topics.
• Use inclusive language. Never assume to know what someone’s experiences are, especially when you are discussing mental health and wellness. It’s also important to remind people that there are no wrong answers and that everyone’s experience is valid and valued.
• Keep the diagnosis in mind. There may be special considerations to keep in mind as you interact with users that have a particular diagnosis. For example, at Limbix, we often work with adolescents with depressive symptoms, and they may occasionally miss a session due to their symptoms. In such cases, it’s important to be patient and understanding.
• Keep it short. Participants with certain conditions may lack motivation or stamina for longer sessions. It’s, therefore, better to structure sessions into short, concrete exercises that take less than an hour.
• Fairly compensate participants. Compensating participants goes a long way to keeping them engaged. However, it’s also important to not overly compensate participants, as too much compensation can unduly, and unethically, influence users to participate in research that they would otherwise be uncomfortable participating in.

Use different participants for ideation vs prototype feedback. It’s important to be able to ideate with one group and then get feedback from another. If the same group ideates on initial concepts and then gives feedback on prototypes, the feedback can be biased because it was based on their own ideas! For this reason, we generally run initial ideation sessions with around 5 participants, and then work with a different set of 5 participants for prototype feedback.

Make the most of remote sessions. Due to the COVID-19 pandemic, we had to move all of our co-design sessions online. While initially challenging, remote research ultimately became a huge advantage. We previously only recruited participants who could physically get to our office. Now we recruit participants from all over the US, which enables us to involve people with more diverse backgrounds and perspectives.

Step 5: Validate and iterate

A single research session is just the beginning of a much larger process. After each session, you’ll need to synthesize and implement your findings, and then consider whether or not to run follow-up sessions using other tools.

Turning back to our reward system example, after running the initial co-design session, you might build some initial prototypes, and then run a group critique & brainstorming session to gain additional feedback. Then you might create a high-fidelity clickable prototype and run a usability test, followed by a diary study that simulates real-world usage of the feature when integrated into the full intervention. Throughout this process, you should also be getting regular feedback from clinical experts to ensure clinical efficacy and safety (which can be viewed as a parallel, but interrelated, cycle of co-design).

Finally, you’ll launch the feature as part of a wider product release and can measure how the introduction of the new feature impacts your quantitative engagement metrics and clinical efficacy. You might also run in-depth post-intervention user interviews to understand how the introduction of the feature impacted the users’ experiences, and generate suggestions for future improvement.

Conclusion

Co-design is one of the most important strategies available for driving engagement in digital health interventions. At Limbix, we incorporate co-design into all phases of our product development and firmly believe that it has played an outsized role in the strong engagement we see in our products today. While there is no ‘magic bullet’ engagement strategy, it really is hard to overstate its importance.

We’ve shared what we believe to be our most important insights into co-designing for digital health engagement. However, there’s still a lot more that we could have addressed, so feel free to reach out if you have any questions. Nothing excites us more than discussing this important topic with others who share our passion!

Thanks to: Judy Liu, Jessica Lake, Jim Liu, Freddy Tang, Tom Hallam, Lindsay Lee, Shikha Nalla, and Chris Sowers for insightful feedback on drafts. Thank you also to Stella Kim for shaping our thinking on this topic.

Footnotes

[1] One important caveat is that users may become biased as they become increasingly familiar with your product, and therefore be less representative of first-time users. It’s therefore worth periodically testing your product with naive participants in addition to your council.

[2] For a detailed explanation of this concept, read the article on meaningful engagement metrics.

Co-design: The secret sauce of digital health engagement was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Engagement is arguably the single most important challenge in digital health today. Yet, while we recognize that engagement is important, there is a distinct lack of clarity regarding what exactly engagement is, and even more importantly, how to measure it.

Increasingly, digital health technologists and researchers agree that we need to find measures of meaningful engagement. That is, we must avoid the temptation to immediately use generic measures like number of sessions, weekly active usage, or program completion, and instead follow a data-driven approach to find the specific engagement metrics that uniquely predict long-term value in each intervention.

The quest for meaningful engagement metrics is not unique to digital health. In fact, many of the most successful technology companies have sought out such a metric, which when found, is closely linked to their ‘aha moment’. For example, Facebook identified ‘number of friends’ as its key engagement metric, and found that long-term retention was greater when users added 7 friends in 10 days. Slack, Twitter, Zynga, LinkedIn, and Twitter have all shared similar aha moments. A key principle is that meaningful engagement metrics are usually specific to the product. While it’s certainly possible that a generic measure of engagement could also be meaningful, it is rarely the case.

A key principle is that meaningful engagement metrics are usually specific to the product.

Finding ‘aha moments’ can be decomposed into two parts. First, you must find meaningful measures of engagement (e.g. friends), and second, you must identify critical levels of those engagement metrics that, once achieved, predict long-term user value (e.g. 10 friends in 7 days). In this article, I’ll cover the first step, finding meaningful measures of engagement. I hope to cover the second step in a future article.

Drawing from work in the consumer and SaaS industries, I’ve identified a three-step process for finding meaningful engagement metrics in digital health interventions.

1. Identify an indicator of value. For most consumer and SaaS products, it’s retention. For most digital health products, it’s clinical outcomes.
2. Identify potential leading indicators of that value. This is an iterative, exploratory process of generating competing hypotheses.
3. Determine whether the indicators are causal or correlational. Finally, you must validate your hypotheses by confirming or rejecting causality.

Step 1: Find the indicator of value

Indicators of value are one area in which there is a fundamental difference between digital health interventions and most consumer or SaaS products. For example, in a consumer product like Facebook, users are looking to be entertained, and therefore retention (consistently coming back to the app) is an excellent proxy for that value. Similarly, in a SaaS product like Slack, users want to achieve a specific objective, such as communicating with their colleagues. Once again, retention (consistently coming back to communicate with colleagues) is an excellent proxy for that value.

For most digital health interventions, the primary indicator of value is clinical outcomes

However, for most digital health interventions, the primary indicator of value is clinical outcomes.¹ Users have health needs and utilize digital health interventions to address those needs. If users come back to the app every day for months (strong retention), but their clinical symptoms do not improve, then they haven’t received the intended value from the product.

Moreover, in most digital health products there are not one, but three key stakeholders — patients, providers, and payors — all of whom value improved clinical outcomes. In this regard, clinical outcomes are a unified indicator of value for all major stakeholders.

Let’s take the example of a digital health intervention for depression. Patients, providers, and payors all care about decreasing depressive symptoms. We’ll continue with this example below.

Step 2: Find leading indicators of value

Now that you’ve identified a measure of value, it’s time to find leading indicators of it. Once found, these will become your metrics of ‘meaningful engagement’.

At this stage, it’s important to resist the temptation to immediately use a generic engagement metric. As mentioned earlier, it’s certainly possible that a generic measure of engagement will be a leading indicator of value, but by and large, meaningful engagement metrics are unique to your product.

To find and validate leading indicators of value, I recommend following the three-step subprocess outlined below.

Step 2a: Identify initial candidates based on theory and external data

If you’re building a new product from scratch, then you’ll need to rely on theory or external data to generate your initial hypotheses. Many digital health interventions are digitized versions of face-to-face interventions, and you can look at the predictors of clinical outcomes in those face-to-face interventions to generate hypotheses.²

Many digital health interventions are digitized versions of face-to-face interventions. You can look at the predictors of clinical outcomes in those face-to-face interventions to generate hypotheses.

Let’s turn back to our example. Let’s say your digital intervention for depression is based on Cognitive Behavioral Therapy (CBT). You investigate what predicts positive clinical outcomes in face-to-face CBT, and identify multiple predictors, three of which are:

1. Showing up for weekly appointments
2. Doing assigned homework (e.g. completing thought records and cognitive reframing exercises)
3. Having a positive therapeutic alliance with the therapist.

Step 2b: Map the theoretical indicators to specific, measurable actions within your intervention

Continuing with our example, your next step is to explore digital analogs for each of the traditional predictors:

1. Showing up for weekly appointments corresponds to weekly active usage (a generic engagement measure).
2. Doing assigned homework corresponds to completing in-app exercises.
3. A digital-analog of therapeutic alliance is harder to identify because it’s a psychological process and is therefore difficult to measure with behavioral metrics (e.g. passive logs). However, you could potentially include a questionnaire that measures therapeutic alliance as part of weekly symptom check-ins, or could even experiment with sporadic prompts within the app.

Step 2c: Validate the proposed relationships with internal data

Once you’ve developed your hypotheses, it’s time to look at the correlation between the hypothesized leading indicators and clinical outcomes. Let’s say you find that:

1. Weekly active usage has a weak, non-significant, relationship with clinical outcomes.
2. There is a moderate, statistically significant relationship between completing specific types of exercises within the app and clinical outcomes.
3. After exploring therapeutic alliance further, you realize that while it is theoretically promising, it’s too difficult to reliably measure. It’s therefore not feasible to use as a primary engagement metric.

Based on the above, you choose completing in-app exercises as the most promising engagement metric and move onto step 3.

Step 3: Determine whether the relationships are causal or correlational

At this point, you’re probably thinking “But wait, correlation doesn’t equal causation!”, and you’re absolutely right. That’s why in this next step you’ll determine whether the hypothesized leading indicator (e.g. completing in-app exercises) causes the desired outcome (improved depressive symptoms), or whether there is actually some third factor (e.g. users’ pre-existing motivation to change) that leads users to both complete the engagement metric and improve their clinical outcomes.

You can test for causality by shipping new product features that increase your target engagement metric (e.g exercise completion rate), and then see whether that leads to a corresponding improvement in your clinical outcome metric (e.g. PHQ-9). For example, you might implement additional reminder notifications for exercise completion, or design a reward system that incentivizes completion of exercises.

Once you ship the feature, you’ll first confirm that the engagement metric does in fact go up as a result of the feature. If the target engagement metric goes up, you will then check whether the correlation with clinical outcomes is maintained. If the correlation holds, then that’s great support for your choice of engagement metric. Congratulations, you’ve likely found a measure of meaningful engagement!³

Putting it all together

Finding meaningful measures of engagement is just the beginning. There are several important next steps that I will touch on briefly.

Blending multiple metrics

There’s rarely only one metric of meaningful engagement in an intervention. Instead, there are probably multiple ones, in which case you’ll be better off combining them into a hybrid measure of engagement. For example, you might categorize someone as meaningfully engaged if they do any 2 out of a list of 5 leading indicators within the program in a given week. Or you might want to create a hybrid metric that blends various factors into a single weighted average engagement value.

Segmentation

Measures of meaningful engagement are also likely to differ across users. E.g., users with more severe symptoms might benefit from a different style of engagement than users with mild-to-moderate symptoms. Optimal engagement style is also likely to depend on users’ pre-existing motivation levels, and may even change for the same user as they progress through the intervention. SilverCloud and Microsoft recently published an article that outlined their use of machine learning to identify different engagement styles. One of the major advantages of digital health interventions over traditional therapies is the ability to collect large amounts of data, which in turn, enable the personalization of interventions to match individual engagement characteristics.

Minimum effective dose

As alluded to at the start of this article, finding meaningful engagement metrics is only half of the equation. You must also determine the minimum effective dose of the engagement metric that predicts long-term user value. E.g., if you determine that completion of in-app exercises leads to improved clinical outcomes, you’ll next want to determine how many exercises should be completed over what time period in order to produce the greatest likelihood of clinical improvement.

Letting your metrics guide your product development process

Finally, once you’ve determined your meaningful engagement metrics and minimum effective dose, you should leverage your engagement toolkit to drive the specific engagement behaviors that lead to clinical outcomes.

If you’ve made it this far in the article, then there’s a good chance you’re as passionate about driving engagement in digital health as I am! I genuinely believe that it’s one of the most critical challenges facing our industry today (apart from perhaps commercialization). While there is no single, ‘magic’ engagement metric, the process outlined in this article will position you well to identify the meaningful engagement metrics that are an important milestone along the journey to building truly impactful interventions.

Thanks to: Product Manager Mel Goetz, Head of Content Elise Vierra, and Director of Science & Innovation Jessica Lake for their contributions to this article.

Footnotes

[1] One notable caveat is for wellness products, like meditation apps, for which retention is arguably the best indicator of value. For example, if a user downloads a meditation app and comes back consistently over time, then the product is probably giving them their desired value. Also, since wellness apps generally have direct-to-consumer business models, you don’t need to be as concerned about payors’ and providers’ emphases on health outcomes. In such cases, you may be better off using retention rather than clinical outcomes as your primary indicator of value.

[2] Leading indicators of clinical improvement are also closely related to the concept of active ingredients. Just as there are active ingredients in traditional pharmaceuticals surrounded by the remainder of the pill — which serves as a delivery mechanism — so too, there are elements of digital interventions that are critical for clinical improvement that are surrounded by the remainder of the software. In fact, there is a whole field of research into mechanisms of change within psychotherapies that seek to identify active ingredients, and such research can provide a great starting point for identifying what may be the leading indicator of clinical improvement within your digital intervention.

[3] An important caveat here is that many digital health interventions do not have sufficient data to test for causality. For example, if your product is a prescription digital therapeutic going through initial clinical trials, then you’re unlikely to have a large enough sample size to draw meaningful conclusions about the relationship between individual engagement metrics and clinical outcomes. If that’s the case, then you’ll have to stop at step 2. However, if you’ve identified a moderate-to-strong correlation that is supported by a robust theoretical rationale for causality, then you can have some confidence that you’ve probably found an important relationship.

‘Aha moments’ in digital health: How to identify engagement metrics that impact clinical outcomes was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Each week, clinical trial participants using Limbix Spark complete the Patient Health Questionnaire (PHQ-8), a measurement of depressive symptoms, in their app. The PHQ-8 is an important measurement for determining the effectiveness of Spark, an investigational treatment for adolescent depression — but it’s also one of the ways the Limbix team makes sure the teens in their clinical trials are safe. If a teen’s PHQ-8 score passes a certain threshold, indicating that their depressive symptoms are more severe, a push alert is sent to Limbix’s study team. They then determine a course of action to ensure the participants’ safety.

It sounds simple enough, but behind that chain of events are lines of code. And behind that code is a small but mighty engineering team led by Limbix CTO, Judy Liu.

Liu is acutely aware that the stakes are high for every engineering choice made for a medical device: especially when teens are on the other side of the screen. If anything in the code breaks, notice of a user’s PHQ score deterioration might not go out to the study team — and a user might not get the help they need. Every day, Liu’s team is reminded how much the details behind their work matters.

Liu, for her part, finds herself energized by this kind of critical thinking — “really understanding the environment that your product is used in, and really understanding the motivation behind a product,” she says. Throughout her career in tech, the Illinois-turned-California local has worked hard to deeply consider product ecosystems. Whether Liu was working on mobile apps for companies like Palm and TripIt, or working on virtual reality at Facebook, she was always drawn to bigger questions: Why are we building this feature? How will stakeholders use it? What purpose does the product serve?

This ethos has informed her role since she joined Limbix in 2017. As the CTO of a growing start-up, Liu confronts questions of context every day. The job is dynamic and involves more than simply building code, she explains: the engineering team is constantly in conversation with the research, product, and UX teams about why they’re building what they’re building.

Those conversations involve answering questions about how providers and patients will interact with each feature, or how Spark’s back-end will integrate with electronic health records (EHR). For Liu and her engineers, data privacy and HIPAA compliance are a top priority as they build and iterate on Spark.

The engineering team also manages Limbix’s “data warehouse,” which consists of de-identified product data that can shed light on trial participants’ engagement with the app, as well as their symptoms of depression, anxiety, and functional outcomes. This data helps the research and product teams understand how teens are interacting with Spark. Beyond monitoring that data, Liu’s engineers have designed tracking measures for identifiable personal and health information in order to build “an audit trail, essentially, for these pieces of data” — which is a standard healthcare requirement, Liu explains. These tracking measures help Limbix’s clinical researchers understand when the data has been viewed and by whom.

She’s the first to admit that building the bridge between complex health care regulations and engineering involves a steep learning curve. Deep subject matter knowledge has proven critical to Limbix’s mission, just as much as technical expertise. “How do you translate this sort of vague regulatory rule into code?” she says. “What does it mean that there’s an audit trail for e-signatures? What type of security policies do we need to have in place?”

For Liu, answering these questions is a welcomed challenge. “What really gets me excited is that if we can get this to work, we can really build a scalable solution to provide evidence-based care to a lot of kids out there…at very accessible price points,” she says, envisioning a world where all kids enrolled in Children’s Health Insurance Program (CHIP) could eventually use Spark.

“The prescription digital therapeutics industry is brand new, and it’s trying to figure out how it fits in the healthcare ecosystem. We’re kind of the new kids on the block,” she says, with a laugh.

It’s a fitting quest for Liu. After all, synthesizing a landscape, translating complexities into code and identifying the specific value-add that a product can offer its industry traces back to the intellectual challenge that has animated her career.

Putting Compassion into Code with Limbix CTO, Judy Liu was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Raised in an Indian-American immigrant family, Dr. Padmanabhan watched friends and relatives navigate mental illness while observing the ways that both cultures stigmatized their experiences. Growing up, she recalls that it was difficult for them to voice feelings about mental health. “Not being able to talk openly about these issues early in life sort of shaped the way that I perceive them now,” Padmanabhan says.

Those lived experiences led her down a path of academic research to understand human brain function and behavior. After studying cognitive science as an undergraduate at Carnegie Mellon University, she worked as a research assistant at the National Institute of Mental Health, and then pursued a PhD in cognitive psychology at the University of Pittsburgh. There, she focused on adolescent development and autism spectrum disorders . Dr. Padmanabhan was a researcher at Stanford University before joining Limbix in 2019.

Today, study operations are her key focus. Conducting research is a mission-critical step on the way to developing a prescription digital therapeutic that can be prescribed in doctors’ offices. Padmanabhan helped oversee Limbix’s first trial, or feasibility study, from 2019 to 2020. This smaller-scale trial of up to 30 participants was meant to gauge whether the therapeutic was feasible at a baseline level. “Are teens adhering to it?” she says. “In our context… it is important to know if this is something that can even be done.”

Dr. Padmanabhan’s team looked for metrics such as completion rate of the self-guided intervention along with preliminary indicators of efficacy, like reduction of depressive symptoms. At the same time, her team conducted qualitative interviews to gain feedback, understand users’ challenges and prompt product improvements.

These days, Padmanabhan is knee-deep in Limbix’s ongoing “real-world evidence trial,” which began in 2020 and is fully virtual. Remote operation has enabled the team, which recruits through “community samples” (social media channels and word of mouth) to achieve broader reach by enrolling over 200 teens across the U.S.

Eventually, Dr. Padmanabhan imagines partnering with primary care clinics that will enroll local patients. But for now, remote operation could enhance Limbix’s equity goals. Future trials for FDA clearance will involve representative samples, and the team is thinking critically about how to serve teens from low-income communities, and racial and ethnic minority groups in particular, says Padmanabhan. Beyond data on race, ethnicity and socioeconomic status, they also track rural, urban and suburban diversity — all in the aim of creating a product that’s accessible for teens of all backgrounds.

Already, research has provided rich insights. Padmanabhan says the variability she’s seen is eye-opening, and participants’ preferences have significantly informed product design and iteration. “It’s hard to make a one-size-fits-all,” Padmanabhan says.

In fact, Spark is designed to do just the opposite, by empowering teens to determine the incremental steps that feel meaningful to them. “Personalization is really important so that people are setting goals to meet them where they’re at, and not necessarily feeling like they have to think bigger than what they’re capable of at any given moment.”

Teens may often bring varied experience with, and trust in, mental health care. For some, Spark may represent their first time seeking help for depression; others may have tried therapy through traditional means. When serving this population, considering family dynamics is especially important: people under 18 need parental consent to participate, potentially limiting options for kids with complicated family relationships, or without parental support.

Understanding nuances that shape teens’ day-to-day lives is integral to Limbix’s work. Dr. Padmanabhan knows adolescence is a critical point of development, when many disorders emerge or begin. “Intervening at that time is critical to try and alter the trajectory of illness,” she says.

And she’s energized by novelty. “Mental healthcare has not changed much in decades,” Padmanabhan says. “We’ve had talk therapy and drugs for a very long time. I really see this as a new frontier for innovation.”

Conducting remote clinical trials with teens was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Limbix, our vision is a world where all adolescents have access to effective, affordable behavioral healthcare, and we believe one of the best ways to achieve this vision is through accessible digital interventions. Engagement is one of the primary challenges to achieving this vision, and to this end, our product team invests heavily in understanding how to drive engagement.

Just as medicine is only effective if patients take the minimum effective dose, so too, digital health interventions are only effective if patients sufficiently engage with them

We follow a three-step process grounded in the well-known build-measure-learn framework, with an emphasis on theory, co-design, and quantitative & qualitative feedback:

1. Identify engagement drivers, with an emphasis on theory. In order to build engaging products, we must first understand the theoretical underpinnings of digital health engagement. Based on psychological & human-computer interaction theory, and published studies, we develop hypotheses to test in the subsequent steps.
2. Fine-tune with user testing and codesign. User-centered design is the ‘secret sauce’ of engagement. In order to create an engaging intervention, it’s critical to co-create and validate features with the people who will use them.
3. Measure and iterate with both quantitative and qualitative feedback. Once we’ve shipped a feature, we measure its impact on engagement and iterate based on both quantitative usage data and in-depth qualitative interviews.

This post is the first in a series in which we’ll share our process for building engaging digital health interventions. In this post, we explore the first step of our process: the theoretical underpinnings of engagement in digital health interventions.

The elements of digital health engagement

The academic literature is an excellent starting place for understanding what drives engagement. Researchers have conducted several comprehensive reviews, and between them, they have identified upwards of 60 different factors that are hypothesized or known to influence digital health engagement.

In this post, we’ve outlined the factors that have most influenced our thinking. For each, we explain what the engagement factor is, summarize research about its effectiveness, and provide examples of how it’s been implemented. To assist with implementation, we’ve divided the techniques into three groups:

1. Foundational. These are the essential elements that you should ensure every digital intervention has.
2. Straightforward with strong empirical support. After implementing the foundational elements, spend your efforts on implementing these relatively straightforward and well-established engagement techniques.
3. Experimental or complex. Finally, consider implementing techniques that either have limited, but promising empirical support, or are well-established but require careful implementation.

There is no single, magic-bullet, engagement factor.

It’s important to point out that there is no single, magic-bullet, engagement factor. The techniques have synergistic effects and work best when implemented together. All factors mentioned in this article have good theoretical and/or empirical support for their effectiveness and the difference between their impact is small. Even the most well-established factors still only have a relatively minor impact in isolation (e.g., one study estimated reminders increase engagement by 4%). However, when multiple factors are used together, the effect on engagement can be truly impactful.

Foundational techniques

Usability

In the context of digital health interventions, usability is defined by how natural and frictionless the interventions are. Usability is best understood as a potential barrier to engagement rather than something that increases engagement on its own. Recent studies comparing engagement across multiple web-based digital health apps found either no relationship or a negative relationship between usability and engagement. The authors explained that very simplistic, content-thin apps can have high usability scores, but can ultimately be very unengaging. The authors concluded that usability is a necessary but insufficient factor for engagement. Take the example of interactivity, which improves engagement by immersing users into the digital content. A buggy or counterintuitive interface (poor usability) takes users out of the experience and thereby diminishes the effects of interactivity on engagement.

Usability is best understood as a potential barrier to engagement rather than something that increases engagement on its own.

Usability is particularly important in digital health interventions for conditions in which users’ cognitive or physical abilities may differ from the general population. For example, users with depression often struggle with focus and motivation, and in such cases, intuitive design is critical to sustaining engagement.

Visual design

Visual design refers to the look and feel of the intervention’s interface, and is thought to improve engagement by attracting the attention of the user, stimulating curiosity, and giving the user a sense of delight. Researchers have suggested that aesthetic appeal can increase the persuasiveness of messages by leading to greater cognitive absorption and processing of the information. This is particularly important in digital health interventions, where comprehension of content is often critical to the effectiveness of the intervention. Visual design can also impact engagement through the well-established effect of color on emotion.

A 2018 study found that better visual design was positively associated with usage metrics for mobile but not web-based apps. The authors did not discuss the reasons why this occurred, but it may be that visual design is more impactful in some delivery formats (e.g. mobile apps) than others.

Similar to usability, good visual design is a foundational element for strong engagement. Poor usability and visual design was something that plagued early digital health interventions. However, as digital health interventions have gained the attention of the wider tech-world, both usability and visual design have improved dramatically.

Good usability and visual design are necessary but insufficient for engagement

Straightforward techniques with strong empirical support

Reminders

Reminders (also referred to as prompts or triggers) involve prompting users to engage in target behaviors. Common examples are push notifications, emails, and text messages. They form a core component of the well-known Fogg Behavior Model. According to the model, three elements must converge for behavior change to occur: motivation, ability, and a prompt. That is, even if a person has sufficient motivation and ability, they will also need to be prompted to engage in the target behavior. For this reason, incorporating reminders is often considered a low hanging fruit of behavior change.

Reminders are a low hanging fruit of behavior change.

Studies have generally found that notifications positively impact engagement and efficacy. A 2009 study found that notifications improved efficacy in the majority (57%) of interventions reviewed. A more recent, 2018, study found that notifications increased engagement in the 24 hours following the notification by a statistically significant but relatively small 3.9%. Another 2018 study found that calls-to-action (of which reminders are a type) were among the factors most highly correlated with increased engagement. Data from our own clinical trials has shown substantial increases in engagement after users receive notifications.

Care must be taken, however, to not overuse notifications. Smartphone users receive upwards of 50 push notifications a day, and sending too many notifications can drive users away. The question of when and how to send reminders is a complex topic, and a whole field of ‘interruptibility research’ has emerged to explore, among other things, the optimal time and method to deliver notifications. Limbix users have been surprisingly responsive to notifications, and have consistently asked for more rather than fewer. The key principle in our experience is to ensure that the notifications are relevant and personalized. As with all engagement techniques, there are many factors that influence success. User testing and codesign are the best ways to fine-tune implementation, something we will write more about in a future article.

Goal setting

Goal setting is one of the most commonly used behavior change techniques in both face-to-face and digital health interventions. A 2017 review found that goal setting had a statistically significant but small additive effect on behavior change. A 2018 review found that goal setting was among the features most highly correlated with engagement. As with many of the techniques mentioned in this article, goal setting is thought to be more effective when paired with other behavior change techniques (e.g self-monitoring or accountability).

As with many of the techniques mentioned in this article, goal setting is thought to be more effective when paired with other behavior change techniques

Goal setting increases engagement by improving motivation. Specifically, it motivates users to try harder at intervention tasks, for longer periods of time, and with less distraction. According to goal setting theory, goal setting is likely to be more effective in improving engagement when clear instruction is given on the goal-setting process, goals are chosen by the user, and goals are appropriately matched to the user’s abilities. While professional coaches can be used to guide the goal-setting process, standalone interventions can leverage algorithms to automate the process of tailoring goals to users' interests and abilities.

Self-monitoring

Self-monitoring is one of the most commonly used behavior change techniques in digital health interventions. It is based on the premise that when users monitor a behavior, they are more likely to make positive changes to that behavior. Or in other words, we ‘manage what we measure’. Self-monitoring is often used in conjunction with goal setting, whereby the user monitors their behavior relative to a goal. Many wearable devices (e.g., Fitbit or Apple Watch) and nutrition apps (e.g. MyFitnessPal) are centered around the combined behavior change techniques of self-monitoring and goal setting.

A recent study found that self-monitoring was significantly associated with increased engagement. Another study found that ‘ongoing feedback’, which is similar to self-monitoring, had a small, positive correlation with engagement.

Interactivity

Interactivity refers to the two-way flow of information between the user and the digital intervention, and is considered a core component of engagement. There is a range of ways that interactivity can be incorporated into interventions, from moderate effort strategies like sliders, drags, and mouseovers, to high-effort features like algorithmic branching, conversational interfaces, and immersive game-design. A recent study found that interactive content was both more engaging and more persuasive than non-interactive content. Increased persuasiveness is particularly important in digital health interventions, where content is delivered in order to change attitudes and behaviors. A 2018 review found a small, positive association between interactivity and engagement.

Narrative

Narratives improve engagement by sparking curiosity and making content more memorable and relatable. The engaging power of narrative is not surprising, given that humans have a well-established propensity to make sense of the world through storytelling. As Yuval Harari writes in his best-selling book Sapiens:

“Storytelling is our specialty. It’s the basis for everything we do as a species.”

Many of the most successful video games incorporate narratives as a core component of their experience. Gamification has become an increasingly common tool to improve engagement, and studies have found that narratives are an important driver of engagement within gamified digital health interventions. Many of our users have commented that our game-like features (including the use of narrative) were among their favorite features of the program.

Accountability

Accountability occurs when a person is called to ‘account’ for their behavior. To be effective, three elements must co-occur: A person must set clear goals, they must monitor their behavior relative to their goals, and they must be called upon to justify their actions if they do not adhere to their goals. Great care must be taken in the third step — justifying non-adherence — in order to avoid inadvertently discouraging users and thereby decreasing engagement. To address this risk, many digital interventions incorporate human coaches that help clarify goals, monitor progress, and reflect on performance. Standalone interventions (those without human support) can also facilitate accountability through self-guided goal setting and monitoring, and by automatically alerting users when deviations occur. The use of an empathetic and encouraging tone, and reiterating the ‘why’ behind the desired behavior, is particularly important for maintaining users' motivation through inevitable setbacks.

Prescription digital therapeutics provide another avenue for accountability in standalone interventions. In the prescription digital therapeutic model, clinicians prescribe a standalone intervention and then meet with the patient at regular intervals for follow up appointments (similar to when a medication is prescribed). Patients’ expectations that they will need to justify non-adherence to their health care provider can provide an accountability boost. In support of this, a recent study found that clinic-referred patients showed 3x greater intervention adherence than patients that were self-referred to the intervention.

Personalization

Personalization (often referred to as tailoring) involves customizing digital interventions to users’ personalities, attitudes, knowledge, goals, and other characteristics. Personalization enhances engagement by increasing the perceived personal relevance of intervention content. Personalization can take many forms, from low effort features like including a user’s name, to more complex features like customized avatars or real-time adaptive content.

A 2013 review found that tailored web-based health interventions led to significantly greater improvement in health outcomes as compared to non-tailored interventions. The authors suggested that the improved health outcomes were likely due to better engagement in the tailored interventions.

One reason why human-delivered interventions may be more engaging than standalone digital interventions is that humans can better personalize the interventions to the needs of patients. However, as technology improves, personalization will too. Just as powerful algorithms in many consumer products (e.g. Facebook or Tiktok) can provide highly impactful personalization at scale, we envision a future where standalone digital health interventions are also highly personalized. Many interventions also leverage algorithms to assist professional supporters to better and more efficiently personalize interventions. We expect ‘AI-augmented’ interventions to play an increasingly important role in scaling healthcare access.

Credibility and Treatment Expectancy

An intervention is credible when users believe that the developers had the necessary expertise to create an effective program. Closely related is the concept of treatment expectancy, which refers to whether a user believes that the intervention will be effective. Researchers generally view these as separate but closely related constructs, with credibility referring to a more logical thought process and treatment expectancy refers to a more emotional process, similar to that of hope or faith.

There are many ways to establish credibility and treatment expectancy, including empirical validation (e.g. randomized clinical trials), regulatory approval (e.g. FDA clearance), development by clinical experts, co-designing with the target population, and having the intervention be recommended or prescribed by a clinician. Good visual design and usability can also positively impact perceived credibility and treatment expectancy.

The relationships between credibility, treatment expectancy, and engagement are intuitive. When users perceive a treatment as credible or expect the treatment to work, they will be more likely to invest time and effort into completing the intervention. This relationship is supported in the literature. A recent review similarly found that credibility and treatment expectancy were among the most important predictors of increased adherence in digital mental health interventions.

In line with this, researchers are exploring the emerging concept of the digital placebo effect. The placebo effect has historically been viewed as something simply to control for in randomized controlled trials. However, treatment expectancy (an active ingredient in the placebo effect) has a very real, positive impact on engagement and clinical outcomes. Digital health interventions present an exciting opportunity to utilize adaptive tailoring to optimize messaging based on users' attitudes and personalities, and thereby produce greater treatment expectancy and better outcomes. While developers should, of course, be required to prove that their digital health interventions have clinical effects beyond that of a placebo, the power of positive treatment expectancy can be leveraged to achieve an even greater clinical effect than would be achievable with clinically active ingredients alone.

Complex or experimental techniques

The following techniques are either well-established but complex (requiring careful implementation), or they are promising but still experimental.

User Control

User control refers to how much autonomy and choice users have when navigating interventions. The effect of user control on engagement is complex and has been theorized to both increase and decrease engagement.

On the one hand, user control can increase engagement by facilitating a greater sense of autonomy. According to self-determination theory, the human need for autonomy is a primary driver of motivation. For this reason, academics have predicted that greater user control and autonomy should lead to greater engagement.

On the other hand, several studies found that increased user control actually decreases engagement. A 2013 study found that users spent significantly more time engaging with an intervention that had a prescriptive, linear path, than with a version where all content was available at once. Another study found similar results, with linear navigation leading to greater consumption and retention of intervention content. This inverse relationship between user control and engagement brings to mind a famous study, titled “When choice is demotivating”, which found that consumers are less likely to purchase products when they are presented with a greater number of options. Consumers can become overwhelmed by too much choice and end up not choosing anything at all!

This may be an example of people’s preferences deviating from their behaviors. A 2012 study found that while users preferred having more control over their navigation through the content, they actually ended up engaging with the intervention less, and retaining less information than when they were given less control.

There are two guiding principles to draw from this research. The first is that while some control is good, too much control has adverse effects. Great care must be taken to identify the level of control that maximizes engagement. Second, when it comes to the complex realm of digital health interventions, it is often better to give users the feeling of control, while closely guiding them towards an optimal path.

It is often better to give users the feeling of control, while closely guiding them towards an optimal path.

Reward systems

Since the early days of behaviorism, researchers have established rewards as one of the most effective ways to influence behavior. Rewards can be broadly divided into two types: extrinsic and intrinsic. Intrinsic rewards relate to the positive consequences of doing something that is inherently enjoyable (e.g., feeling a state of flow), whereas extrinsic rewards refer to external incentives for engaging in a behavior (e.g., money or praise).

Reward systems within digital health interventions generally incentivize target behaviors by providing extrinsic rewards, such as badges, points, or level progression. Rewards can be provided for the target behavior itself (e.g. going for a run), effort towards the target behavior (e.g. scheduling a run), or approximations to the target behavior (e.g going for a walk)

A recent study found a small positive correlation between extrinsic rewards and engagement. Rewards are often grouped into the larger category of gamification elements, which are designed to provide extrinsic motivation to engage with the intervention. While there is general consensus that gamification elements improve engagement in digital health interventions, there are very few studies examining the precise impact of rewards by comparing the same intervention with and without rewards or other gamification elements.

While the effect of extrinsic rewards on behavior change is well established, rewards also run the risk of unintended negative effects on long term behavior change. Studies have shown that extrinsically rewarding someone for a behavior that they are intrinsically motivated to do decreases the likelihood that they will continue the behavior once the reward is no longer provided.

Although rewards are clearly a powerful force to influence behavior, it remains unclear how best to implement rewards in digital health interventions so as to avoid unintended consequences. It is for this reason that we consider reward systems to be a complex and high effort engagement technique

Social support

Social support features motivate users by leveraging social influence. The Persuasive System Design Model outlines 7 different types of social support, including social learning, competition, and cooperation, among others.

Social support features should not follow a one-size-fits-all approach, and developers must take care to identify the appropriate type of social support for each intervention. For example, encouraging users to share intervention accomplishments with their real-world peers might be appropriate for fitness interventions. Such an approach may be less effective for mental health interventions because participants may be less willing to share accomplishments due to significant stigma. Instead, facilitating the giving and receiving of support among intervention participants could be more effective. Moreover, certain social support techniques are likely to be more effective for certain people based on their personality type and other attributes. This is another example of where multiple engagement techniques (personalization and social support) can be combined to yield greater improvements to engagement.

A 2012 review of studies incorporating social support found that social support features had a positive, but statistically insignificant correlation with adherence. The study did not take into account the quality of implementation, and the authors suggested that social support likely does increase adherence when implemented well. This is an important insight that applies to all techniques mentioned in this article. There are substantial differences in the impact on engagement between techniques that are implemented well and those that are implemented poorly.

Professional support

Professional support (e.g. from coaches or clinicians) is arguably a component of social support but warrants its own section. Many digital health interventions incorporate professional support, but there is conflicting evidence about its impact on engagement. Several studies have indicated that professional support increases engagement. A 2019 review found that dropout from clinical trials was 12% for interventions that included professional support compared with almost 34% in stand-alone interventions. A 2014 review similarly found that professional support led to significantly higher intervention completion rates.

However, other reviews indicate that professional support does not significantly improve engagement. A 2020 study found that professional support did not increase adherence relative to automated notifications. A 2017 study similarly found that human support did not significantly impact intervention usage, and a 2016 review found that professional support did not significantly increase adherence. On the extreme end, a 2017 review found that digital health interventions for depression that included professional support had non-significant clinical effects, whereas interventions without professional support had significant effects.

So what do we make of all of this? As already mentioned, there are often substantial differences in the quality of implementation of a technique across studies, and this is further compounded by the wide range of coaching methodologies that are available. The need for human support likely also depends on the clinical severity, the target diagnosis, the type of treatment, and the individual characteristics of the patient. These factors could account for the conflicting findings. However, the impact of professional support on engagement is still likely smaller than previously thought.

It’s also important to take into account the cost and benefit of human support. Digital health interventions could enable a future where many treatments are infinitely scalable and can thereby address the formidable healthcare access issues throughout the world. However, this vision is more feasible for interventions that are comprised of standalone software. Human-in-the-loop features increase logistical complexity and cost, and constrain the ability of the intervention to scale.

We foresee standalone interventions playing an important role in increasing access to effective and affordable care.

It has also been suggested that as interventions and associated technology become more advanced, the need for human support could diminish. Perhaps we can draw a comparison to the self-driving car industry. Cars can already drive autonomously in certain circumstances, but human drivers are still needed in many situations. While we believe that professional support will always play an integral role in health interventions, we foresee standalone interventions playing an important role in increasing access to effective and affordable care.

Digital therapeutic alliance

Therapeutic alliance refers to the quality of the working relationship between a therapist and patient. Adherence is a major issue in face-to-face therapy (around 50% of patients drop out of treatment prematurely), and therapeutic alliance is consistently shown to be the single biggest predictor of adherence.

Given the importance of therapeutic alliance for engagement in face-to-face therapy, there is growing interest in how therapeutic alliance translates to digital interventions. This construct has been termed the digital therapeutic alliance. A 2018 survey among national mental health stakeholders in the UK identified digital therapeutic alliance as a top 10 research priority in digital mental health.

A recently established measure of digital therapeutic alliance includes 5 dimensions:

While digital therapeutic alliance is still a new and emerging concept, early data are promising. A 2018 study found that participants’ ratings of digital therapeutic alliance were positively associated with program log-ins, frequency of self-monitoring, and number of treatment modules completed. Given the strong theoretical rationale for the relationship between digital therapeutic alliance and engagement, we expect it to emerge as an impactful driver of engagement in digital health interventions.

Conclusion

In this article, we’ve outlined techniques that have shaped our thinking on how to build engaging digital health interventions. For those building interventions, we recommend starting with the foundations of usability and visual design. However, these are necessary but insufficient for engagement, and you should quickly move on to the straightforward and well-established techniques that form the backbone of engaging digital health interventions. Finally, consider integrating one or more of the more complex or experimental techniques. If you do, you will be well on your way to creating an engaging and impactful intervention.

We’d love to hear about what techniques have worked best for you. What insights have you gained that are not captured here?

We’d love to hear about what techniques have worked best for you. What insights have you gained that are not captured here? What would you like to hear more about from us? By sharing what’s worked, we hope to assist others that are tackling similar problems and thereby accelerate learning across the industry. By working together, we can build a world where all those in need have access to effective, affordable health care.

Further reading

The following are excellent reviews of engagement factors and behavior change techniques.

• Perksi and colleagues (2017). A comprehensive review of factors shown or hypothesized to influence engagement in digital health interventions.
• Baumel & Kane (2018). An analysis of the impact of engagement features on real-world engagement.
• Michie and Colleagues (2013). A taxonomy of behavior change techniques
• Oinas-Kukkonen and colleagues (2009). Another taxonomy and process model of behavior change techniques.

Acknowledgments

Thank you to our Head of Content Elise Vierra, Product Designer Xin Koepsell, UX Researcher Stella Kim, and Director of Research Jessica Lake for contributing to this article. Thank you also to Jim Liu and Shikha Nalla for insightful feedback.

Engagement techniques for digital health interventions was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

As an undergraduate student and field hockey player at Stanford University, academic and athletic success felt like forces that dominated Elise Ogle’s life. Those pressures took a toll, one that was difficult to recognize at the time: “You can see the first-hand effects of that [pressure] on mental health when you don’t get the support you need, or just understand enough about your own mental health to be able to recognize when you’re slipping,” she says.

The 28-year-old San Jose resident, now Head of Content at Limbix, says she struggled with depression throughout college and was hospitalized. Not long after, her close friend and roommate committed suicide. “That was a really big turning point in my own mental health journey,” Elise says. It became more apparent that so many people are experiencing mental health challenges. Simply knowing where to start finding help — not to mention navigating a labyrinth of insurance logistics and red tape — could feel daunting for anyone, let alone those in crisis.

Elise, who studied communication and psychology, began thinking differently about what brought her meaning in life. She also began working in a lab, helping researchers explore the effects of new media on psychology and behavior. She stayed at Stanford to pursue her master’s degree and work in the Virtual Human Interaction Lab, studying the effects of virtual experiences on empathy.

Though her roots were in research, Elise found herself drawn to the narrative aspect of her work: “creating these stories, telling these stories, and being able to see this thing in my imagination actually come to fruition,” as she puts it. But beyond advancing research, Elise wanted to apply technological innovations in real life.

So she searched for a company that could marry her academic interests and passion for mental health. She found that sweet spot in Limbix in 2018; today, Elise creates content for the digital prescription therapeutic, Spark. She also coaches field hockey for teenagers and college club teams, describing her mission on the field and at Limbix as two sides of the same coin: motivating teenagers to work hard, develop skills, and experience a sense of mastery.

We sat down with Elise to learn more about her approach to content creation and the steps required to bring clinically effective copy to a smartphone screen. Our conversation has been edited for length and clarity.

***

What do you see as the disconnect between the help that people need and their ability to get help?

The first challenge is access: being able to get help when you need it, and even the basic understanding of what the appropriate help is. There’s still a lot of stigma around mental health. Certainly when I was growing up, it was never something we learned about. Understanding more about your own mental health can help people understand when it’s time to access a different level of care, and even to know what to do to take that first step.

Another challenge is how to navigate the mental healthcare system. I looked for a therapist after quarantine started, and holy moly — it was really difficult, as an adult. I can only imagine how much harder it would be for someone who doesn’t have the support that I have.

Then I think there’s a gap in support as far as empathy goes. It’s hard to get to the point where you recognize that you need help, and even harder to convey that to someone. Whether you’re reaching out to a friend or family member, or your primary care provider, or to a therapist — making that first initial contact is really difficult when you’re experiencing emotional distress. It’s important that people are met with support and validation right away, and that their feelings are not dismissed. Everyone is living a different story and has a range of emotions as they seek care. It’s important to meet people where they are.

What has made you most energized to serve this population — teenagers — in particular?

Definitely my personal experiences with mental health and coaching. My husband is a full-time youth coach, and, observing his interactions with the kids he coaches, you can directly see the benefits of having someone in your life that gives you support. I’ve seen all types of kids, and different types of parents and support systems around each kid. With that high-level understanding of what some people have that other people don’t, I see how important it is to have someone step in when you need it. It’s energizing to be able to be that for someone, and to work on a technology that can hopefully make that accessible for many different types of people. It’s also energizing, in the day-to-day sense, to talk with teens through co-design or feedback sessions. They are the reason that we do what we do.

What’s the most challenging aspect of this work and the bridge that you’re trying to build?

There’s a lot of different stakeholders in the creation process of the prescription digital therapeutic, beyond just the people who do the writing, like myself. There’s product design and user interaction with that product. Other important pieces are safety and clinical efficacy — making sure that we create something engaging and that teens want to use, but also making sure everything in the app is evidence-based. Then there’s another element, running clinical trials, with specific considerations for clinical trials for a medical device.

It’s challenging to create content that meets so many different stakeholders’ expectations of what that content should look like. And to know that teenagers experiencing depression, a vulnerable population, are going to be interacting with our vision for the product. This is a relatively new space. It’s not like we could fit a cookie-cutter mold of a traditional company. But the fact that we’re pioneers is exciting.

How do you describe your purpose and role at Limbix, beyond your title? In laypeople’s terms, what are you really there to do?

My role at Limbix is to work with subject matter experts in adolescent psychology, take their research protocols and translate them into a mobile app that’s engaging for teens. I make sure that anything someone sees in the app is valid — that if you were going to face-to-face therapy, your therapist would be teaching you these principles. From a higher level, my role is to help teens feel empowered through the program to understand their depression and manage their symptoms.

What do you wake up every morning jazzed to do, when it comes to what you’re creating?

I’m excited to create a tool that I wish had existed for me when I was a teen. That’s my philosophy generally, when I’m coaching as well: I exist as a coach to teach things beyond field hockey — guidance that I wish I had learned and gotten when I was their age. With access to that kind of support, people can grow so much faster. They can fail, too, but in a safer way.

What does your day-to-day routine look like and what suite of tasks is typically on your docket?

That depends on where we are in our process. We have different phases of planning, building, and testing. Right now we’re in a building, or creating, phase. We have morning meetings to sync with members of our product team, iron out logistics and make sure we have the tools we need. Then it’s time to dive into the content creation process. I work closely with our product designer to build content, which means the copy, illustrations, and interactions that users will consume on the screen. We use software that allows us to collaborate on prototypes virtually.

To avoid long screens of text, we try to incorporate different ways to interact with the content so it’s more engaging. For example, there might be a task where the user learns about something, but receives reinforcement by dragging and dropping the right things in the right order. Another part of my job is conceptualizing illustrations and short animations used to augment the text. We want to use art as a tool to better teach something and recognize that people consume information differently.

To get a bit more granular, I’d love to understand the content creation process from A to Z. What are some of the key steps, checkpoints and milestones, from the conceptualization stage to polished copy that will appear in the app?

We start with looking at existing protocols for teaching behavioral activation — especially those geared toward adolescents, which have different nuances than adult protocols. Then we pull out the most important user needs for that protocol. Let’s say we were focusing on one element of activation, like problem-solving. I would focus on the protocols that have taught problem-solving, and from there, identify high-level points the content must include.

Some team members with PhDs serve as our internal experts, but we also work closely with psychologists at the National Mental Health Innovation Center. I’d talk to them about user needs that we’ve identified, our vision for the outcome and the outline of this particular part of the program. We then iterate on it until everyone feels like the content teaches the most important points to the user.

From there, I get into fleshing out the outline — actual words the user will see. I work with our product designer, who adds those interactive elements I talked about. We create individual tasks that will appear in the app — step one for problem solving, step two for problem solving, which make up the path the user goes through. From there, we keep iterating until we have a working prototype — a series of screens with either placeholder illustrations or ones I’ve mocked up.

We go back to our subject matter experts to get feedback on what we’ve created and to make sure everything is clinically valid. We also get their thoughts from a provider’s perspective, to see if they have different ideas for interactions that could take place that might achieve better aims.

How do you incorporate teenagers’ feedback into Spark? What role do they play in this process?

We have a couple of ways in which teens can give feedback and get involved in the creation process. One of those avenues is joining us for user research. We have a UX researcher who organizes teens that have experienced or are experiencing depression to share thoughts and feedback on upcoming content, designs or product features. We do general feedback sessions with concept testing.

There are also avenues where teens can participate in what are called co-design sessions, which includes more design-type activities. We talk about the goals of what we’re trying to accomplish, and teens help us conceptualize different interactive elements or content they feel would be relevant.

Then we also have teens who enroll in our clinical trials. If they’d like to, they can join us after for more feedback sessions to talk about their own specific experience using the app. They can discuss areas they thought could use improvement or that they really liked.

What makes you most hopeful about the work that you’re doing, and more broadly about the work that Limbix is doing?

It comes back to talking to teens and hearing their feedback, especially those participated in our clinical trials. What’s also really cool is working with such an interdisciplinary team. We have engineers, we have designers, we have researchers — all of these smart people who work really hard and are putting all of that energy and effort into building something that’s going to help people.

I find hope in the meaning behind what we’re doing — bringing it back to that user. I could have been that user, or my friend could have been that user. I’m always remembering our mission.

***

If you’re interested in learning more about Limbix and our ongoing trials, visit limbix.com.

From Conceptualization to Content: Q&A With Elise Ogle was originally published in Limbix Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.