Value as a construct for comparing psychotherapy with digital therapeutics

Jeb S. Brown, PhD, Center for Clinical Informatics

Edward R. Jones, PhD, Learn to Live, Inc.

Ashley Simon, MA, Center for Clinical Informatics

Published by the Society for the Advancement of Psychotherapy

Introduction

Estimating the value of medical and behavioral services has received an increasing amount of attention in recent decades (Happell, 2008; Rocco, 2019; Porter & Teisberg, 2004; Porter & Teisberg, 2006; Porter, 2010, Teisberg et al., 2020). Porter and Teisberg (2006) compared methods for evaluating value in their book Redefining Health Care: Creating Value-Based Competition on Results, and more recently Porter (2010) framed the issue in the New England Journal of Medicine (p. 1).

Value — neither an abstract ideal nor a code word for cost reduction — should deepen the framework for performance improvement in health care. Rigorous, disciplined measurement and improvement of value is the best way to drive system progress. Yet value in health care remains largely unmeasured and misunderstood.

Value is an effort to combine the quality of care and its cost quantitatively. In simple terms, value=magnitude of improvement/cost of care. This study aims to clarify the value concept in behavioral healthcare by defining a Value Index and applying it to a comparison of psychotherapy and digital therapeutics.

The digital platform studied in this article consists of online self-guided cognitive behavioral therapy (digital CBT) modules for various behavioral health conditions. It is offered by Learn to Live, Inc., and the authors have previously studied other aspects of the company's services (Brown et al., 2020; Brown & Jones, 2020; Brown & Jones, 2022a; Brown & Jones, 2022b). In addition to data from the Learn to Live platform, these studies utilize data from a large database of measured outcomes in routine outpatient psychotherapy using well-validated questionnaires (Brown et al., 2015) contained in the ACORN Collaboration data warehouse.

Creating a Value Index

Jones and Brown (2023) outlined a calculation for a Value Index that aligns with the medical literature. A Value Index simplifies comparisons that are otherwise complicated by using standardized components. For example, we have already defined value as the ratio of outcome and cost. In that case, we can use units of effect size (a statistical measure of clinical improvement) as the standardized measure of outcome, and dollar units can be used as the standardized measure of cost. The Value Index then needs a reference point for cost. If this is $1000, the Value Index calculates the quantity of effect size units purchased for every $1000 in healthcare costs. Payers and health systems usually understand only the quantity of services purchased. However, the Value Index helps them understand the amount of clinical improvement they are purchasing for a given population (Jones & Brown, 2023).

Benchmarking Results and Estimating Costs

This study examines two real-world datasets, one for routine psychotherapy in the community and the other for the Learn to Live digital CBT platform users. However, another measure is available for benchmarking results. Researchers have studied the results of over four decades of clinical trials of various forms of psychotherapy, and this work establishes an average effect size of 0.8 for clinical improvement from therapy (Wampold & Imel, 2015).

Estimates of costs for psychotherapy can vary considerably with the type of insurance coverage or government-funded care, region of the country, and urban versus rural settings. For these analyses, we estimate that the average cost to an employer is approximately $75 per session, though it may be higher for some employers. This estimate is admittedly only an educated guess. However, it is based on three decades of experience by the authors in evaluating costs and revenues for large clinics and employers, including direct access to insurance companies' cost data.

Estimating the cost for the Learn to Live digital platform users is complicated because the program is often funded on population size rather than by users (i.e., cost per member per month). Therefore, the actual utilization of the program will vary by employer. For this analysis, we estimated that the cost per employee completing lessons is approximately $30. The Value Index model allows substituting other cost estimates that more accurately reflect an employer's experience to determine if the comparison between interventions changes substantially.

Description of the Sample

The digital therapeutics sample included users enrolled in the Learn to Live program between June 1, 2021, and September 30, 2022. The selection criteria for the sample included users with a first assessment score in the clinical range and at least two completed lessons and accompanying assessments. The total number of lessons available is seven. In addition, at least two assessments are necessary to calculate effect size scores. A total of 6,672 users met these criteria.

As noted previously, the sample receiving outpatient psychotherapy was drawn from the ACORN Collaboration data warehouse, as was the case in previous articles evaluating Learn to Live results (Brown et al., 2020; Brown & Jones, 2020; Brown & Jones, 2022a; Brown & Jones, 2022b). The selection criteria for the ACORN sample included adults only who had completed between two and twenty sessions of outpatient psychotherapy. These criteria resulted in a sample of 154,421 adults. Within this sample, 79% completed therapy by the seventh session, permitting ready comparison to the Learn to Live sample that reaches a maximum of seven online lessons.

Results

The overall results for both digital CBT and traditional psychotherapy were quite good. The digital therapeutics sample had an effect size of .58, with an average of 3.6 lessons completed. Those who completed all seven lessons (17% of the sample) had an effect size of .96, significantly exceeding benchmarks obtained from psychotherapy's clinical trials. By way of comparison, for the outpatient psychotherapy sample if restricted to only those patients completing treatment within seven sessions (79% of the entire sample), the mean effect size was .74, with an average of 3.4 sessions. For most patients, improvement in the first five to seven sessions is relatively rapid, after which patients begin to leave treatment. The following table presents these results.

While the digital CBT sample showed less change in early lessons, the digital users caught up to the outpatient sample by lesson/session five. They were equivalent by the seventh intervention, with an effect size of 0.96 for digital users compared to .94 for the therapy sample. As presented in the chart above, the Value Index for digital therapeutics is six times higher than for psychotherapy. Given the early stage of such analyses, there are no benchmarks against which these values can be compared. However, the metric is a simple way to call attention to the question of how much clinical improvement is connected to the services being purchased.

Discussion and Implications

This comparison between psychotherapy and digital therapeutics highlights a new metric, the Value Index. It does not presume these are equivalent services needing to be differentiated in this way. Apart from differences in content and delivery method, the consumers choosing one or the other are likely to differ in motivations, needs, and goals for treatment. Some consumers may benefit from both; others may benefit from one more than the other. In evaluating services, purchasers should review the percentage of a population choosing any service and why they prefer one. This comparison is intended to give the purchaser another way to compare services.

One implication for purchasers is that offering both options might provide the best results at the greatest value. For example, easy access to digital services may well meet many employees' needs, yet some will need access to more intensive in-person treatment. In addition, integrating the two processes might provide even more significant benefits by reducing the number of psychotherapy sessions needed to achieve effect sizes of .8 or larger reliably. The value metric might be integrated into the care management process as well. For example, cases showing low value could be targeted for intervention. Someone with an effect size of only 0.2 after five digital lessons might be flagged for in-person services. An effect size of 0.2 after 12 psychotherapy sessions might trigger a recommendation for adding services such as digital therapeutics.

There is a need for more research, but the best evidence to date suggests that digital therapeutics are a cost-effective option for employers and health plans to consider. Their relative value can be quantified based on outcomes and cost. Furthermore, value is an excellent way to evaluate all health services, and we should endeavor to achieve the most significant health outcomes for the dollars expended.

References

Brown, J. & Jones, E. R. (2020). Impact of coaching on rates of utilization and clinical change for digital self-care modules based on cognitive behavioral therapy. Society for the Advancement of Psychotherapy. Retrieved 4/10/23, from http://www.societyforpsychotherapy.org/impact-of-coaching-on-rates-of-utilization-and-clinical-change-for-digital-self-care-modules-based-on-cognitive-behavioral-therapy

Brown, J., & Jones, E. R. (2022a). Improving results for digital therapeutics. Society for the Advancement of Psychotherapy. Retrieved 4/10/23, from  http://www.societyforpsychotherapy.org/improving-results-for-digital-therapeutics

Brown, J. & Jones, E. R. (2022b). Improving results for digital therapeutics with social support. Society for the Advancement of Psychotherapy. Retrieved 4/10/23, from http://www.societyforpsychotherapy.org/improving-results-for-digital-therapeutics-with-social-support

Brown, J. & Jones, E. R. (2023). Calculating value as the primary economic measure for our field. Behavioral Healthcare Executive. Retrieved 4/10/23, fromhttps://www.hmpgloballearningnetwork.com/site/bhe/perspectives/calculating-value-primary-economic-measure-our-field

Brown, J., Jones, E. R., & Cazauvieilh C. (2020, May). Effectiveness for online cognitive behavioral therapy versus outpatient treatment. Society for the Advancement of Psychotherapy. Retrieved 4/10/23, from https://societyforpsychotherapy.org/effectiveness-for-online-cognitive-behavioral-therapy-versus-outpatient-treatment

Brown, J., Simon, A. S., Cameron, J. & Minami, T. (2015). A collaborative outcome resource network (ACORN): Tools for increasing the Value of psychotherapy. Psychotherapy, 52, 412–421. doi:10.1037/pst0000033

Happell, B. (2008). Meaningful information or a bureaucratic exercise? Exploring the Value of routine outcome measurement in mental health. Issues in Mental Health Nursing, 29(10), 1098-1114. doi: 10.1080/01612840802319852.

Porter, M. E. & Teisberg, E. O. (2004). Redefining competition in healthcare. Harvard Business Review. Retrieved 4/10/23, from https://hbr.org/2004/06/redefining-competition-in-health-care

Porter, M. E. & Teisberg, E. O. (2006). Redefining health care: Creating value-based competition on results. Harvard Business Review Press.

Porter, M. E. (2010). What is Value in healthcare? New England Journal of Medicine, 363(26) 2477–81. doi: 10.1056/NEJMp1011024

Rocco, R. P. (2019). High-value mental health care and the person in the room. Psychiatric Services, 71(2) 110-111. https://doi.org/10.1176/appi.ps.201900357

Teisberg, E. O., Wallace, S. & O’Hara, S. (2020). Defining and implementing value-based health care: A strategic framework. Academic Medicine, 95(5). p. 682–685. https://doi.org/10.1097/ACM.0000000000003122

Wampold, B. E. & Imel, Z. E. (2015). The Great Psychotherapy Debate: The evidence for what makes psychotherapy work. Routledge.