1. Background and Summary

In order to do as much good as possible, we need to compare how much good different things do in a single ‘currency’. At the Happier Lives Institute (HLI), we believe the best approach is to measure the effects of different interventions in terms of ‘units’ of subjective wellbeing (e.g. self-reports of happiness and life satisfaction). 

In this post, we discuss our new research comparing the cost-effectiveness of psychotherapy to cash transfers. To see the advantage of making comparisons in terms of subjective wellbeing, it will help to flag some alternative methods. We could assess the effect they have on wealth, but this would fail to capture the benefits of psychotherapy. It’s implausible to think that treating depression is only good insofar as it helps you to earn more. We could assess their effects using standard measures of health, such as a Disability-Adjusted Life-Year (DALY), but it’s similarly mistaken to think that alleviating extreme poverty is only good insofar as it helps you to be healthier. We could make some arbitrary assumptions about how much a given change in income and DALYs each contribute to wellbeing; this would allow us to ‘trade’ between them. But this would just be a  guess and could be badly wrong. If we measure the effects on subjective wellbeing, how individuals feel and think about their lives, we can provide an evidence-based comparison in units that more fully capture what we think really matters. 

Efforts to work out the global priorities for improving subjective wellbeing are relatively new. Earlier work conducted by HLI’s Director, Michael Plant, has indicated that this approach might reveal different priorities, with mental health standing out as one area that is crucial for subjective wellbeing. Plant’s (2018, 2019 ch. 7) prior back-of-the-envelope calculations indicated that StrongMinds, a mental health charity which treats women with depression in Africa, could be as cost-effective as GiveWell’s top charity recommendations. 

These initial findings motivated us to do a much more rigorous analysis of the same interventions in terms of subjective wellbeing, so we undertook meta-analyses in each case. These aimed to address three questions:

1. Is assessing cost-effectiveness in terms of subjective wellbeing feasible: are there enough data that we can make these sorts of comparisons without making major assumptions to fill in the blanks?

 

2. Is this approach worthwhile: does it indicate new or different priorities?

 

3. Does this specific comparison between cash transfers and psychotherapy indicate that donors and decision-makers should change the way they allocate their resources, assuming they want to do the most good?

Our research focused specifically on studies in low- and middle-income countries (LMICs). Our meta-analyses consist of 45 studies (total participants=116,999) for cash transfers and 39 studies (total participants=29,643) for psychotherapy. We assessed their cost-effectiveness using measures of subjective wellbeing (SWB) and mental health (MHa), which we combined (see section 2.2 for more details). 

We estimate that the average psychotherapy intervention in our dataset would be 12 times (95% CI: 4, 27) more cost-effective than the average monthly cash transfer. We used this wider evidence base to estimate the cost-effectiveness of two charities that are highly effective at implementing each type of intervention: Give Directly (which provides $1,000 lump-sum cash transfers) and StrongMinds (which provides psychotherapy). When we repeated the analysis for these specific charities, adjusting for how they differed to the average intervention of their type, we found that StrongMinds is 12 times (95% CI: 4, 24) more cost-effective than GiveDirectly. 

A few aspects of this research are worth highlighting. First, this is an empirically substantial, meta-analytic approach compared to the back-of-the-envelope calculation in Plant (2019), therefore we can be more confident in our pooled estimates. Second, meta-analyses only tend to report the initial effects of the intervention. We estimated the effects over time and the average costs which means we can calculate cost-effectiveness (which is not possible just with initial effects). Third, we used Monte Carlo simulations to generate our estimates. Rather than using a single number for each part of the cost-effectiveness analysis (e.g. a single number for cost) Monte Carlo simulations involve specifying a range of possible values for each input, then recalculating the results over and over, using a randomly drawn number for each input. This allows us to better account for uncertainty and to assign confidence intervals to our final cost-effectiveness estimate.

What are the main conclusions to draw from this new analysis? Let’s return to the three stated aims in turn.

First, such an analysis was feasible. There was more than enough data for a meta-analysis, although we did have to pool ‘classic’ subjective wellbeing measures (happiness and life satisfaction) with mental health measures (see section 2.2). Aggregating in this way has some precedent (Banerjee et al., 2020; Egger et al., 2020; Haushofer et al., 2020; Luhmann et al., 2012).

Second, performing cost-effectiveness analysis in terms of subjective wellbeing does seem worthwhile. We found that StrongMinds was 12 times more cost-effective than GiveDirectly. This 12x multiple puts StrongMinds roughly on a par with GiveWell’s top-rated life-improving charities – these mostly focus on deworming. Although GiveWell recommends GiveDirectly, it estimates it to be 10-20x less cost-effective than it’s top deworming charities, in terms of years of doubled consumption. Hence, if we take these multiples at face value, StrongMinds is in the same ballpark as the top-rated life-improving interventions. 

The third aim of this research was to determine if donors should allocate their resources differently. The picture here is more complicated. As noted, taking the multiples at face value, StrongMinds is on a par with GiveWell’s top life-improving interventions, it is not more cost-effective. However, we have not yet looked deeply into deworming ourselves and we would not be surprised if doing so indicated substantially different conclusions.

We suspect that deworming will turn out to be less effective than GiveWell currently estimates (relative to cash transfers and in terms of subjective wellbeing). In GiveWell’s model of the effect of deworming, its benefits come almost entirely from improved educational attendance which, in turn, causes a small annual increase in income for many years. Hence, although it is a health intervention, the effect comes through reducing poverty. One issue is that a small, annual income increase may have less effect on wellbeing compared to a single, potentially life-changing lump sum, even if the total increase in lifetime income is the same (the $1,000 lump sum given by GiveDirectly is equivalent to a year of household income). We have raised this concern with GiveWell and you can read the relevant section of our conversation notes here. In our meta-analysis of cash transfers, we find that monthly transfers are about 2.5 times less cost-effective than large lump sums, which suggests that a non-trivial downward adjustment to deworming (relative to cash transfers) may be appropriate. However, it’s also possible we will conclude that deworming has a more substantial and positive impact on health than GiveWell currently models, which would raise its cost-effectiveness.

For any donors who need to make an immediate decision, our weakly-held view is that StrongMinds is more cost-effective than any of GiveWells’ top life-improving charities. We plan to conduct further analysis of deworming over the next few months and expect to be able to make a more confident claim then.

Given all this, we believe it is both feasible and worthwhile to investigate more interventions in terms of subjective wellbeing. As well as deworming, we have several interventions that we’re excited to look at. Based on our shallow analyses so far, we think there’s a good chance these interventions will be more cost-effective than psychotherapy. They run the gamut of ‘micro’ interventions, such as providing clean water, cement flooring, and cataract surgery, to ‘macro’ interventions, such as advocating to integrate wellbeing metrics in public policy to improve institutional decision-making. 

The following sections of this post go on to summarise our three in-depth cost-effectiveness analyses: one on cash transfers (which includes our analysis of GiveDirectly); one on psychotherapy in general; and one on StrongMinds specifically. Our results are summarized below in Figure 1, where the lines with a steeper slope reflect a higher cost-effectiveness in terms of MHa and SWB improvements. Each point is a single run of a Monte Carlo simulation for the intervention, mapping the uncertainty around our estimates of the effects and the costs. Although our estimates for psychotherapy and StrongMinds are more uncertain than cash transfers, we still estimate that they are more cost-effective at improving subjective wellbeing. 

Figure 1: Cost-effectiveness of psychotherapy compared to cash transfers

Note: We assume that GiveDirectly CTs are as effective as other lump-sum CTs, but we think GiveDirectly probably has lower operating costs. This is why we do not display them separately. 

2. Cost-effectiveness analyses (CEAs)

2.1 Cash transfers (see full analysis)

Cash transfers (CTs) are direct payments made to people living in poverty. They have been extensively studied and implemented in low- and middle-income countries (LMICs) and offer a simple and scalable way to reach people suffering from extreme financial hardship. 

Our cash transfers cost-effectiveness analysis (CEA) determines the cost-effectiveness of lump-sum CTs and monthly CTs in LMICs using standard deviation (SD) changes in subjective wellbeing (SWB) and affective mental health (MHa) . We use ‘affective mental health’ to refer to the class of mental health disorders and measures (e.g. PHQ9, CESD20, GHQ12, and GAD7) that relate to lower levels of affect or mood (depression, distress, and anxiety). Measures of affective mental health differ from measures of SWB because they also include questions that ask about how well someone is functioning such as the quality of someone’s appetite, sleep quality, and concentration. These factors influence, but do not constitute, someone’s mood. This analysis provides us with a well-evidenced benchmark that we can use to compare a wide range of interventions.

Much of our analysis extends a meta-analysis of the effect of cash transfers on SWB and MHa in LMICs by McGuire, Kaiser, and Bach-Mortensen (2020). That paper is the result of a collaboration between HLI researchers and academics at the University of Oxford and has been conditionally accepted for publication by Nature Human Behaviour. We think this meta-analysis summarizes the entire extant literature on how cash transfers affect SWB. Our CEA uses the same data (45 studies, 114,274 individuals in LMICs) and shares some analysis on the effectiveness of cash transfers. It expands on the meta-analysis by adding cost information and estimating the total benefit we expect a recipient of a cash transfer to accrue. Please refer to the meta-analysis for more background on cash transfers and a more thorough description of the evidence base. 

Our models for estimating the effectiveness of lump-sum CTs and monthly CTs are presented in Figure 2 below. The chart on the left shows the relationship between time and the effect on SWB and MHa. As we expect, the effects decay over time. The chart on the right shows the relationship between the value of a CT and the effect on SWB and MHa. Unsurprisingly, the effect is greater for larger CTs (although the data is quite noisy). We haven’t differentiated between the effects on SWB and MHa. Our sample of studies was too underpowered to break down the analyses by both delivery mechanism and measure type (SWB or MHa). But the point estimates suggested that the differences between the measure types were small. For instance, using only SWB measures leads to a 2% and 13% increase in the total effect for monthly and lump CTs. These seemed like sufficient reasons to keep SWB and MHa pooled for our main analysis. 

Figure 2: The effect of cash transfers in relation to time and value

Note: These results only display the bivariate relationship between value, time, and the effect. 

We present the estimated cost-effectiveness of CTs in Table 1, alongside the per-person effects and per-transfer costs. To get the confidence intervals for the effect and cost we ran Monte Carlo simulations in R where each parameter was drawn from a normal distribution based on its standard error (for regression coefficients) or standard deviations for the cost. 

Table 1: Estimated cost-effectiveness of lump-sum CTs and monthly CTs

Note: Below the estimate is its 95% confidence interval which was calculated by inputting the regression and cost results into a Monte Carlo simulation.  

Notably, the estimated total effect of a $1,000 lump-sum CT provided by GiveDirectly is about twice as large as the effect of $1,000 transferred in monthly increments by (mostly) governments. This seems like a suspiciously large difference at first, since we do not think a priori that giving people of similar means the same sum should have dramatically different effects. Some possible explanations we have considered include:

• Lump-sum CTs can be invested more profitably, leading to higher total consumption and benefit. However, monthly CTs could lead to smoother consumption and that could benefit the recipient. 
• Governments in LMICs suffer from corruption, while GiveDirectly is a respected charity known for the strength of its organisation.
• Many government CTs require individuals to travel in-person to a collection point while GiveDirectly CTs are transferred automatically using the mobile banking platform M-Pesa.
• GiveDirectly also provides cell phones and a bank account to recipients without it. Although they take the value of the phone out of the transfer, this could plausibly provide an additional benefit to the recipient. 

There are three main limitations to our analysis.

Firstly, the recipient is plausibly not the only person impacted by a cash transfer. They can share it with their partner, children, and even friends or neighbours. Such sharing should benefit non-recipients’ wellbeing. However, it’s also possible that any benefit that non-recipients receive could be offset by envy of their neighbour’s good fortune. There appears to be no evidence of significant negative within-village spillover effects, but there is some evidence for positive within-household and across-village spillover effects. We have not included these spillover effects in our main analysis because of the large uncertainty about the relative magnitude of spillovers across interventions and the slim evidence available to estimate the household spillover effects.

Secondly, we have not extensively reviewed the evidence that explores the mechanisms through which CTs improve the SWB of their recipients. This limits our ability to estimate the effects of income gains through other means, such as deworming. For instance, we have so far found little evidence regarding how much of the benefit of a CT is contingent on recipients’ relative versus absolute improvement in material circumstances. Is the benefit due to recipients making beneficial comparisons to those who didn’t receive a CT or is it because of the absolute change in the recipients’ living standards as a result of higher consumption? If the channel for CTs to improve SWB and MHa primarily runs through social comparison, then that would suggest that the effects would be smaller for an intervention that increases everyone’s income in an area.

Thirdly, we have sparse data on the long term effects of CTs on SWB. There is only one study that follows up with its recipients after five years. Having long-term follow-ups of CTs is important for understanding the persistence of CTs benefit, and thus the total benefit they provide to their recipients. 

2.2 Psychotherapy (see full analysis)

Having determined the cost-effectiveness of cash transfers in terms of subjective wellbeing, we can now conduct a direct comparison with the cost-effectiveness of psychotherapy.

Psychotherapies vary considerably in the strategies they employ to improve mental health, but some common types of psychotherapy are cognitive behavioural therapy (CBT) and interpersonal therapy (IPT). Our analysis does not focus on a particular form of psychotherapy. Previous meta-analyses find mixed evidence supporting the superiority of any one form of psychotherapy for treating depression (Cuijpers et al., 2019). 

Instead, we focus our analysis on the average intervention-level cost-effectiveness of any form of face-to-face psychotherapy delivered to groups or by non-specialists deployed in LMICs. As before, we seek to measure the effect as the benefit they provide to subjective wellbeing (SWB) and affective mental health (MHa). 

We extracted data from 39 studies that appeared to be delivered by non-specialists and/or to groups from five meta-analytic sources, and any additional studies we found in our search for the costs of psychotherapy. These studies are not exhaustive. We stopped collecting new studies due to time constraints and our estimation of diminishing returns. We aimed to include all RCTs of psychotherapy with outcome measures of SWB or MHa but only found studies with measures of MHa. 

In Table 2, we display our estimated post-treatment effects and how long they last for the average psychotherapy intervention in our sample. The post-treatment effects are estimated to be between 0.342 and 0.611 SDs of MHa. After running further regressions, we found evidence that group psychotherapy is more effective than psychotherapy delivered to individuals which is in line with other meta-analyses (Barkowski et al., 2020; Cuijpers et al., 2019). One explanation for the superiority is that the peer relationships formed in a group provide an additional source of value beyond the patient-therapist relationship. 

Table 2: Post-treatment effect and decay through time

Before we compare the total effect of psychotherapy to cash transfers, we discount interventions with relatively lower-quality evidence. We estimate that the evidence base for psychotherapy overestimates its efficacy relative to cash transfers by 11% (95% CI: 0%, 40%) because psychotherapy has lower sample sizes on average and fewer unpublished studies, both of which are related to larger effect sizes in meta-analyses.

We reviewed 28 sources that estimated the cost of psychotherapy and included 11 in our summary of the costs of delivering psychotherapy (see Figure 3 below). Nearly all are from academic studies except the cost figures for StrongMinds. The cost of treating an additional person with lay-delivered psychotherapy ranges from $50 to $659.

Figure 3: Distribution of the cost of psychotherapy interventions

Having established estimates for the cost and the effectiveness of psychotherapy, we can now show in Table 3, that psychotherapy is estimated to be 12 times (95% CI: 4, 27) more cost-effective than cash transfers.

Table 3: Comparison of  cash transfers to psychotherapy in LMICs

Note: 95% CIs are presented in parenthesis below the estimate.

In Figure 4, we take the results in Table 3 and show the results of simulations that compare the cost-effectiveness of each intervention. The total effect is on the y-axis and the total cost is on the x-axis for both interventions. Each point represents a run of a simulation. The higher the slope of the line, the more cost-effective the intervention. What should be clear is that the distributions of cost-effectiveness do not overlap. However, we do not include spillover effects on the household or the community in our comparison. Spillover effects are much more uncertain for reasons we explained earlier.

Figure 4: Cost-effectiveness of psychotherapy compared to cash transfers

The following considerations and limitations are discussed in detail in our full psychotherapy CEA report:

• There may be issues with assuming that a 1 SD improvement in MHa scores is equivalently informative about wellbeing as a 1 SD increase in SWB measures (such as happiness or life satisfaction). For example, changes in MHa may poorly predict changes in SWB. This does not appear to be a concern in practice because the measures seem both highly correlated and to give similar answers in meta-analyses of psychological interventions. That being said, we have not looked into the matter deeply. If using different measures gives different answers, then we would need to establish a conversion rate between MHa and SWB measures. Converting between measures of wellbeing requires both evidence of how these measures relate and judgements on which measures better proxy what really matters. We acknowledge that this is a source of uncertainty that further research could work to reduce. 
• The mental health scores for recipients of different programmes might have different-sized standard deviations. For example, the SD could be 15 for cash transfers and 20 for psychotherapy, on a given mental health scale. This would bias our comparisons across interventions. 
• Cost data are sparse for psychotherapy. Studies that report costs often make it unclear what their cost encompasses, which makes the comparison of costs across studies less certain. 
• Data on the long-term effects of psychotherapy (beyond 2 years) are also very sparse. Given we need to know the total effect over time, this means a key parameter – duration – is estimated with relatively little information. 
• We do not incorporate spillover effects of psychotherapy into our main analysis. This may matter if the relative sizes of household spillovers differ substantially between interventions,  but at the moment we do not have clear evidence to indicate that they do.

2.3 Strong Minds (see full analysis)

Having established that psychotherapy appears to be more cost-effective than cash transfers in terms of subjective wellbeing, we now need to identify the best donation opportunities for those who wish to direct more financial resources towards this intervention. We will now explain why we think StrongMinds is likely to be one of the best opportunities of this type.

In 2019, HLI began the Mental Health Programme Evaluation Project (MHPEP) to identify the most cost-effective charities delivering psychotherapy interventions in LMICs. The first round of the evaluation began with a review of 76 interventions from the Mental Health Innovation Network, from which we identified 13 priority programmes for further evaluation. Our screening criteria included: whether the intervention is targeted at depression, anxiety or stress disorders in LMICs; whether a controlled trial has been conducted on the programme; and an initial evaluation of cost-effectiveness. You can read more about the process we followed here.

In 2020, we began reaching out to organisations that deliver one of our priority programs. Unfortunately, many of the people we approached did not respond or told us they didn’t have enough time to participate. The only organisation that was on our shortlist and did provide detailed cost information was StrongMinds, a non-profit founded in 2013 that provides group interpersonal psychotherapy (g-IPT) to impoverished women in Uganda and Zambia. As a result, StrongMinds was one of the organisations we were most excited about prior to doing any detailed analysis. Friendship Bench, another promising mental health NGO, has since offered to provide us with their cost data, but we have not been able to review it yet.

Our cost-effectiveness analysis of StrongMinds builds on previous work by Founders Pledge (Halstead et al., 2019) in three ways. First, we combined broader evidence from the psychotherapy literature with direct evidence of StrongMinds’ effectiveness in order to increase the robustness of our estimates. Secondly, we updated the direct evidence of StrongMinds’ costs and effectiveness to reflect the most recent information. Lastly, we designed our analysis to include the cost-effectiveness of all StrongMinds’ programmes. This allows us to compare the impact of a donation to StrongMinds to other interventions, such as unconditional cash transfers delivered by GiveDirectly. 

We estimate the total effect of StrongMinds on the individual recipient to be an expected 1.92 SD (95% CI: 1.1, 2.8) improvement in MHa scores. Figure 5 visualises the trajectory of the effects of StrongMinds’ core programme compared to the average lay- or group-delivered psychotherapy through time.   

Figure 5: Trajectory of StrongMinds compared to lay psychotherapy in LMICs

StrongMinds records their average cost of providing treatment to a person and has shared the most recent figures for each programme with us. We assume that StrongMinds can continue to treat people at levels of cost comparable to previous years. It is also worth noting that StrongMinds defines treatment as attending more than six sessions (out of 12) for face-to-face modes and more than four for teletherapy. If we had used the cost per person reached (attended at least one session), then the average cost would decrease substantially. 

Having incorporated all our uncertainties into a Monte Carlo simulation, we estimate that a $1,000 donation to StrongMinds will result in a 12 SD (95% CI: 7.2, 20) improvement in MHa. This range of uncertainty is similar to that of the core programme (95% CI: 8.2 to 24 SDs). This may sound very large, but we expect that $1,000 will cover the treatment of around seven people. We show the distribution of this estimate in Figure 6 below. 

Figure 6: Distribution of overall cost-effectiveness of a $1000 donation to StrongMinds 

The following considerations and limitations are discussed in detail in our full StrongMinds CEA report:

• Will StrongMinds’ transition to implementing g-IPT through partners positively impact its cost-effectiveness?
• Do current calculations underestimate StrongMinds’ cost-effectiveness by excluding the effect and cost of those who did not complete treatment?

2.4 Comparison

Figure 7 illustrates a simulation of the comparison between psychotherapy, StrongMinds, monthly cash transfers, and GiveDirectly. Each point is an estimate given by a single run of a Monte Carlo simulation. Lines with a steeper slope reflect a higher cost-effectiveness in terms of SWB and MHa. Bold lines reflect the cost-effectiveness gradient of interventions, while grey lines are for reference. In this comparison, a $1,000 donation to StrongMinds is around 12 times (95% CI: 4, 24) more cost-effective than a comparable donation to GiveDirectly. 

Figure 7: Cost-effectiveness of StrongMinds compared to GiveDirectly

A word of caution is warranted when interpreting these results. Our comparison to cash transfers is only based on affective mental health. We were unable to find any studies that used direct measures of SWB to determine the effects of psychotherapy, so it remains to be seen whether the impacts are substantially different when using those measures instead. 

Additionally, this comparison only includes the effects on the individual and it is possible that the spillover effects differ considerably between interventions. That being said, even if we take the upper range of GiveDirectly’s total effect on the household of the recipient (8 SDs), psychotherapy is still around twice as cost-effective.  

In Table 4 we present the exact figures shown in the comparison of Figure 7. The key takeaway from showing the exact figures we used is to show that this result is driven mostly by the difference in the cost of each intervention, not the effectiveness. 

Table 4: StrongMinds compared to GiveDirectly

Note: 95% confidence intervals are shown below the estimates. 

3. Conclusions and next steps 

We think these reports show that working out how to do the most good by using subjective wellbeing is feasible and worthwhile. In particular, we believe that our analysis convincingly demonstrates that high-quality mental health interventions can be as good, or better than, highly-regarded and well-evidenced economic interventions. 

Of course, there are many more interventions that need to be analysed in terms of subjective wellbeing before a fuller picture begins to emerge, but we do want to stress to prospective donors that we consider StrongMinds to be an excellent donation opportunity if your aim is to improve global health and wellbeing. As of October 2021, they have a funding gap of $6.6 million for 2022. They are also planning for rapid growth (treating 50,000 in 2022, 100,000 in 2023, and 150,000 in 2024) equalling 300,000 women treated over the next three years. To fuel this growth trajectory,  they will need to raise $30 million in total over the next three years.

We suspect there are other highly effective donation opportunities waiting to be discovered. We have several avenues to explore and we can do this because our method allows us to compare different outcomes with a single metric. Our research pipeline includes a range of other promising interventions to analyse (conditional on us acquiring the necessary funding). These include Friendship Bench, mental health apps, cataract surgery, cement flooring, drug liberalisation (including better access to opioids and psychedelics) as well as advocacy to integrate wellbeing metrics in public policy (as a replacement/complement to GDP) and to make employee wellbeing a central indicator of environmental, social, and corporate governance (ESG) performance.

In addition, there are a number of theoretical questions about the nature and measurement of wellbeing that require further investigation. For instance, in order to compare StrongMinds to GiveWell’s death-averting recommendations we must first establish the location of the ‘neutral point’ on happiness scales — a vital, but highly contested factor in assessing the cost-effectiveness of life-saving interventions in terms of subjective wellbeing. For more, see our analysis using subjective wellbeing to compare life-saving to life-improving interventions. Further research is also required to establish the best methods for converting between different subjective wellbeing measures. You can learn more about our future plans in our research agenda.

If you found this post valuable, and you would like to support our future research, please consider making a donation to the Happier Lives Institute. We are actively seeking new donors to fund our future research. If you would like to discuss our research further or are thinking about making a donation, please contact hello@happierlivesinstitute.org.