As most people in any empirical or scientific field know, the gold standard for experimenting and establishing causality is the randomized controlled trial (RCT). In an RCT, subjects are randomly assigned to one of two conditions: an experimental group or a control group. The experimental group receives the intervention or drug and the control group receives standard care or a placebo (basically, the equivalent of the status quo). The idea behind the randomized controlled trial is to control the circumstances surrounding the experimental question as much as possible. This allows researchers to collect information on two groups that have essentially the same experience, except for the fact that one group received the intervention and the other did not. Because of this design, researchers can then measure outcomes at the end of the experiment and attribute any differences between the two groups to the intervention itself. In other words, RCTs help establish causality. In the absence of randomization, this type of control is almost impossible and it is much more difficult to conclude that the intervention is the cause of the outcome.
RCTs have been in use for a long time, mostly in clinical medicine. But are RCTs the best approach when it comes to dealing with complex systems such as healthcare or the economy? This is a question that has plagued global health and development experts for quite some time. In recent years, there has been more pressure particularly on development agencies to provide evidence that their interventions are effective. In some cases, the people calling for these outcome measures assert that RCTs are the only acceptable form of evidence. Some global health and development experts, such as Paul Farmer, have argued that this insistence is unhelpful and sometimes even unethical, since it is only ethical to conduct an RCT if you are not sure whether the intervention will be effective and in many cases, such as in the case of humanitarian assistance, it seems patently unethical to withhold interventions from a group of people just for the purposes of creating a control group. As Farmer argues, global health practitioners often “know the right thing to do” but struggle with figuring out the best way to deliver it. This circumstance does not call for an RCT, he claims, but rather resources and research on implementation.
Others, however, disagree with Farmer. One of the most enthusiastic champions of RCTs for health and development is Esther Duflo, a professor of economics at MIT and co-founder and co-director of the Abdul Latif Jameel Poverty Action Lab (J-PAL). In their book, Poor Economics, Duflo and co-author Abhijit Banerjee argue that we may not actually know that much about what works in global health and development. In their view, then, the quest to alleviate poverty worldwide does depend on conducting rigorous RCTs in order to understand in a more thorough and empirical manner what works and what doesn’t.
Duflo is not without her critics either. Many claim that, even putting aside the fact that RCTs are not always ethically or practically feasible and that they tend to be very expensive, the conditions in an RCT do not represent those of the real world and may not reflect how interventions will actually when rolled out in real-time on a larger scale. This is a problem of internal versus external validity, as public health experts have defined it. Internal validity refers to the extent to which you can actually assign the results of an experiment to the experimental intervention. On this parameter, RCTs do extremely well because they are specifically designed to avoid bias and confounding and often do lead to results that are valid in and of themselves. External validity, however, refers to the extent to which an experiment’s results are generalizable to the larger population. Since RCTs test only a small portion of the population in unusual and controlled conditions, they are less likely to have high external validity even if their results are internally sound. This is a serious concern in contexts such as global health and development, since these interventions interact intimately with large, complex systems and with other factors such as culture, belief, and geography.
RCTs certainly have their benefits. But they also have their drawbacks. Some global health experts are champions of RCTs and others are skeptical. The debate between these two camps will likely never be fully resolved. In the meantime, it is always a good idea to keep all these aspects in mind when designing an evaluation or experiment.