Double-blind lessons

 

 

 

 

 

 

 

 

 

 

ECONOMICS lab experiments and their field siblings—the randomised control trials (RCTs) used in development economics—are important building blocks of a relatively recent line of research. And like anything new and successful, it attracts criticism. Some of this criticism, however, is fully justified.

In a new paper,  Erwin Bulte of Wageningen University and his colleagues conduct a double-blind (!) test of an agricultural intervention—that is, the treated don’t know whether they are receiving the treatment or the placebo. The treatment is a modern seed of cowpeas, the placebo is the traditional seed. As a second experiment in a different set of villages, they do a normal RCT where the treated know that they are receiving the modern seed. Comparing the results of both experiments reveals some striking results. When the farmers don’t know which seed they are planting, there is no difference between the modern and the traditional seed in terms of yield. When they do know that they are being treated, the modern seeds yield considerably more. What the authors call the “pseudo-placebo effect” therefore accounts for the whole effect that a typical RCT would have found.

In economics in general, double-blind testing makes little sense: if one input into your production function changes, you should adjust other inputs as well. In order to do that, you need to know whether you are being treated. This is why the word “treatment” has always been misplaced as it suggests passivity on the part of the treated. In medicine, that may be true; in economics it is certainly not. An agricultural intervention is a perfect example: high yielding varieties require different inputs as they are usually more sensitive to fertiliser or irrigation, and farmers should employ these better seeds on the best plots.

The authors are perfectly aware of all this, of course. What their study does, therefore, is to point to three potential problems with RCTs.

First, adjustment on the part of the farmer is economically justified and an important aspect of any economic intervention. However, behavioural responses have costs. Any RCT that does not properly account for this will provide a misleading gross effect of the intervention—or an upper bound on the true effect, as Mr Bulte puts it. Their double-blind effect on the other hand is a lower bound: farmers cannot adjust their behaviour in a targeted way because they don’t know whether they have modern or traditional seeds. The problem is that researchers are not always aware of, and thus cannot account for, all the behavioural adjustments that the treated could make.

Second, is the behavioural response of the farmer optimal? If it is not, a one-time RCT may find a large effect that would not be the long-term effect that policymakers are ultimately interested in. And in fact, Mr Bulte and his colleagues find this to be a problem. The effect that a normal RCT would have found in their setting is based on behavioural responses only: the farmers were just overly optimistic at the beginning about the power of modern seeds and increased various inputs. This artificially increased the yield of the modern seeds, but not in an economically profitable way. The farmers, who know their business inside out, therefore overwhelmingly declined to use the modern seeds on an ongoing basis. Around 60% of farmers with the modern seeds even suggested that their overall production had declined, although part of this may be due to disappointment with the failure to meet expectations. A one-time RCT may therefore find the effects of optimistic behavioural adjustments, but no long-run equilibrium.

Finally, the main reason why RCTs became so popular was that they offered the possibility of cleanly measuring the overall effect of an intervention. However, most researchers have moved past the stage where the overall effect was all that mattered. Understanding the mechanisms behind this overall effect—the “black box” as Alix Zwane of the Bill & Melinda Gates Foundation calls it—has moved to centre stage now and RCTs are being modified accordingly. A double-blind study like the one above shows how important it is to understand the adjustments that farmers—or any economic agents, really—make in response to being treated. Or in response to merely being surveyed.

Empirical work in an area as messy as economics will never be as clean and simple as in, say, chemistry. It is therefore important to constantly double-check the methods. A double-blind test, while not the gold-standard in economics, may point the profession to important pitfalls.       economist.com