"Your recent leader (“In praise of human guinea pigs”, December 12th) argues strongly in favour of using randomised controlled trials (RCTs) in public policy as they are used in medicine. You are insufficiently cautious in pointing to the serious methodological concerns around using RCTs as a basis for public policy analysis and formation...
We would argue that the biggest problem is the difficulty in applying a methodology developed in medicine, where test subjects share the same genetic blueprint and new treatments are developed at the molecular level, to the social sciences, where our models are nowhere close to the precision of theories in the natural sciences and where there are many unobserved and unobservable differences between human beings that can confound the best designed RCT.
In the language of statistics, this is known as the problem of “generalisability” or “external validity”. In layman’s terms, how and to what extent can one use the results of a randomised trial to inform policy in another place or at another point in time? This question is at the frontier of research, and preliminary findings by one of us (Rajeev Dehejia and co-authors) suggests strongly that extreme caution must be exercised in this regard.
What is more, unlike the natural sciences, social scientists have an array of increasingly sophisticated data-based methods and data sets with which to address public policy questions. By all means add RCTs as a new tool in our kit, but in our enthusiasm to embrace what is currently fashionable, not lose light of their limitations, nor discard classical tools of analysis already at our disposal."