Randomised controlled trials (RCTs) have become a popular methodological choice for policy analysis in the developing world. Through a review of literature in multiple disciplines, this paper describes the various ethical and methodological considerations when choosing to adopt RCTs for policy decisions. Unlike previous critical analysis of RCTs, this paper contextualises its critique to India, a country that has been the site of well over a hundred RCTs. Through illustrations of recent Indian policy RCTs, on corruption, livelihoods, PDS and conflict etc, the paper raises concern about violation of ethical requirements like equipoise, informed consent, data harms, human costs to research participants and research staff. Following this, the paper discusses methodological limitations of RCTs for Indian policy making including heterogeneity, researcher effects, generalisability, policy-relevant unobserved mechanisms and other socio-political considerations. The paper ends with a description of alternative approaches and a simple checklist for practitioners, specifically policy makers, to assess the feasibility of RCTs for informing decision making in their context.