Broadening the Range of Designs and Methods for Impact Evaluations: Report of a Study Commissioned by the Department for International Development

This report covers a study commissioned by DFID entitled 'Broadening the Range of Designs and Methods for Impact Evaluations.' Up to now most investment in IE has gone into a narrow range of mainly experimental and statistical methods and designs that according to the study's Terms of Reference, DFID has found are only applicable to a small proportion of their current programme portfolio. This study is intended to broaden that range and open up complex and difficult to evaluate programmes to the possibility of IE. 

Three elements - evaluation questions; appropriate designs and methods; and programme attributes - have to be reconciled when designing IE. Reviews of existing evaluations suggests that sometimes methods chosen are unable to answer the evaluation posed; or the characteristics of development programmes are not taken into account when choosing designs or deciding on evaluation questions. 

Demonstrating that interventions cause development effects depends on theories and rules of causal inference that can support causal claims. Some of the most potentially useful approaches to causal inference are not generally known or applied in the evaluation of international development and aid. Multiple causality and confirgurations; and theory-based evaluation that can analyse causal mechanisms are particularly weak. There is greater understanding of counterfactual logics, the approach to causal inference that underpins experimental approaches to IE. 

Designs need to build on causal inference approaches each of which have their strengths and weaknesses, one of the reasons that combining designs and methods - so called 'mixed methods' - are valuable. Combining methods has also become easier because the clear distinctions between quantitative and qualitative methods have become blurred, with quantitative methods that are non-statistical and new forms of within-case analysis made easier by computer aided tools.

On the basis of literature and practice, a basic classification of potential designs is outlined. Of the five design approaches identified - Experimental, Statistical, Theory-based, Case-based and Participatory - the study has in line with its ToR concentrated on the potential of the latter three. 

The study has concluded that most development interventions are 'contributory causes'. They 'work' as part of a causal package in combination with other 'helping factors' such as stakeholder behavior, related programmes and policies, institutional capacities, cultural factors or socio-economic trends. Designs and methods for IE need to be able to unpick these causal packages.