Discuss

The growing discussion and use of contribution analysis, an evaluative methodology for determining causality developed by John Mayne in 2001, in the evaluation of international assistance holds promise in its application in assessing the outcomes and impacts of peacebuilding initiatives.

Two recent papers by 3ie and DFID have outlined and extended the range of approaches and methods for impact evaluation. Included among these is a contribution analysis approach to attribution of cause and effect in evaluation.

Direct attribution of cause and effect is difficult in peacebuilding for a variety of reasons: intangibility of results, complex causal chains of what is oft long-term change, and the complexity and interconnectedness of the range of issues and environments in which we work. Furthermore, the applicability of experimental and quasi-experimental methods and approaches is still being debated with as of yet no clear consensus. In light of these trends, peacebuilders have been searching for evaluation approaches that are not only capable of capturing results, but which also resonate with the common worldview that many peacebuilders share: that peace is a collective, long term effort.

What is it?

Contribution analysis is an evaluation approach to determining the extent to which a program contributed to observable outcomes and impacts in the environment. It does not seek direct attribution of cause-effect.  

Rather, contribution analysis collects evidence on a specific circumscribed causal issue (i.e., a specific outcome or impact based on a specific theory of change) and develops multiple explanations for the evidenced change. These explanations are then weighed according to their plausibility, strength of evidence, and strength of logical linkage to the theory of change in a participatory manner that includes input from beneficiaries and other stakeholders. Cumulatively, the chosen and plausible explanations constitute a ‘contribution story’ that outlines which links in the results chain are strong and which are weak, the overall credibility of the story (does the pattern of results and links validate the results chain?), whether or not stakeholders agree with the story and observed results, and the main weaknesses of the story.

The contribution story results in a determination of the extent to which a program did or did not contribute to observed changes and therefore the validity of the theory that the program embodies.  

There are six essential steps in contribution analysis:

Identify the causal issue to be addressed. For example, which level of outcome or impact is sought, at the micro, meso or macro levels? This matters, as a program could be found to have contributed to, for example, ethnic relations at the community level, but found to not have contributed at the national level. This also requires exploring the extent to which contribution is expected, and the identification of other key influencing factors on the outcomes.  

Identify and/or develop a theory of change, assumptions and risks. As a theory-based approach to evaluation, contribution analysis requires the identification, and agreement amongst key actors and stakeholders, of a plausible theory of change, its assumptions and their associated risks. Here, the evaluator needs to determine the level of expected contribution made by a program and it can be useful to determine the program’s ‘circles of influence’ (Montague, et al., 2002):

• “direct control—where the programme has fairly direct control of the results, typically at the output level;
• direct influence—where the programme has a direct influence on the expected results, such as the reactions and behaviours of its clients through direct contact, typically the immediate outcomes and perhaps some intermediate outcomes;
• indirect influence—where the programme can exert significantly less influence on the expected results due to its lack of direct contact with those involved and/or the significant influence of other factors.”1

It’s also necessary to identify the roles of other influencing factors and rival explanations for why the observed change occurred. Based on these factors, the evaluator can determine the extent to which the theory of change is contested in this particular instance.

Gather evidence to assess the strengths and weaknesses of the links in the theory of change. Evidence can be gathered from a range of sources for: the observed results; for each of the links in the results chain; for other influencing factors; and, for rival explanations. As a mixed-methods approach, many data collection tools can be utilized to support contribution analysis, but critically, should include monitoring and project design data.

Assemble and assess the contribution claim. Here the evaluator begins to develop the contribution story that explains how a program contributed to certain observed results. The quality of the story should be assessed based on the strengths and weaknesses of the postulated theory of change in light of the available evidence, the relevance and strengths of other influencing factors, and the relevance and strength of the evidence gathered to support rival explanations.

Seek out additional evidence. Additional evidence for the contribution story should be identified and sought based on the assessment of the initial contribution story.

Revise and strengthen contribution story. New evidence collected in the previous step is incorporated into the contribution story. Steps four through six should then be repeated until a plausible contribution story can be constructed and agreed upon.

Contribution Analysis in Peacebuilding: More Exploration Needed

If you subscribe to the Sage journal publication ‘Evaluation’, you will have seen that the entire July issue is dedicated to contribution analysis. There are some very worthwhile articles in this special issue, and if any of the issues in this blog sound interesting, I strongly encourage you to find a way to get access to the journal because as a field, we need to do more to grapple with this approach and its applicability. Blogs such as this do wonders for introducing a topic, but are limited in promoting in-depth analytical exploration of and dialogue on this methodology.

So, what needs to be done? First we need organizations and donors willing to test out this approach and provide some thought leadership in applying this methodology to our work. This involves short pieces like this, providing an introductory overview of the topic. But we also need in-depth exploration of what the methodology looks like, tool development, and serious thought on what contributions are plausible and valid in a particular context for a particular type of project, and identifying the implications of these conclusions for the design, monitoring and evaluation of projects using contribution analysis: do we need to be using nested logframes to recognize complexity in the design phase, if so under which conditions?; do we also need to develop implementation theories of change in addition to the action theory of change—to what extent and under what conditions does one effect the other?; etc.

And of course, actions must follow thought, and for that we need to budget an appropriate amount to conduct a contribution analysis well (which means hiring evaluators with experience in this methodology, which again comes back to funding).

The increasing exploration of contribution analysis in international development holds promise for peacebuilding evaluation. We should not underestimate the value of learning from other fields’ experiences with new and emerging evaluation approaches and methods, but nor should this replace our own research and critical thinking.

• 1. John Mayne, “Contribution analysis: an approach to exploring cause and effect,” ILACT Brief 16, May 2008.