M&E in closing spaces – how to assess and mitigate risks

Many NGOs today work in politically restrictive environments. Because of the sensitivity of these programs, however, the exchange of lessons learned and good practices is often limited. And yet, these environments – characterized (in the context of U.S. Government [USG] funding) by restricted liberties, limited civil society, and a repressive government that rejects USAID assistance and/or has a severely adverse relationship with the United States[fn]“USAID Guidance on Programming in Closed Spaces” obtained at http://www.usaid.gov/sites/default/files/documents/1866/USAID%20Guidance%20on%20Programming%20in%20Closed%20Spaces%20Ident.pdf[/fn]–can present unique challenges for monitoring and evaluation (M&E) and may carry significant risks to the organization, local partners, and beneficiaries. What methods and approaches can we use to verify that activities are progressing, evaluate results, and be responsive to changes on the ground given these very real risks and unique challenges?

IREX has found it useful to identify three types of risks: physical, digital, and reputational. These risks are assessed (and continually re-assessed) throughout the M&E ‘value chain,’ from data collection and management to analysis and reporting. Every environment will be different, so what may make sense in one project may be inappropriate for another. The challenge is balancing the safety of all parties with the desire to collect useful, valid, and reliable data. Consider reviewing the DG Strategic Assessment Framework developed by the DRG Center to determine focus areas for your risk assessment.

In the following paragraphs, we 1) give examples of digital risks and mitigation measures, demonstrating the need for comprehensive risk assessments at the very outset, and 2) reflect on methods and approaches for strengthening the rigor of M&E in closed or closing spaces.

Digital risks and mitigation strategies – The use of ICT to (at the very least) communicate with local partners or beneficiaries is almost unavoidable. In many cases, your organization may not have an office in the country and must work through local partners or organizations in neighboring countries. However, what is considered sensitive information can vary. This may include everything from all program data to only donor information. Consider how, in one context, encryption may be used to transmit information over a secure network, while in another, this may raise a red flag for authorities. The appropriate method and technology will depend on the capacity of your staff and partners, the sensitivity of the data, and the context-specific risks. In high-risk environments, air gapped computers (physically isolated from unsecure networks, such as the Internet), wall safes, and encryption may be required. However, in all cases, minimal standards and protocols (e.g. use of third party emails, anonymizing software, pseudonyms) should be established from the outset and enforced. In many IREX programs, process indicators that track implementation of security procedures are used to keep all parties accountable. For more on tools and guidance to improve digital security, check out the Digital First Aid Kit by the Digital Defenders Partnership.

Strengthen verification and rigor – In many cases, working in politically restrictive environments means you will have very little direct contact with beneficiaries. Instead, you may need to rely heavily on local partners and on self-reported data, resulting in weak baselines and potentially faulty assumptions about progress. Strengthening local partner capacity for M&E becomes integral. It’s also important to inculcate a “culture of safety” in all program operations, especially since partners may have higher tolerance for risk and may adopt practices that run counter to best risk-mitigation protocols. 

Given these unique challenges, an approach to strengthening a program’s M&E system may include using local project-affiliated monitors who have a vested interest in ensuring successful outcomes. Additionally, the use of mixed methods to provide additional insights can be invaluable. One technique that has proven particularly useful for IREX in restrictive environments where representative surveys are not possible is Most Significant Change (MSC). With its focus on collecting significant change stories, which are vetted by a panel using pre-determined criteria, MSC can be used in conjunction with other approaches. Lastly, monitoring should be systematized. All contact with local beneficiaries can be regularly tracked and documented (even informally), to confirm that beneficiaries are reached at least once and some more than once[fn] “Restrictive Environments: Monitoring and Evaluation Challenges and Guidelines,” USAID, Oct 20, 2011, p. 3.[/fn].

Security is paramount – Whatever the program or context, the security of your staff, partners, and beneficiaries should be paramount. Every program should make a deliberate effort to diagnose their unique security risks and to develop and enforce appropriate risk mitigation and management measures. The principle of “do no harm” should always apply. So do the best practices applied to any M&E system planning, which include streamlining the M&E process to the extent possible, minimizing the data collection burden, and integrating “project realities” throughout the DM&E process. While these environments present unique M&E challenges, it is still possible to apply creative approaches to M&E that do not compromise the physical, digital, or reputational safety of your staff, partners, or beneficiaries.