Real World Evaluation: Working Under Budget, Time, Data and Political Constraints

Reconstructing Baseline Conditions

Many evaluations do not begin until the project has been underway for some time or perhaps is even nearing completion. It is very common under these circumstances to find that no baseline data have been collected at the beginning of the project. This is most commonly the case for the comparison group, but it is also often true for the project group. The absence of baseline data is usually one of the most serious threats to validity, and therefore, RWE proposes a number of different ways to reconstruct baseline data. Some of these approaches, described in Table 8 (see also Chapter 5), include the following:

• Using secondary data (see following section)
• Using individual recall (respondents are asked to recall the situation of their family or community at around the time the project began)
• Using PRA and other participatory techniques to reconstruct the history of the community and to assess the changes that have been produced by the project
• Interviews with key informants, preferably persons who know the target community, as well as other communities, and therefore have a perspective on relative changes occurring over time

Although all these methods provide potentially valuable information, there are significant threats to validity inherent in any recall method. These result from lack of precise memory, the tendency to confuse the precise time period (so that events that took place earlier may be reported as having occurred since the project began or vice versa), and in some cases, deliberate distortion. Consequently, it is important to treat all recall data with caution and to always use mixed-method approaches to triangulate independent estimates of the reported information from different sources.