Evaluations During Difficult Times
Author: HUR HASSNAIN
Evaluations in contexts that are not easily accessible, constantly changing, and unpredictable, for example, the situation posed by the current global pandemic, are challenging and call for special considerations. Now that in-person interactions with respondents becoming impossible, we would like to illustrate the use of satellite data that can be applied for remote evaluations of development and humanitarian interventions. “The use of geospatial data analysis is particularly relevant under COVID19 since most of the field travel linked to our evaluations is restricted” echoed Luisa Belli, an Evaluation officer at the FAO. We hope this blog can be a useful resource for the global evaluation community during these times, where shared learning and experiences are very much needed.
At any given time, there are several thousand satellites in space collecting a vast amount of data on changes that occur on earth. These data have been widely used to understand the different systems on earth, however, there is limited guidance on how to incorporate these into an evaluation. This is one of the main reasons behind the limited use of satellite data and spatial methods in evaluations. The International Development Evaluation Association [IDEAS] is about to launch its guide on Evaluation in Fragility, Conflict, and Violence, including during pandemics. It contains additional information on geospatial approaches along with other methods to gather data remotely.
During the times of a pandemic, war, or natural disaster, the effectiveness of interventions is difficult to evaluate. However, remote sensing methods could be leveraged in such situations. Satellite remote sensing derived indicators, and proxy indicators are particularly useful in collecting time-series data on land cover changes, water quality, and quantity, natural disaster, crop detection, and agricultural yield estimation, pollution levels, economic activities, urbanization, track illegal activities and even migration.
In this blog, we used satellite data to assess the night-time light emittance to validate the progress made by a solar lighting project implemented in rural Sierra Leone. We show how the results of space observation can be interpreted and used by the project implementers, donors, and other stakeholders to make effective decisions on similar interventions in Sierra Leone and elsewhere.
Assessing a rural electrification initiative in a challenging context
The Rural Renewable Energy Project (RREP) is an electrification project funded by the UK Department for International Development (DfID) and implemented by the United Nations Office for Project Services (UNOPS) to support the Government of Sierra Leone’s goal of universal access to electricity (SDG-7). The project aims to provide access to off-grid solar electricity to up to 94 communities in the country, specifically targeting the Community Health Centers to store vaccines and provide health services.
Two electrified sites, a school in a rural village benefitting from electricity and a Community Health Center (Photo by Hur Hassnain)
You will be surprised to know that only 2.5 per cent of the rural population in Sierra Leone have access to electricity. The RREP is expected to improve the country’s economic development through an increase in access to rural energy resources. Rural sustainable electrification is also expected to increase welfare by improving income, health, and education, and lowering fuel expenses. It will also contribute to reducing greenhouse gas emissions.
Reaching the selected communities for the RREP intervention is a big challenge due to the poor road infrastructure in the country. And travel is a time and cost-intensive process there. Rural communities speak different languages and have very low literacy levels at about 33 percent. Although satellite data analysis cannot completely replace the conventional ways of doing evaluations to answer effectiveness questions or capture community perceptions, it can be very useful in real-time monitoring and/or evaluating in situations of a pandemic, conflict, violence, and/or during lack of time and financial resources.
What did the data show?
We used nighttime satellite data to examine the change in light intensity at the Bumpeh Community Health Center in Bo district in Sierra Leone. Nighttime light data acquired by NASA VIIRS is effective in monitoring electricity, urbanization, and economic activity. Figure (a) shows the night time data for Sierra Leone for 2012 and 2019. One can see an expansion of light intensity around Freetown and a relatively insignificant increase in other parts.
Figure a: Nighttime lights data for Sierra Leone, 2012(Left), and 2019(Right). Freetown can be seen in the upper-left corner, and the Bo district is in the lower right part of the images. Data Source(NASA VIIRS)
The analysis of nightlights trend for the Bumpeh CHC shows an increase in ambient light, indicating the effectiveness of the electrification project (Figure b).
Figure b: The times series of light intensity at the project site. The red line shows the time of implementation of solar-based electrification.
The analysis of nighttime lights trend before and after the project shows that there is a statistically significant difference in the pre-project and post-project mean ( t-test, p-val ~0). Overall the mean light intensity has increased from 0.15 to 0.31, an increase of 106 percent (Figure c).
Figure c: The change in light intensity before and after project implementation.
These results from the analysis of nighttime data were also confirmed through telephone interviews with local contacts. This exemplar also highlights how remote sensing data can be used as objective evidence that also helps triangulate findings from other methods.
That’s great - so how do we go about it?
The first step to using geospatial methods is to ask whether the evaluation involves looking at changes that can be analyzed through satellite data. The availability of accurate location information on project sites is crucial as well.
Along with assessing the suitability of using remotely sensed data, the evaluation team also has to identify the relevant satellite-based indicators or proxy indicators for capturing change. That will help estimate costs, and to make decisions about using readily available products, which may be free or commercially available, or develop customized satellite data products.
Satellite data provide objective evidence on how and where; however for causal and explanatory information (the ‘why’), the findings should be triangulated with other sources of information. When field validation becomes challenging, expert opinions, literature review, news reports, telephone interviews, social media or geographic platforms such as OpenStreetMap for crowdsourcing data can be utilized.
Evaluation offices often are not accustomed to using geospatial methods and may not have in-house expertise, therefore, collaborating with universities or academic institutions, space agencies, and specialized firms will help reduce the cost and deliver high-quality evaluative evidence.
In evaluation, innovation might not be imperative during business as usual scenarios, but it could be an investment for difficult circumstances. Evaluation offices that have invested in innovations will be more resilient and prepared to deploy these methods within a short time frame. Under the current circumstances, some readily available user-friendly platforms can be explored – the Global Resource Watch, the Water Explorer, the UN Biodiversity Lab, and the Sentinel Hub, etc. For more enthusiastic evaluators the current situation could be an opportunity to expand geospatial skills through some popular massive open online courses (MOOCs).
We wish you a happy earth exploration..10..9..8….
Disclaimer: Views expressed here are those of the authors and do not necessarily reflect the opinion or position of their employers nor is this a promotion of the project discussed.