Mitigating the risks of remote data collection for evaluations

Author: Hur Hassnain

Within the research and evaluation world, the use of remote data collection, especially with the use of ICTs, has become increasingly prevalent. It provides us with the opportunity to gather primary data in contexts where it is not possible to have direct interaction with respondents, e.g., during the current COVID-19 pandemic and/or when the environment is not safe. However, remote data collection also poses several potential risks which need to be assessed on a case-by-case basis.

In the first blog post of the #Evalcrisis series Embracing the Pandemic we discussed how ICT tools can provide solutions to conduct data collection remotely during the current crisis as well as in hard to reach contexts and we provided the pros and cons of utilizing different methods e.g. remote surveys, online interviews, geospatial technology.

Whilst remote data collection and the use of ICTs can bring huge potential for evaluations conducted during the current crisis and in hard to reach contexts, evaluators and commissioners need to be mindful of the risks which should be considered.

What are the risks?

Risks can include increased attention to the evaluation activities, the risk that sensitive data gets in wrong hands and the risk to geo-localize surveyors and enumerators. It is important to remember that many people still experience limited internet/mobile phone access therefore conducting remote data collection in some contexts could lead to the risk of sample selection bias in the primary data collected. Added to this are factors such as the literacy of target respondents and gender-related barriers to the use of social media or phones. The European Commission DG DEVCO/ESS’s Call to Action Paper on Evaluation in Hard to Reach Areas provides a more in depth account on this topic, especially conflict sensitivity, Do No Harm and ethics in doing or managing evaluations in such contexts. 

How can potential risks be minimized?

This is a question that almost every evaluator in today’s time is asking. Although the discussion on this may require a series of blog posts but let us start from the following steps to avoid interruptions to your remote data collection activities:

  • Do a risk assessment: Identify potential unintended (negative) consequences and limitation measures. Would the remote data collection methods you are thinking to use create additional risk for anyone? Is the potential benefit worth it? Think about consequences that could result from the introduction or use of ICTs, including domestic violence against women, theft, and harassment from authorities.
  • Conduct a context analysis: Understand the nature of the context. If planning to use ICT tools – how is technology seen in the community for women and men separately? Are the tools suitable for the context you are evaluating? Choose the right tools considering the socio-cultural, political and technological environment and its feasibility. SAVE Toolkit ‘Technologies for Monitoring in Insecure Environments’, (2016) is a great reference to consider when defining your data gathering plan.
  • Be aware of local laws and policies: Study the privacy policies of the mobile service providers, government data collection and storage regulations. If you are using ICT devices make sure they have appropriate security and privacy features. Whilst ICTs can offer some options for enhanced protection such as password protection, encryption, or panic buttons to delete data, we also have to be careful to navigate local laws and context and be conscious of risks.
  • Leverage earth observation (EO) data: This data can help you to understand the context in hard to reach areas. EO analytics can be used for planning the evaluation as well as evaluating the outcomes. Read more about evaluations from space observation in the EvalCrisis blog.
  • Ensure good quality enumerator training: Conducting data collection remotely requires enumerators and local evaluators to work more independently. This requires them to have a sound knowledge of the purpose of the data collection and the protocols to follow (including specific protocols related to the current crisis). Enumerators should receive context specific conflict and gender sensitivity training and be aware of who they can contact should they need support before/during the data collection activities. It is good practice to get feedback from enumerators after the data collection each day and to involve them in the analysis of the data to help with sensemaking. With the current pandemic, the need for local evaluators/enumerators and their knowledge and understanding of the context has become much more important. It is also a blessing in disguise for the local and especially young and emerging evaluators #YEEs from the global south to get more firsthand experience on the ground.
  • Involve communities and individual stakeholders: Ensure that people understand why data is being gathered for the evaluation and that they are fully aware of their rights. Consent must always be obtained prior to any data collection and personal information should only be gathered when necessary.
  • Be aware of different levels of access and inclusion: Marginalized members of a community or group may be left out if ICT-enabled M&E is not designed with inclusion in mind. Build local capacity and consider ways to overcome barriers which could lead to sample selection bias.
  • Invest time in creating a trusting environment: Doing this before and during the data collection can help to avoid problems or biased answers. This can involve raising awareness amongst the communities about why the data is being collected and how it will be used. If using ICTs, it is important to inform people about how and why the technology is being used for gathering data.
  • Use mixed/appropriate method approaches: If possible remote data collection should not replace face-to-face contact but instead complement it. Of course, during the current crisis and in some contexts, the option of face-to-face contact is not possible. Do what is feasible and safe.
  • Ensure that data is used responsibly: There is both a legal and moral obligation to manage data gathered for evaluations appropriately. Sources online such as Oxfam’s Responsible Data Management Training Pack (2017) can help you with planning processes to manage data collected.
  • Triangulate your findings: To avoid selection and other bias while using remote data collection, use participatory and mixed methods. Where possible use data from other sources (e.g., National Statistical Offices, other agencies, donors, academia, the UN) to triangulate with your own findings.
  • Close the feedback loop: Share the evaluation results effectively with programme participants and partners e.g. during validation workshops and involve them in the analysis of the data to support better decision-making and learning. Does the data make sense to them? Are the overall findings expected? Check out this blog post on ‘closing the evaluation feedback loop’ for some firsthand guidance and tips on this topic. It is good practice to request feedback from evaluation stakeholders when the remote data collection process has completed. Findings can help to improve data collection activities in the future.

Disclaimer: Views expressed here are those of the authors and do not necessarily reflect the opinion or position of their employers.

This article was first published on the European Commission knowledge sharing website: https://europa.eu/capacity4dev/devco-ess/news/evalcrisis-blog-04-mitigating-risks-remote-data-collection