Author: Hur Hassnain
In contexts where it is not possible to have direct interaction with respondents, e.g., during pandemics and/or when the environment is not safe, conducting data gathering for research and evaluation remotely is a commonly accepted way to generate primary data. However, the possibility of causing risks to evaluators and researchers as well as the participants of an evaluation is to be assessed on a case-by-case basis. The European Commission DG DEVCO/ESS’s Call to Action Paper on Evaluation in Hard to Reach Areas provides a great account on this topic. In hard to reach areas and in situations of global crisis -such as the COVID-19 pandemics, it is important to reassess which planned evaluations remain a priority and to analyse the possibilities of conducting them through the use of appropriate methods and tools. Postponement options may need to be considered if it is not possible to meet the appropriate conditions required to carry out an evaluation. Remote data collection methods have become commonplace for many organisations and consequently there is an increasing number of methods and tools which can be utilized. The Secure Access in Volatile Environments (SAVE) research programme suggests that the use of remote monitoring and evaluation works best in conjunction with agencies’ functioning monitoring systems, and when it is limited to areas with constrained access..
There is now a growing use of digital tools and this has influenced the way information is gathered, analysed, reported and shared. There is however not much documented evidence on the efficiency and effectiveness of using Information and Communication Technology (ICT). There is a trade-off between robustness of data and no data at all. In a global crisis it is advisable to invest in good enough data rather that delaying for better data. When face-to-face meetings are difficult to organise, ICT solutions can offer creative solutions to compensate for the lack of direct interaction. Among the tools are remote sensing including satellite imagery and the use of drones, artificial intelligence, robots, block-chain and social media monitoring while the use of smart or simple mobile phones and SMS are arguably the easiest and most widespread technology available. If done properly, SMS-based surveys, as well as ‘simple’ phone interviews, are a great way to reduce risk to evaluators and respondents in areas that are hard to reach and in situations of global crisis.
An evaluator needs to be aware of both the pros and cons of using ICTs for evaluation especially in difficult situations like pandemics and while evaluating in the contexts of fragility, conflict and violence (FCV). Examples of the benefits ICT tools can bring include allowing for more systematic tracking of progress as well as aggregation and sharing of data at various levels which can improve participation in the planning process and overall decision-making. New and user-friendly software tools are now easily available to enhance the development and management of theories of change from a distance. In addition, cost savings can accrue from replacing paper, and the elimination of manual data entry. This helps to reduce the carbon footprint of the evaluation activities contributing to environment sensitivity and also increasing the efficiency of the analysis of large sample sizes. The validity of an evaluation’s findings depends in a large part on the data quality. ICTs can play a helpful role in checking to ensure the right subjects were selected and interviewed, or that questions were asked correctly, in the right order and with the correct follow-ups. The use of ICTs can eliminate the need to re-collect data over and over, or to manually copy information from poorly legible manual entries to a digital format.
When planning for the use of ICT and mobile phone-based techniques, evaluators and evaluation commissioners must be aware of the possible negative effects of digital and phone divide in the area of intervention. A scarce / limited access to the internet (or to mobile phones) by target respondents can introduce an important element of sample selection bias in the primary data collected during the evaluation. DataReportal is a formidable and free ally for this analysis, with its in-depth reports of internet access, mobile phones distribution and social media use in more than 230 countries worldwide. However, further factors shall be taken into consideration, such as literacy of target respondents, gender-related barriers to the use of social media or phones, etc. There are various ICT based tools and techniques which can be used. Here we present a selection – each presenting their own list of benefits and challenges.
The phone and SMS based surveys do not need internet access and are doable on any kind of phones. It is mainly used for, short, largely closed ended questions, while it is challenging for qualitative types of enquiries. This method works well in contexts where literacy levels are low. This option requires access to phone numbers so more preparation time is needed. It can also be an expensive option and requires high quality enumerator training.
There are several ways of administering surveys remotely either through a weblink or from a mobile app. It is a low-cost method which allows both qualitative & quantitative responses. Surveys can be sent on mass via email and the survey can be longer than an SMS. Available options include Kobotoolbox, surveymonkey, Sprockler, SurveyCTO, Google Forms, etc.
Online Interviews and FGDs can be conducted using audio-visual interfaces such as GoToMeeting, Skype, Microsoft Teams, Webex, Zoom. The benefit of this method is the option to record the session for playback at a later date, however, as per face to face interviews, efforts need to be made to keep participants engaged in the online conversations. For the FGDs, the low bandwidth of some participants may result in intermittent audio/video that can impact on group dynamics.
Social media analysis can contribute to evaluations in many different ways, but -when used- evaluators and evaluation commissioners shall be aware of the fact that opinions and views on social media tend to be extreme, represent only users of the media themselves and do not necessarily reflect the sentiment of the ‘silent majority’ or large parts of the population. Use of social media is furthermore determined by the possibility to access to internet by the target population, as a consequence of both infrastructural elements and cultural barriers
Social media analysis can contribute to evaluations in many different ways, but -when used- evaluators and evaluation commissioners shall be aware of the fact that opinions and views on social media tend to be extreme, represent only users of the media themselves and do not necessarily reflect the sentiment of the ‘silent majority’ or large parts of the population. Use of social media is furthermore determined by the possibility to access to internet by the target population, as a consequence of both infrastructural elements and cultural barriers
Twitter data can be used in terms of sentiment analysis, key phrase extraction and social network analysis of tweets.
Facebook can be used for gathering data as well as analysis, phrase extraction and social network analysis of posts and trends. People can form groups, both private as well as public where they can hold discussions on various topics.
Whatsapp can be used to capture social behaviour within everyday life contexts and to explore situations and experiences of media use in dialogue with the participants. Methods can involve mobile instant messaging interviews (MIMI) through diaries and mobile experience sampling. (Kaufmann, K., & Peil, C. 2019). The use of qualitative data from Whatsapp can be used to gather complex and personal stories within the context.
Participants can complete a diary or journal log which can be structured (like a questionnaire) or unstructured. This method can use interval-based or event-based sampling (i.e. they record something every hour or every day vs. when it occurs, which may be more irregular). Different online platforms can be used to record the logs which can be visual (photo-based, collage or written) or spoken (voice recordings/memos).
This can be used to gather, show, and analyse imagery and data of certain geographical areas from space observations to identify outputs, outcomes and impact.
GEMS enables teams to use simple open-source tools to collect digital data from areas where access is often not possible. This method has been rolled out in many World Bank funded projects in FCV contexts.
These can be administered remotely and are used to make sense of what happens in organisations and communities. Stories can be related to personal experiences, observations, and situations. Quantifiable data is gathered which contains the meaning, feelings, and motivations in the context shared. Both Sprockler and Sensemaker are two online tools which can be used for story gathering exercises. For more information see:
In PV methods, the respondents take control of the content and making of the video hence enhancing the ownership of the product and empowerment in return. The final products may not be of professional quality standards but can be improved further with a light touch editing by an expert to complete the process.
When deciding on whether to use ICT tools and which to use it may be helpful to consider the dimensions of the 5’A’s of technology access: availability, affordability, awareness, ability, and agency. Structuring analysis around these five dimensions helps to decentre the technology and to highlight the social and political factors that can limit technology access.
This blog post is written by Hur Hassnain and Marco Lorenzoni and was first published on the European Commission’s knowledge sharing website: https://europa.eu/capacity4dev/devco-ess/news/evalcrisis-blog-no-01-embracing-pandemic