Conducting remote cognitive interviews: telephone or video conferencing?

Our Methodology and Innovation Hub includes a team of experts who specialise in developing and testing survey questions to ensure measures elicit high quality data. Our day-to-day work often involves conducting interviews with members of the public to assess whether they understand survey questions as intended and can provide valid and reliable answers.

With the suspension of face-to-face interviewing in the wake of the COVID-19 crisis, researchers have had to adapt quickly to facilitate a smooth transition to new modes and methods. Whilst remote cognitive interviewing is by no means a new concept, it has become an essential tool for our researchers in recent months. The team has run several projects since the start of the pandemic, providing a valuable opportunity to pilot some different ways of conducting interviews remotely. Some of the main learnings from the last few months are outlined in this blog post.

One of the principal benefits of remote interviewing (either by telephone or video conferencing) is that it affords participants and researchers more flexibility when arranging and carrying out interviews. Giving participants the ability to be interviewed from a location of their choosing means that they can fit the interview around their daily routine, providing a strong incentive for many working people.

Previously, traditional methods using laboratories or hall tests may have been a barrier to some groups. Remote interviewing opens up the pool of potential participants to people who are less able to travel, such as older people or people with disabilities. Eliminating the need for travel also means that the sample is not constrained by geographic boundaries, and results in more diversity of location without the associated travel costs. Furthermore, interviewers can be more flexible if participants need to reschedule interviews, as travel arrangements and room booking are added complications when interviewing in person.

Certain video conferencing features allow researchers to replicate the conditions of face-to-face interviews. For example, screen sharing functionality allows the interviewer to present visual elements, such as showcards, or to show participants the questions as they would appear in an online survey. The video element is particularly useful when observing participants’ reactions to certain question wording or online survey features, as these nuances could be lost over the telephone. For example, interviewers can be more responsive to looks of confusion or boredom and can ask pertinent questions to understand the cause of that reaction. Participants are also likely to feel more comfortable talking about sensitive topics if they can see the interviewer. Some studies have even found that participants built rapport with interviewers faster over Skype than when interviewed in person.

Despite the benefits that video conferencing presents, there are still occasions when offering telephone interviews is more appropriate. For some, technology and internet literacy are still a barrier to participation, and it’s therefore advisable to offer telephone interviews where possible. When conducting cognitive interviews, where the aim is to explore whether participants understand and interpret questions in a consistent way, telephone interviews are a perfectly viable alternative. Our recent projects have found anecdotal evidence that, when offered the choice, people tend to opt for telephone interviews over video conferencing most of the time. This is likely due to the perceived ease of answering the phone, compared with actively setting up a device and joining a video conferencing session.

Irrespective of preference, if the aim of the research is to test how easily participants can navigate a web questionnaire, video conferencing is the superior choice. We have recently conducted usability testing interviews remotely for the first time and found that video conferencing had a number of specific advantages over telephone. The aim of usability testing is to observe participants completing a survey and identify any areas of its design that may make it more physically or cognitively challenging. Video conferencing lends itself well to this type of interview, as many platforms allow participants to screenshare so that interviewers can observe them completing the survey in real time, without the use of lab recording technology. This provides a wealth of non-verbal information which can indicate that the participant is having difficulty, from hesitation to scrolling up a page to reread the question. This is particularly useful when participants are less inclined or able to verbalise their thoughts and reactions, for example with participants who speak English as a second language.

Until face-to-face fieldwork returns, remote interviewing remains an excellent option, and perhaps one that could be utilised alongside more conventional modes in the future. Whilst face-to-face interviewing is typically considered the ‘gold standard’ of qualitative data collection, it’s clear that telephone and video conferencing do have legitimate advantages, both in terms of accessibility and practicality. The mass uptake of video conferencing during the lockdown also demonstrates that this technology is becoming more prevalent in our lives. It’s therefore reasonable to expect that the appetite for this type of research will grow in subsequent years, both for customers and the general public.