A few weeks ago, along with ten other NatCen colleagues I attended the biannual European Survey Research (ESRA) Conference in sunny Slovenia. Between us we chaired three sessions and presented 11 papers on our recent methods work. Survey methods may sound like a bit of a niche topic, I admit, but it definitely didn’t feel like it with 650 attendees from over 40 countries and up to 12 parallel sessions running at any one time!
One of the topics that got a lot of attention was using ‘mixed mode’. A survey mixes modes when it uses more than one channel to ask people the questions - that could mean sending people a questionnaire by post but also giving them an opportunity to complete the questionnaire online.
One of the reasons researchers are enthusiastic about mixed modes is that it holds a promise of saving money in the long run, an increasingly important concern in the current economic environment. Take an expensive face-to-face survey where interviewers visit respondents to do the interviews. If people could complete some of these interviews in a cheaper mode, say by telephone or online, then you would expect the survey to be cheaper.
While it makes sense in principle, there isn’t much research to show whether that’s actually the case. At NatCen, we’ve carried out a number of mixed mode studies over the past few years, so we’re in a good position to shed some light on this issue. My presentation at ESRA focussed on the response and costs in Understanding Society Innovation Panel wave 5, while my colleague Alun Humphrey spoke about our experience with the mixed mode European Social Survey.
Understanding Society’s Innovation Panel is a test bed for all sorts of experiments with questions and procedures that are too risky for the main panel. In wave 5 (2012), one of these experiments was mixed mode. We invited two thirds of our sample members to take part in a web survey and only if they didn’t respond online, did we then send an interviewer to visit. One third were not told about an online option and where only visited by interviewers.
We were a bit worried that not many people would be prepared to spend an hour answering questions online, so you can imagine how delighted we were when nearly a quarter of respondents went ahead and completed interviews for everybody in their household online. However, we weren’t as delighted to find that when we combined the people who responded late to interviewers and those who had taken part online, then the total response rate was lower than among the one third of the sample who had been interviewed face-to-face without the option of going online.
Did this help to save costs? In fact, the cost of the study did go up (and not down) compared to when it would have been done just by interviewers. This wasn’t all that surprising, because moving such a complex questionnaire online takes time and therefore money. The set-up costs aside, we did save money on fieldwork, but not quite as much as we would have expected. At the outset we thought we would save by having to pay interviewers for fewer completed interviews and for fewer miles travelled. The former held true, but the latter didn’t. That’s because the vast majority of travel is between interviewer’s home and the area where they work and not between the addresses within an area. So one additional address doesn’t make that much of a difference.
Are there any other circumstances where we could save money with mixed modes? I simulated the costs for different sample sizes to find out and what I could see was that there’s definitely a point at larger sample sizes when mixed modes become cheaper than face-to-face. However, with smaller samples, the set-up costs are going to be larger than what can be saved in fieldwork. Where that point is, is probably dependent on how complex a survey is – for simpler tasks mixed modes become cost effective for smaller sample sizes than for more complex surveys.
So to conclude, mixed modes do have potential to save money. However, it will not automatically happen on any survey. We need to consider the complexity of the survey and the sample size to make an informed decision about whether it is worth going mixed modes with a survey.