This article originally appeared in the December issue of Research Matters magazine, published quarterly by the Social Research Association (SRA). SRA members can access the latest issue of Research Matters online now. Back issues are made available to non-members three months after publication.
Within the last five years, social survey data collection in the UK has changed significantly. Gone are the days when face-to-face interviewing was the obvious data-collection mode for academic and government-funded social surveys. Instead we see the growing use of online data collection, sometimes on its own but predominantly in combination with more traditional data-collection modes such as postal, telephone and face-to-face. This development has revealed questionnaire design practices that are not suitable for online data collection, and particularly not for smartphones which are increasingly the device of choice for online survey participants.
Our decades-long use of face-to-face interviewers to collect survey data has allowed us to develop questionnaires that rely on interviewers to motivate and help respondents to produce full and accurate answers. This has allowed us to slip into some bad habits to meet the complex data needs of many stakeholders, resulting in long and dull questionnaires, and questions that are complicated and burdensome to answer. The issue facing survey customers and designers is whether to adapt these questionnaires so that they can be completed online or to develop new measures that collect comparable data across modes and devices.
Getting the basics right
The temptation is to stick with what we know and to adapt existing questions rather than starting from scratch. That applies to new surveys as well as existing surveys. Existing surveys, however, have the added complication that there is an entrenched desire to maintain valuable time series which can thwart changes to question design – even when it can be demonstrated that this will improve data quality. However, if we switch to online data collection without fixing our bad questionnaire design, we get the worst of two worlds: uninterpretable trends with unimproved questions. To successfully use online data collection in social surveys, it is vital to get the basics right.
1. Good questionnaire design
It is often claimed that this new era of online data collection requires new questionnaire design principles. However, adhering to the recommendations made in classic text books that were written before the internet became publicly available would go a long way towards making questionnaires suitable for online completion, including on smartphones. This includes keeping questionnaires short, minimising text length, reducing response lists, using simple language, minimising cognitive burden by keeping tasks manageable and so on (see for example Payne, 1951; Kahn et al, 1957; Sudman et al, 1982; Converse et al, 1986). We need to acknowledge that there is a problem with how we currently design questionnaires, and that we have neglected to enforce the basics of our craft.
2. A user-centred approach
Given the absence of an interviewer to motivate and assist online survey participants, it is vital that we adopt a user-centred approach to questionnaire design. To meet the data needs of survey customers, we need to understand what motivates people to take part in online surveys; how they access online questionnaires; and how they process questions and provide answers online. Like the Government Digital Service, we need to adopt a ‘user-first’ mantra.
3. A multi-disciplinary approach
Designing good questionnaires for online use requires a multi-disciplinary approach. In addition to subject experts and survey methodologists, it requires technical knowledge and expertise in visual design, usability testing and data processing.
4. Closer collaboration between survey customers and designers
Finally, ensuring that questionnaires are fit for online and smartphone completion will benefit from closer collaboration between survey customers and those responsible for survey design and delivery. Decisions about questionnaire design are often made by customers after consulting internal and external stakeholders, and this can result in lengthy questionnaires and cognitively burdensome questions. Involving survey designers from the outset can help to manage expectations of what can be collected online.
References: Payne, S.L. (1951). The art of asking questions. Oxford, England: Princeton U. Press. Kahn, R.L. and Cannell, C.F. (1957). The dynamics of interviewing: theory, technique and cases. New York: Wiley. Sudman, S. and Bradburn, N.M. (1982). Asking questions: a practical guide to questionnaire design. San Francisco, CA: Jossey-Bass. Converse, J.M. and Presser, S. (1986). Survey questions: handcrafting the standardized questionnaire. Newbury Park, CA: Sage Publications.