Incentives in the Ethnic Minority British Election Study and National Travel Attitudes Survey
Part I: Push-to-Web and mixed incentives in the Ethnic Minority British Election Study 2021 Pilot
The 2021 Census revealed a growing population of people from ethnic minority backgrounds.
Yet, the largest representative study of political and social attitudes of Britain’s ethnic minority population, the 2010 Ethnic Minority British Election Study (EMBES) – a face-to-face probability study – is over a decade old.
Building on the successful transition of BES2019/2020 to push-to-web as an immediate response to Covid, the EMBES 2021 Pilot investigated the feasibility of conducting research among this population using a push-to-web methodology.
While push-to-web has been assessed to produce reasonably representative results for studies of general population, we address its suitability for recruiting a new representative sample of ethnic minorities.
We adapted the British Election Study questionnaire to incorporate an initial screener questionnaire, with different incentives depending on eligibility for the main questionnaire.
We found that response rates were lower than expected. Leveraging the fact that we used two recruitment approaches, we suggest that this more complicated incentive structure reduced response rates.
Further analysis however suggests poor response rates were not limited specifically to ethnic minority respondents, or to the questionnaire with the screener.
This indicates that general population samples recruited through push-to-web methodologies may under-represent geographies where minorities usually concentrate.
Our results therefore have important lessons not just for ethnic minority surveys, but also generalist surveys which want to make use of the cheaper push-to-web methodology in recruiting a probability sample.
Part II: Every penny counts? Exploring the impact of incentives on response, sample composition, and data quality in the National Travel Attitudes Survey panel
Over the past few decades, surveys have consistently seen declining response rates globally.
The use of incentives has proven to be one of the most effective strategies to counteract this trend.
Extensive research indicates that incentives have a small to moderate positive effect on response rates and can reduce field efforts, leading to lower survey costs.
However, the impact of incentives varies based on the research design, survey mode, the type and value of the incentive, and the timing of the offer.
While incentives are known to positively impact response rates, their effect on non-response bias and data quality remains less clear.
In this study, we present the results of a randomised experiment embedded in wave 8 (2023) of the National Travel Attitudes Survey (NTAS), a mixed-mode panel based on respondents to the English National Travel Survey.
Sample cases were randomly assigned to receive one of three conditional incentives: £5, £10, or £15 Love2Shop vouchers. Informed by the Total Survey framework, our analysis examines the impact of incentives on three primary outcomes: (1) response (i.e., response rates, share of web responses, and time to completion); (2) non-response bias (comparing the achieved samples with benchmark data); and (3) data quality (including non-substantive responses, item endorsement, validity, internal consistency, and acquiesce).
The findings indicate that higher-value incentives increased response rates without having a negative impact on sample composition or data quality.
Eva AizpuruaResearch Director National Centre for Social Research
Eva Aizpurua, PhD, is an experienced survey methodologist with a broad range of experience across academia, the private sector, and non-profit organizations. As a Research Director at NatCen, she focuses on developing optimal designs for social surveys undergoing remodelling. Previously, Eva worked as a Research Scientist on the Demography & Survey Science team at Meta. She has also held Research Fellow positions at the European Social Survey Headquarters and Trinity College Dublin, and was a Postdoctoral Scholar at the University of Northern Iowa. Eva currently serves as the Editor-in-Chief for Survey Practice.
Shane HoweResearch Director National Centre for Social Research
Shane Howe is a Research Director based in the Centre for Social Survey Transformation. His experience focuses on quantitative methods with an emphasis on survey and questionnaire design. He has worked on several large-scale cross-sectional household surveys, including the English Housing Survey, Taking Part and British Social Attitudes.
He has recently been part of several projects which aim to understand the impact of shifting modes on survey data and minimise disruption to trend series data. Recently he has helped to transition the British Social Attitudes questionnaire from face-to-face data collection to online and telephone assisted interviewing. In addition to this, he has also played a leading role in developing the National Travel Attitudes Study, a multi-mode panel study on behalf of the Department for Transport, using both web and telephone interviews.
Shane’s main interests reside in questionnaire design and improving the user experience for respondents in a self-completion environment.
Nicole MartinPolitical scientist University of ManchesterNicole Martin is a political scientist interested in how social cleavages shape electoral outcomes, and, as a result, who has power in advanced industrial democracies. More specifically, Martin studies how the long-term social changes of migration and educational expansion shape electoral behaviour in Great Britain and beyond. Martin frequently works with experiments in large-scale social surveys, including Understanding Society, the UK Household Longitudinal Study, which has large ethnic minority and immigrant boost samples.