How can we make surveys more inclusive?

The article was first published in SRA's Research Matters – September 2024 edition, click here to read the article.

Children and young people (CYP) with special educational needs and disabilities (SEND) represent more than 17% of the school population in England, equating to over 1.5 million pupils. These CYP face additional barriers compared with their peers and, on average, do less well on measures such as educational achievement and labour market outcomes.

Currently we do not have enough robust, nationally representative data on how these CYP are getting on in other aspects of their lives nor quantitative data on their school experiences, including whether the support they receive shapes their experiences and later outcomes.

What we did

In a recent study, the SEND Futures Discovery Phase, we set out to understand how to gather such data. Across two waves, we collected quantitative data from a nationally representative sample of around 3,000 pupils with SEND in year 8 and year 9 (aged between 12 and 14) and their parents/carers across England. The survey used face-to-face in-home interviews, telephone interviews and online surveys.

A key part of the study was to trial different approaches to engaging a wide and diverse range of young people. The study included young people with a range of needs, and a large sub-sample from groups who have tended to be under-represented in surveys.

What we learned 

It is possible to do a representative large-scale survey with young people with SEND and their families. We achieved response rates that were comparable with other studies and incorporated protocols to make the study more inclusive. However, we were not able to reach all the young people we set out to hear from, and learned important lessons about areas we need to think more about.

We identified several strategies that can help make our surveys more inclusive. 

• Around half of the young people who took part received support from an interviewer and/or a parent. This introduces variability in the data collection which is largely unknown and (especially in the case of parents) difficult to control. Developing protocols/advice on how to provide and record support can help address uncertainty for interviewers and participants, and help assess data quality issues.
• Using different survey modes may encourage and enable different groups to take part. In line with conventional (survey) wisdom, a face-to-face approach was generally associated with higher response and engagement. Nevertheless, an online mode may be particularly suitable for some young people, including some autistic young people and those with social, emotional and mental health needs.
• A shorter questionnaire can improve engagement, including those who struggle to concentrate and/or who have literacy needs. We saw this in a response experiment and in comments from interviewers and participants.
• The use of accessible communication tools designed for fieldwork, such as showcards with pictures and ‘taking part cards’ helped enable some young peopleto take part (though were perceived by some as ‘babyish’).

• Additional interviewer training on the experiences of CYP with SEND may have improved participant experience (but did not have any impact on response levels).

  The complexity of people’s needs and circumstances means that no single survey approach can be inclusive and engaging for everyone. What benefits one person may be unhelpful to another. Rather, a flexible approach that tailors content and protocols to the individual’s needs is likely to improve inclusivity and engagement. However, such an approach contrasts with the usual objective of survey research to collect standardised data in order to ensure comparability. 

  Who we want to hear from and what we want to know will influence the balance between tailored and standardised approaches. Sometimes an individualised, ‘non-standard’, approach is necessary, especially for populations for whom a ‘standard’ survey approach is inaccessible. If we want to design and implement effective, inclusive studies it is crucial to reflect on the need to use a ‘non-standard’ approach from the outset of the research. Then, once we have reflected on the implications of this, we can use the strategies shared by us and by colleagues in this issue.

  Note 

  The SEND Futures Discovery Phase was commissioned by the Department for Education to test feasibility and inform development of a potential future longitudinal study of CYP with SEND and their parents/carers. It was carried out by the National Centre for Social Research in collaboration with the National Children’s Bureau.