Evaluation of the REAL programme

About the study

The Raising Early Achievement in Literacy programme (REAL) is an intervention that works with practitioners (usually in the early years) to support children’s early literacy development. The programme aims to help practitioners to build parents’ knowledge and confidence in creating a home learning environment that supports and encourages children’s reading, writing and oral language.

The National Centre for Social Research (NatCen) was commissioned by the Education Endowment Foundation (EEF) to independently evaluate the REAL programme.

The REAL programme was delivered by the National Children’s Bureau (NCB). It was delivered over five school terms, starting in the spring term (2020) of the nursery year and finishing in the summer term (2021) of the reception year. The age of children in the intervention was three to five years old across the delivery period. The activities included training and networking events for practitioners and home visits by practitioners to families in the treatment group.

Practitioners were required to conduct eight to ten home visits per family over the course of the intervention, each lasting 30-60 minutes and involving sharing of learning resources, modelling of activities for the parents and high-quality conversations with the parents about the activities and the child’s learning. However, due to the COVID-19 pandemic, from March 2020, in-person home visits were paused. By that point, practitioners had managed to deliver on average two home visits per family. From September 2020, the intervention was re-started, but all remaining home visits were conducted remotely as a result of social distancing guidance. NCB recommended that remote visits could be delivered using a variety of remote methods (e.g. telephone calls, video calls, sharing resources in the post).

The programme was delivered in 53 schools in the North West of England. Practitioners delivering the programme were typically nursery teachers or members of schools’ senior leadership teams with responsibility for the Early Years Foundation Stage.

The evaluation design originally included a two-arm multi-site family-level randomised controlled efficacy trial. However, the impact evaluation was cancelled in March 2021 due to the COVID-19 pandemic. A standalone implementation and process evaluation (IPE) was still carried out on the basis that IPE evidence on feasibility and perceived outcomes would still be useful for further development of the intervention. The IPE took place in all participating settings and aimed to assess delivery and perceived outcomes on practitioners, families and children.

Aim(s) of the study

The main research questions for this evaluation were as follows:

1. How is The REAL Programme delivered, and what are the facilitators and barriers to delivery?
2. What are the perceived benefits of the programme for early years practitioners, families and children?
3. What are the implications of COVID-19 for delivery and perceived impacts?
4. What can be learnt for future delivery of The REAL Programme?

Findings

• The intervention was very well received by practitioners, most of whom maintained high levels of commitment to the programme throughout its duration. However, fidelity in the implementation was not very high, both in terms of dosage (the number of home visits delivered during the intervention), with only 60% of practitioners meeting the minimum requirements, and quality, with modelling of activities and high-quality conversations with parents not always being part of ‘remote home visits’.
• Key benefits of the intervention as perceived by practitioners were their improved knowledge of early literacy development, improved knowledge of how to work with families, greater understanding of children’s home environments, and increased confidence in supporting parents. In relation to early in-person home visits, practitioners described how visiting homes was ‘eye-opening’ because it showed the lack of suitable home learning resources.
• The evidence of positive outcomes for parents and children was less consistent. There was no evidence from the parent surveys that home learning environment in the treatment group families improved over the course of the intervention. However, in the practitioner survey, the majority reported that parents’ participation in their children’s learning and parents’ engagement in early literacy activities (i.e. features of a home learning environment) had increased as a result of taking part in the REAL programme.
• Remote delivery made it more difficult for practitioners to build relationships with parents and engage harder-to-reach families and undoubtedly affected the progress made towards parent and child outcomes, which was perceived by practitioners to be more limited than the progress towards practitioner outcomes. Over half (57%) of practitioners who responded to the survey said REAL home visits would preferably be ‘all in-person’, and 43% said they would prefer a mix of in-person and online. No respondents said they would prefer delivery to be exclusively online in future.
• Buy-in from schools’ senior leadership teams was a key factor facilitating delivery, as it supported adequate ring-fencing of time for practitioners to plan and carry out home visits. Main barriers to delivery included high teaching workloads, the need to schedule home visits around parents’ availability, a high rate of short notice cancellations by parents, low engagement with the programme from some parents, and the delivery across two school years. Many of these barriers were exacerbated by the COVID-19 pandemic.

Methodology

Data collection included observations of training and networking events, two surveys of practitioners (baseline and endline), three surveys of parents (baseline, interim and endline), interviews with practitioners and parents, and compliance data collection. The evaluation started in December 2018 and completed in July 2021.