Yorkshire and Humber COVID-19 campaign pilot

About the study

The Department for Health and Social Care (DHSC) funded a number of pilots of initiatives related to self-isolation in areas with high COVID-19 transmission. The programmes aimed to test creative ways to encourage people most at risk of catching and transmitting COVID-19 to test and to self-isolate if they test positive. Magpie created a communications campaign, named ‘COVID-19 Explained’ to make information about COVID-19 simpler and more accessible for people that are less likely to comply with COVID-19 regulations or to be vaccinated. NatCen was commissioned to conduct an evaluation of this pilot in Yorkshire and Humber. 

The campaign employed behavioural insights techniques to achieve increased vaccine confidence and increased COVID-19 testing. The evaluation originally included an assessment of the impact of the programme on engagement with Test and Trace and on self-isolation compliance, as measured by data sourced from NHS Test and Trace. However, the government changes on self-isolation affected the outcome data collection on self-isolation compliance and consequently analysis of impact on self-isolation compliance was not feasible. The campaign was launched in November 2021 and stopped in February 2022 in line with a reduction in COVID-19 communications nationally. 

Our evaluation of this pilot comprised two strands:

• An impact evaluation (IE), which used a synthetic control method (SCM) to quantify the impact of the pilot on desired outcomes of vaccination and testing.
• An implementation and process evaluation (IPE), which conducted a small number of interviews to understand the processes by which the campaign was implemented to meet its objectives, what influenced this, lessons learned throughout, and recommendations for what could have been improved.

Findings

Our IE findings showed:

• No conclusive evidence of any positive or negative impact of the campaign on testing as measured by the number of PCR tests per 1,000 people or the number of LTF tests per 1,000 people.
• No conclusive positive or negative effect of the campaign was found on vaccination take-up as measured by the cumulative rate of adults with one dose of an approved COVID-19 vaccine and the cumulative rate of adults with both doses.

Our IPE findings showed that:

• Based on the data collected from stakeholder and community leader interviews, we found evidence that the activities set out in the Theory of Change (ToC) were generally realised.
• The campaign identified a gap in bottom-up COVID-19 communications campaigns in the Yorkshire and Humber region. However, the co-production of materials sometimes meant that local ‘bottom-up’ needs conflicted with the ‘top-down’ messages from local authority and NHS sources, compounded by the many stakeholders required to sign off elements of the campaign.
• We found that the campaign worked with community leaders to co-create and test campaign materials, based on local needs and understanding, to raise awareness and educate the population about COVID-19. We found that these materials promoted the lived experiences of local people, alongside work with community leaders, to build trust within the community.
• The level of engagement of ‘trusted people’ to share campaign materials was difficult to ascertain. While community leaders reported sharing materials, they did so in various ways and degrees, usually as part of their wider work. This could, in part, be due to one limitation of the evaluation: that the community leader sample was small and purposive; that community leaders were recruited just as they had been notified that the campaign had been stopped.

There are several potential interpretations of the inconclusive findings around the impact of the campaign on testing and vaccination take up. Taking into consideration the evidence from the IPE, one possibility is that key elements of the campaign were delayed, coupled with the fact that key elements lasted too short a time or were modified. It should be noted that while evidence from the IPE suggested that most of the activities set out in the ToC were generally realised, there were many changes made to the campaign that may have meant that the campaign was not strong enough to produce change. Another element that could impact on the success, particularly in terms of measuring effectiveness on testing, is the fact that the measures used to detect change in testing uptake are very volatile. A final consideration for interpretation is that the measures used to detect impact on vaccination and testing aren’t especially well-aligned to the short-term outcomes that ‘COVID-19 Explained’ targets, that is outcomes on vaccination and testing are distant and not very sensitive to detect changes as result of the ‘COVID-19 Explained’ campaign.