Evaluation of a funding programme aimed at improving success rates for students resitting GCSE maths

About the study

Basic Maths Premium (BMP) was an intervention developed by the Department for Education (DfE) that aimed to improve achievement in maths for 16- to 18-year-old students with a prior attainment of grade 3 or below in GCSE maths. This study aimed to test the effects of providing additional financial support to post-16 settings to boost GCSE attainment and the relative effectiveness of different modes of provision. There were three different models of financial additional support being tested: A) guaranteed payment of £500 per eligible student enrolled by November 2018; B) guaranteed payment of £250 per eligible student enrolled by November 2018 and a further £250 in the 2020/2021 academic year for every student who achieved the required pass in maths by Summer 2020; C) £500 in the academic year 2020/2021 for every student enrolled by November 2018 who went on to achieve the required pass in maths by summer 2020.

Findings

There was no evidence that BMP had an impact on the likelihood that eligible students passed a level 2 maths resit exam by November 2019. This is the case for BMP as a whole, and for each of the three funding models separately. 

There was no evidence of a differential impact of BMP on level 2 maths resit attainment according to students’ previous attainment, free school meals eligibility, institution size, or the amount of BMP funding spent by participating institutions. 

There was no evidence that BMP had an impact on the likelihood that eligible students sit a GCSE exam rather than a Functional Skills Level 2 exam or no maths resit at all. 

Perceived benefits for students included improved understanding of maths, confidence and motivation, increased attendance in lessons and attainment through additional teaching and contact hours, smaller classes, employing more experienced staff, and motivating staff through opportunities for development and progression. Perceived benefits for teaching staff included lower workload and pressure, improved opportunities for development and progression, and increased confidence. 

Models A and B were affected by severe implementation issues as ‘guaranteed’ payments were delivered late; ultimately not disbursed until March 2019. Settings in Model C reported that payment-by-results made it harder to finance support upfront and uncertainty about how much funding settings would receive made it difficult to allocate and spend resources.

Overall, this trial did not find evidence to support the premise that additional funding would translate into an improvement in level 2 maths attainment among students resitting these exams in post-16 education.

Methodology

The BMP pilot included an impact evaluation (IE) and an implementation and process evaluation (IPE). The IE was designed as a three-armed cluster-randomised trial with a quasi-experimentally defined comparison group. It was not possible to have an experimentally assigned control group for this trial since the DfE had committed to providing BMP funding to all eligible institutions. The IPE complemented the impact evaluation. It explored:

• institutions’ compliance with the intervention;
• fidelity of implementation to the intended model including levels of spending;
• usual practice to support GCSE maths resit students in treatment group settings;
• the resources and activities funded through the BMP; and
• institutions’ responsiveness, perceived outcomes, and costs of taking part.

The IE used the following sources of data for the statistical analysis: 

• Data from DfE’s National Pupil Database (NPD) to measure the primary (pass in maths level 2 during the 2018/2019 academic year or in the November 2019 resit period) and secondary (whether or not an eligible student re-sat GCSE maths during the evaluation time horizon) outcome and to derive covariates for the analysis.
• Individualised Learner Record (ILR) and school census data to help define student eligibility for the trial analysis. 
• BMP sample file data to identify treatment groups for randomisation. 
• Data linkage from three waves of a survey conducted with participating BMP schools. This data was used to carry out the dosage analysis as part of the trial. 
• Institution-level information drawn from the following publicly available sources: AEA data, Get Information about Schools, and the Further Education Skills Dataset. These datasets were used to derive additional setting-level covariates.

The primary analysis for the IE combined propensity score estimation with regression analysis to estimate causal impact. 

The IPE synthesised data collected from initial and follow-up interviews with heads of maths at treatment institutions and three surveys of treatment institutions.

A cost evaluation was conducted to compare the average cost of the intervention per funding model from the perspective of participating institutions.