Phonics Denialism and the Education Policy Institute. Part 2
The EPI asks "What can quantitative analyses tell us about the national impact of the phonics screening check?" and then gives a ludicrous answer.
The story so far
Last November, the Education Policy Institute (EPI) published a report that really must be seen to be believed. The report examined the introduction of the Phonics Screening Check (PSC) in England. This is a short, statutory assessment taken by Year 1 pupils1 to assess their ability to decode words using synthetic phonics. Children are asked to read 40 words — some real and some made up — to check whether they can decode accurately. Those who fail to meet the expected standard are reassessed in Year 2. The main conclusion of the EPI’s report is that the improvements in the teaching of literacy, after the PSC was introduced, don't count. How this conclusion is squeezed out of data that show no such thing is impressively brazen.
In Part 1, I looked at how the EPI report interpreted PIRLS data.
In this post, I will look at how the report analysed the data from Key Stage 1 reading tests.
What the EPI did with Key Stage 1 reading tests
These are 70-minute tests of reading comprehension, taken in Year 2, that are internally marked by teachers. A test of this sort cannot be expected to be particularly reliable, so I will acknowledge from the outset that this is not a good way to compare literacy, or the effectiveness of literacy teaching, as they change over time. However, this is not the main problem with how the EPI used the data. Even if one were to accept that the Key Stage 1 reading test results were a suitable tool, one wouldn’t use them the way the EPI did.
In the three years after the Phonics Screening Check was introduced, the results of the reading test were better than in the three years before. The report shows an average pass rate between 85% and 87% in the three years before the PSC and between 90% and 91% in the three years afterwards. If this shows anything2, it shows an improvement. The EPI gets around this by assuming that instead of comparing scores from before and after the PSC, we should compare the rate at which scores are rising.3
The report explains:
Trends in key stage 1 reading and writing:
After the introduction of the PSC, an existing general upward trend in the proportion of children meeting the ‘expected standard’ in key stage 1 (KS1) reading continued, though at a slower and stalling rate compared to that before the introduction of the check. In the three cohorts immediately before the PSC’s introduction, after controlling for pupil and school factors, 85 per cent (2010), 86 per cent (2011), then 88 per cent (2012) met the ‘expected standard;’ in the three cohorts immediately afterwards, 90 per cent (2013), 91 per cent (2014), and then 91 per cent (2015).¹
It would be perfectly reasonable to argue that the increases in reading scores are not significant or reliable results. However, it is ludicrous to describe it as the slowing and stalling of an existing trend. That's indefensible. It doesn't matter what the issue is; anyone who interprets data like this is seeking to mislead. Before the introduction of the PSC, policy had already moved to make Systematic Synthetic Phonics compulsory. The fact that scores had improved dramatically at this point does not mean that the improvement after the PSC doesn’t count. To argue that continued, but less dramatic, improvement is a bad thing is to assume that previous gains would be repeated indefinitely.
How to misread a trend
Why should anyone assume that the rise in reading scores that happened after SSP became policy would be seen again and again, even in the absence of further interventions? There are several reasons why they wouldn’t.
Three data points are not enough to establish a trend.
We don’t expect improvements in the years immediately after a one-off policy change to result in a “trend”. We would expect to see a rise, followed by stability.
These data are percentages and, therefore, highly unlikely to follow a linear pattern. We should expect that moving from 88% to 91% will be more challenging than moving from 85% to 88%. The nearer you get to 100%, the harder further increases are going to be.
Missing that last point is the most unforgivable error here. It’s not just that the “trend” in the percentages from 2010 to 2012 could not have been expected to continue for years afterwards; it’s that it would be a mathematical impossibility. To claim the pass rate could have continued to rise by the same amount is to imply that, in the absence of the PSC, pass rates could have been over 100% by now. The PSC is being blamed for test results not following an impossible trajectory.
No evidence except the evidence
As I’ve already acknowledged, the test results don’t prove that the PSC worked. PIRLS data (see part 1) are more indicative, but also have limitations. However, the EPI are not saying "be careful". They are claiming that nobody found "a discernible positive impact of the PSC", even though some of the things they chose to look at appeared to show an improvement after it was introduced.
This is just embarrassing. If one is not going to be convinced by improved KS1 reading results (or PIRLS scores) after the check, why even look at them? The figures may not tell us much about the PSC, but their inclusion in the report tells us a lot about the authors' intentions.4 They selected that metric, and when it showed the opposite of what they wanted, they used trickery to wave it away.
Contradictions and convenient claims
Another flaw in the report is that it casually asserts:
As would be expected, children who ‘passed’ the PSC are more likely to go on to reach the ‘expected level’ in KS2 reading than those who ‘failed’.
This most definitely was not expected by the opponents of the PSC, who repeatedly argued that it was not a good indicator of reading ability. They even claimed that good readers would fail it because good readers don't sound out words. The EPI report even repeats a claim that the PSC ‘misidentifies pupils’.5 There is potentially a contradiction in stating that passing the PSC is a sign that one is more likely to reach the expected level in reading and that the PSC misidentifies pupils. Either the greater likelihood of passing, or the alleged misidentification, must not be particularly dramatic. If they were, then both claims could not be true.
Accurately representing the debate at the time of the introduction of the PSC is important. In the unlikely event that the check is not serving a useful diagnostic purpose, it might still serve a pedagogical one. Its introduction did seem to reveal a lot of fake phonics teaching. Before the PSC, there had been serious arguments about analytic phonics, a proposed alternative to SSP. When the people supporting analytic phonics opposed the phonics check, they often openly admitted that the children they were teaching "phonics" to would be unable to sound out words. It became far less plausible that opponents of SSP were teaching phonics adequately, but not using SSP. It became far more obvious that they wanted a situation where failure to teach phonics (to any meaningful degree) would not be detected or prevented.
To conclude…
There's a blatant thumb on the scale in this report’s weighing of the evidence. I worry it will be presented as if it were research, rather than propaganda, by people who already oppose evidence-based reading methods.
Year 1 pupils are aged 5 to 6.
I would argue that it doesn’t show anything. It is not good evidence for the effectiveness of the PSC, because of the limitations of the reading test.
This was a trick that was also used when the EPI was trying to dismiss the improvements shown by PIRLS, as shown in Part 1.
I was not shocked to learn that the main writer of the report was opposing the teaching of early phonics back when the check was new. She told the Education Select Committee in 2015 that you should not start teaching phonics in reception:
Because we have had these shifts in the curriculum recently in terms of the phonics reading test being introduced and that putting downward pressure on phonics teaching, we don’t know yet, because we don’t yet have the longitudinal evidence to say for definite. However, the research would support the idea that this is going to have a detrimental effect on children. If they are being put in a position where they are being set up to fail in what they are doing when they enter school, it does not take much extrapolation to think that this may have a negative effect later on.
DfE guidance recommends that children are taught phonics at four, at an age when it is normal not to have the capacity to say the sounds they need to say to learn phonics.
Page 17
Great analysis, Andrew. Thanks.