The marking system which threatened to downgrade the exam results was not a “mutant algorithm” as claimed by the prime minister, but contained errors that were “predictable” and “could have been foreseen”, according to a leading statistician.
Boris Johnson told a group of schoolchildren on Wednesday that “I’m afraid your grades were almost derailed by a mutant algorithm,” a reference to the exam marking system which was scrapped after almost 40% of A-level results were downgraded from teachers’ predictions.
Boris Johnson said the algorithm was ‘mutant’
Stian Westlake, the CEO of the Royal Statistical Society (RSS) said the “colourful phrase” did reflect the fact that ministers, officials and students “were surprised by the results that it generated”.
But, he said, the algorithm “wasn’t a mutant or a freak of nature. This was something that could have been foreseen”.
Mr Westlake, a former adviser to the Universities Minister, said the results were “a predictable surprise” because of the demand that it reduce grade inflation. “The fact that this could have produced a lot of inaccuracy, which translates into unfair grades for individuals, that was known,” he said.
Education Secretary Gavin Williamson has blamed exam regulator Ofqual, claiming he “consistently asked a large number of challenging questions about the system” and was reassured about its “robustness and its fairness”.
Education secretary, Gavin Williamson said he asked ‘challenging’ questions about the system
But Ofqual has said that it was instructed by ministers to keep grade inflation down, which meant students’ results would have to be downgraded from teachers’ predicted grades, which tend to be higher than actual results.
Professor Jo-Anne Baird, a member of Ofqual’s standing advisory committee, told the BBC: “The Secretary of State for Education asked them to produce a system which didn’t allow exam results to go up from previous years. So effectively this government has been very good at controlling grade inflation and that’s essentially what they asked for, a system that did that.”
Asked whether or not the exam marking algorithm was “mutant”, Downing Street declined to comment.
Leyton Sixth Form College student Wiktoria Sniadowska, who was predicted AAA by her school then given BBC by the algorithm, said she felt let down by the prime minister’s “mutant algorithm” comments.
“Boris Johnson is prime minister, he’s in charge. If it’s a mutant, we knew it was going to be created this way for months in advance, so why didn’t he stop it? Why didn’t he say: ‘This is wrong, we need to do it differently’?”
Mr Westlake said the row showed the need for transparency in the creation of algorithms, not only because it helped prevent errors, but also because it allowed a public debate about their aims.
“I think the really important lesson from this is that we can’t treat statistics as a holy grail,” he said.
“We need to prepare them with transparency, we need to make sure we understand the assumptions that are going into them and we need to make sure that we use them in the right way.
“If you’ve got statistics that only have a certain degree of accuracy, as for example the A-level adjustments did, we can’t pretend they are more accurate than they really are.”
The RSS offered to help Ofqual with the algorithm in April, writing to the regulator to suggest that it take advice from external experts, but Ofqual would only agree to consider the assistance if the academics signed a non-disclosure agreement (NDA) which prevented them from commenting in any way on the final choice of the model for five years after the results were released.
Ofqual’s Roger Taylor disputes the RSS account of their correspondence
Ofqual chair Roger Taylor disputed this account, describing the NDA as a “normal and entirely ethical mechanism to ensure people can speak freely in the discussions and can see analyses of confidential data without fear of information being made public during the deliberative process”.
The RSS defended its stance, saying it had been advised that the NDA was “very widely drafted” and described Ofqual’s complaints as “baseless”. The letter concluded: “It’s time for Ofqual to cease defending the indefensible and focus on fixing the issue.”
Sky News revealed last weekend that the chair of Ofqual owned a medical data firm that was involved in a major NHS care scandal. Roger Taylor’s company was criticised by the public inquiry into the Mid Staffs scandal for not giving “complete clarity” about an algorithm – the same fault Ofqual stands accused of.