The Post Office Scandal
The Post Office scandal has been described as the ‘largest miscarriage of justice in British history‘. There are many strands to the story and many reasons why things took the path they did. Statistics isn’t the biggest of these, but it did play its part.
The facts of the case are now well-known:
- The UK Post Office introduce the so-called Horizon IT system to manage accounts across all Post Office branches;
- From the very start, Post Office branch managers reported problems with the system;
- Between 1999 and 2015, over 900 branch managers were prosecuted for money that was reported to be missing by the Horizon accounting system. In over 700 of these cases, the Post office itself led the prosecution. Many others were forced to use their own money to compensate the deficits Horizon had identified. At least four branch managers committed suicide. Many others were subject to bankruptcy and family breakdowns;
- Throughout that period, campaigners and media groups increasingly challenged the evidence that led to the prosecutions, arguing – with evidence – that the Horizon system itself was faulty;
- In 2017 group litigation against the Post Office was made my a number of branch managers;
- In 2019 the judge ruled against the Post Office, noting that the Horizon system contained bugs, errors and defects;
- Convictions started to be overturned from 2020, and legislation is now in place that will ensure all criminal convictions arising from this scandal are overturned;
- A public inquiry into the scandal is currently hearing evidence from all parties. Evidence has been seen that shows Post Office executive managers know about faults in the Horizon system as early as 2013;
- As of today, no executive at the Post Office or Fujitsu, the Horizon developers, have been prosecuted.
- Despite all evidence, as recently as February of this year the Post Office management have said that they stand by the vast majority of prosecutions they made.
The Mis-Use of Statistics
Statistics comes into the case in at least 2 ways.
First, Post Office experts claimed in court that the chance of a transaction being faulty due to the Horizon system was 0.0002%, or 2-in-a-million. This meant, they claimed, that if money had disappeared from a Post Office branch account, the chance of it being due to an error made by the Horizon system were 0.0002%. The argument was therefore made that when money did disappear, it was virtually certain to be due to error or criminality on the part of the branch manager.
This is the well-known Prosector’s Fallacy. An accused branch manager is either Innocent or Guilty. If they are Innocent, the probability of the Evidence (it is argued) is 0.0002%. In statistical terms, this is a conditional probability
P(Evidence given Innocent) = 0.0002%
This meant, it was claimed, that where money did go missing, the chance of it not being due to criminality or negligence was vanishingly small:
P(Innocent given Evidence) = 0.0002%.
In other words, when errors occurred the chance of the branch manager being guilty was 99.9998%.
Except, it isn’t. This is rubbish. As James Christie – an IT audit expert – explains in this article, Horizon processed around 8 million transactions on a daily basis, so a 0.0002% error rate would lead to around 16 errors per day. If you focus on a branch at which one off these discrepancies occurs, it’s illogical to say that there was just a 0.0002% chance that the mistake was due to Horizon. To do so is – quite literally – to adopt the Prosecutor’s Fallacy.
The Prosecutor’s Fallacy is an easy mistake to make – chances are we all do so from time to time – but its mis-use in the Post Office scandal seems to have been an attempt to distort statistical evidence so as to justify the incorrect conviction of nearly 1000 innocent individuals.
Fortunately, the High Court judge could see the flaw in the reasoning. Here’s what he wrote:
Consider a hypothetical bug, bug X. Also consider that bug X impacts upon branch accounts in a single branch upon a single occasion leading to a shortfall in the branch for that branch trading period… Analysis and resolution of the correct and true situation of the branch accounts between the Post Office and the SPM for the trading period in question does not depend upon whether, in all the other millions of branch accounts, there was no such incidence of bug X… Expert IT evidence of most assistance in that exercise would be whether or not bug X exists or existed, and what were its effects. It is of no assistance to have an exercise that in effect says the statistical likelihood of any bug having an impact upon the branch accounts of that branch in that period is very low.
And also…
The … analysis is, in my judgment, so riddled with plainly insupportable assumptions as to make it of no evidential value. It is the mathematical or arithmetic equivalent of stating that, given there are 3 million sets of branch accounts, and given there are so many sets of branch accounts of which no complaint is made, the Horizon system is mostly right, most of the time.’
In summary: the Post Office tried to use a classic sleight-of-hand statistical trick to justify its prosecution of branch managers. And though it had succeeded on a case-by-case basis, the prosecutions were revoked in the High Court once the statistical – as well as all other – evidence was properly reviewed.
The Non-Use of Statistics
A second issue, it seems to me, is where Statistics wasn’t used. There are around 11,500 Post Office branches in the UK, and over the 16-year period starting in 1999 almost 1000 office managers were prosecuted. That’s around 8% of people who have dedicated their lives to the Post Office by buying a franchise to run a local branch. I don’t have the data, but it’s safe to assume that before the Horizon system was introduced, there were very few prosecutions of branch managers for financial irregularities.
So here’s the question. You’re the Chief Executive of a large organisation. You have responsibilities to the stakeholders, to the customers and to the employees of your organisation. You’ve just introduced a new company-wide software system for accounting. As the proportion of branch managers being prosecuted for fraud gradually increases from something close to 0% to around 8%, at what point do you start to question the system rather than the operators?
Admittedly, you didn’t immediately have 8% of your branches suffering financial irregularities. It started with a few. Then a few more. Then quite a few. Then it reached 8%. And all the way through, you had two simple hypothesis to consider: Hypothesis A: the Horizon system is perfect, and all irregularities are due to fraudulent activity by branch managers; Hypothesis B: the Horizon system is not perfect and could explain the observed financial irregularities.
There’s a whole branch of statistics called sequential analysis dedicated precisely to problems of this type. These are situations where you don’t get your whole sample in one go, but the data trickle in, and you need to determine if at any point the evidence tilts from supporting one hypothesis to the other. So the methodology is available to carry out this sort of analysis, but no one at the Post Office seems to have done so.
But even without a formal analysis of this type, surely – surely! – the most casual consideration of these statistics at some point during the period from 1999-2015 would have led to at least some reasonable doubt that Hypothesis B was a more likely explanation than Hypothesis A. Instead, the statistics of what was happening – just like the human consequences – seem to have been hardly considered. And where statistics were used, they were used incorrectly to shift blame away from the system and on to the individuals being prosecuted.
Summary
Statistics isn’t the main culprit in this story, but it was used in an attempt to distort the evidence in favour of the Post Office. And where Statistics might have been used to question the sudden rise in irregularities after the introduction of the Horizon system, it never was. Anyone looking at the facts of this story is bound to be outraged; as a Statistician the outrage is compounded by some shame as well.
