“I don’t like your mum”

VAR, eh?

So, does video-assisted refereeing (VAR) improve the quality of decision-making in football matches?

Of course, that’s not the only question about VAR: assuming there is an improvement, one has to ask whether it’s worth either the expense or the impact it has on the flow of games when an action is reviewed. But these are subjective questions, whereas the issue about improvements in decision-making is more objective, at least in principle. With this in mind, IFAB, the body responsible for determining the laws of football, have sponsored statistical research into the extent to which VAR improves the accuracy of refereeing decisions.

But before looking at that, it’s worth summarising how the VAR process works. VAR is limited to an evaluation of decisions made in respect of four types of events:

• Goals
• Penalties
• Straight red cards
• Mistaken identity in the award of cards

And there are two modes of operation of VAR:

• Check mode
• Review mode

The check mode runs in the background throughout the whole game, without initiation by the referee. All incidents of the above type are viewed and considered  by the VAR, and those where a potential error are checked, with the assistance of replays if necessary. Such checks are used to identify situations where the referee is judged to have made a ‘clear and obvious error’ or there has been a ‘serious missed incident’.  Mistakes for other types of incidents – e.g. the possible award of a free kick – or mistakes that are not judged to be obvious errors should be discarded during the check process.

When a check by VAR does reveal a possible mistake of the above type, the referee is notified, who is then at liberty to carry out a review of the incident. The review can consist solely of a description of the event from the VAR to the referee, or it can comprise a video review of the incident by the referee using a screen at the side of the pitch. The referee is not obliged to undertake a review of an incident, even if flagged by the VAR following a check. On the other hand, the referee may choose to carry out a review of an incident, even if it has not been flagged by the VAR.

Hope that’s all clear.

Anyway, the IFAB report analysed more than 800 competitive games in which VAR was used, and includes the following statistics:

• 56.9% of checks were for penalties and goals; almost all of the others were for red card incidents;
• On average there were fewer than 5 checks per match;
• The median check time of the VAR was 20 seconds
• The accuracy of reviewable decisions before VAR was applied was 93%.
• 68.8% of matches had no review
• On average, there is one clear and obvious error every 3 matches
• The decision accuracy after VAR is applied is 98.9%.
• The median duration of a review is 60 seconds
• The average playing time lost due to VAR is less than 1% of the total playing time.
• In 24% of matches, VAR led to a change in a referee’s decision; in 8% of matches this change led to a decisive change in the match outcome.
• A clear and obvious error was not corrected by VAR in around  5% of matches.

This all seems very impressive. A great use of Statistics to check the implementation of the process and to validate its ongoing use. And maybe that’s the right conclusion. Maybe. It’s just that, as a statistician, I’m still left with a lot of questions. Including:

1. What was the process for checking events, both before and after VAR? Who decided if a decision, either with or without VAR, was correct or not?
2. It would be fairest if the analysis of incidents in this experiment were done ‘blind’. That’s to say, when an event is reviewed, the analyst should be unaware of what the eventual decision of the referee was. This would avoid the possibility of the experimenter – perhaps unintentionally – being drawn towards incorrect agreement with the VAR process decision.
3. It’s obviously the case when watching football, that even with the benefit of slow-motion replays, many decisions are marginal. They could genuinely go either way, without being regarded as wrong decisions. As such, the impressive-looking 93% and 98.9% correct decision rates are probably more fairly described as rates of not incorrect decisions.
4. There’s the possibility that incidents are missed by the referee, missed by VAR and missed by whoever is doing this analysis. As such, there’s a category of errors that are completely ignored here.
5. Similarly, maybe there’s an average of only 5 checks per match because many relevant incidents are being missed by VAR.
6. The use of the median to give average check and review times could be disguising the fact that some of these controls take a very long time indeed. It would be a very safe bet that the mean times are much bigger than the medians, and would give a somewhat different picture of the extent to which the process interrupts games when applied.

So, I remain sceptical. The headline statistics are encouraging, but there are aspects about the design of this experiment and the presentation of results that I find questionable. And that’s before we assess value in terms of cost and impact on the flow of games.

On the other hand, there’s at least some evidence that VAR is having incidental effects that aren’t picked up by the above experiment. It was reported that in Italy Serie A,  the number of red cards given for dissent during the first season of VAR was one, compared with eleven in the previous season. The implication being that VAR is not just correcting mistakes, but also leading to players moderating their behaviour on the pitch. Not that this improvement is being universally adopted by all players in all leagues of course. But anyway, this fact that VAR might actually be improving the game in terms of the way it’s played, above and beyond any potential improvements to the refereeing process, is an interesting aspect, potentially in VAR’s favour, which falls completely outside the scope of the IFAB study discussed above.

But in terms of VAR’s impact on refereeing decisions, I can’t help feeling that the IFAB study was designed, executed and presented in a way that shines the best possible light on VAR’s performance.

Incidentally, if you’re puzzled by the title of this post, you need to open the link I gave above, and exercise your fluency in Spanish vernacular.