Algorithm anarchy

The algorithms powering the Horizon software that Fujitsu provided for the Post Office were said to be completely reliable when it was introduced in 1999, so threatened postmaster Lee Castleton was made bankrupt when he could not pay £25,000 cash plus £321,000 in legal costs. Others were forced to sell up, moved abroad, threatened suicide, died of the stress or ended up in jail, writes Professor Brian J Ford

In reality, the algorithms for the Fujitsu Horizon system were filled with coding errors though the fact was not admitted until late last year.

Similar errors had the USS Fitzgerald collide with a huge cargo vessel in June 2017 with nine crewmen killed. New software aboard had imposed control, so nobody knew the other vessel was there. Nine weeks later, the USS John S. McCain turned into the path of a gigantic oil tanker causing 10 more sailors to die. Officers aboard the McCain found they had no control over steering and the commanding officer had declared ‘ship not under command’.

In October 2018 a Boeing 737 Max 8 dived into the Java Sea, killing 189 people. Five months later, another crashed at Bishoftu, Ethiopia, and 157 people perished. The planes had been fitted with oversized engines, and algorithms had been coded to make the planes handle as if they hadn’t. In each case the highly-trained pilots knew what to do – but those algorithms wouldn’t let them. So they perished. Boeing blamed the pilots.

Around the same time, the Viking Sky cruise ship was left rolling with heavy furniture crashing about because algorithms stopped her engines in a storm. Some 480 passengers were airlifted off, many rushed to hospital. Less dangerous (but still life-changing) was the experience of teenagers in August 2020 who had a rude awakening when their exam results were reassessed by algorithm. Many lost university places.

Algorithms rule, like the “computer says no” lament by David Walliams’s bank official Carol Beer in TV’s Little Britain

The public are assured that it is a “technical hitch” or a “computer malfunction” that causes such problems. But algorithms rule, like the “computer says no” lament by David Walliams’s bank official Carol Beer in TV’s Little Britain. Computers only say what they’re programmed to say. Algorithms are like complex knitting patterns coded by youngsters with no more experience of the real world than you could write on a box of Kleenex. I doubt there are any that are free of bugs, and there are sometimes errors that cost lives.

We have been deluded into calling the new technology artificial intelligence, though AI is actually of unfathomable stupidity. It is heuristic digital automation, and the latest step in automating routine tasks that began in 1771, when Richard Arkwright invented the automated spinning mill. Spinners, then weavers, were soon out of work. The flag bearers who used to precede motor cars, and the shit-shovellers who kept London streets clear of excessive horse manure, all bemoaned the loss of their jobs, just as housemaids were superseded by washing machines and vacuum cleaners, and metal type was swept away by the printer on your desk.   

Nobody seems to know just how much money has been lost from the Post Office through this latest debacle, or where it went. If you cause harm to people you are culpable and could end in jail. That’s already true of people who kill with knives, or design a dangerous device. It must also apply to software. Writing an algorithm is not just a handy source of salary. It’s a matter of life and death.