David McDonnell, a former Deputy Development Manager on the Horizon IT project with Fujitsu/ICL gave evidence during the Horizon IT inquiry on the morning of Wed 16 November.

What he had to say was devastating. In October 1998 McDonnell co-wrote a report on the Horizon EPOS system with Jan Holmes, a Fujitsu/ICL internal auditor (and a very interesting witness in the afternoon of 16 November). The Task Force report was put together to try to address the serious number errors in the Horizon project at the time. You can read it here.

The Task Force report has surfaced already during the inquiry. It was damning, and included this choice phrase:

‘Whoever wrote this code clearly has no understanding of elementary mathematics or the most basic rules of programming.’

Over three hours McDonnell gave some extraordinary evidence. You can listen to the highlights on Episode 22 of Investigating the Post Office Scandal (which includes clips from the last four witnesses), or you can watch the entire evidence session on video and read along with the full transcript, by clicking on the inquiry website here. 

McDonnell describes the EPOS programming team as like ‘the Wild West’.

When asked what he meant, he replied: ‘There were no standards in place, there were no design documents. The culture of the development team was – I wouldn’t say it was a holiday camp, but it was free format. There was no structure, no discipline; it was crazy, never seen anything like it.’

Of the code itself, he said:

‘it was so bad. It was beyond anything I’ve ever seen. Even in the 25/30 years since that project, I’ve never seen anything like that before. Some of the stuff that we found buried in the code was unbelievable. There was unreachable code… It was a mess.’

Among techies at Fujitsu/ICL, McDonnell quickly discovered the EPOS team were the ‘joke of the building’, telling the inquiry ‘everybody knew, specifically the test team who, when I spoke to those guys, they would make it very clear that the quality of code that was being delivered was to such a bad, poor level that they’re wasting their time testing it.’

The solution

McDonnell’s rather obvious solution was to get some better coders in and re-write the cash account from scratch. He was overruled by the Horizon Programme Manager, Terry Austin. McDonnell found the resistance to his solution odd, and believes the way it has been portrayed by other witnesses at the inquiry as too big a problem to realistically deal with, ‘betrayed a basic misunderstanding of how the EPOS system was built or even potentially suggests an attempt to obfuscate the issue.’

He described it using a Lego analogy:

‘if you understood that it was built out of Lego bricks, you could replace the Lego bricks one at a time starting
with the most critical, the most important, which I would argue was the cash account. Here, you could even — because it was a batch process that wasn’t part of the counter client/customer interaction, you could rewrite that as a separate module and have it running as a shadow process on the counter. You could run the cash account twice at the end of the day or whenever, as a secondary confirmation, and use the replacement module to check the validity of the first one. Once you’d proved that it worked, you could take the old one out and just continue with the new one. This was not a large task. It was not something that – I couldn’t understand why they didn’t do it, because it was such a – it’s not a small piece of work but relatively small, and you could have done it without introducing any danger to anything else on the counter.’

The CSR+ solution

McDonnell says matters reached a head when he was called into Terry Austin’s office and offered a promotion. He accepted on the condition the EPOS cash account was re-written. McDonnell says Austin became ‘frustrated’ by his insistence:

‘He wasn’t very happy with me putting a condition on that acceptance. It was clear that the cash account wasn’t going to get written. That conversation was very quickly brought to a halt, and I was ushered out of the office, and I never really spoke to Terry after that again.’

Fujitsu, in its wisdom, chose to try to fix the code, which took a year, and by November 1999, after the system had been accepted, they were getting a similar number of errors and bugs. Fujitsu got acceptance from the Post Office by agreeing to write a new bit of code, known as the CSR+ release which would monitor the cash accounting discrepancies within the system. McDonnell had been moved off the project by that state, but described the CSR+ release as a ‘big bone of contention at the time.’ The way he saw it:

‘At the end of the Task Force they were given the report that we co-authored detailing what the senior engineers, senior auditing guy, and all of the experienced people around the project were saying, detailing the problems. It’s like the captain of the ship’s been told that there’s a hole in the boat and it’s filling with water by the engineers. Instead of fixing the hole, what they did was they went away and constructed this CSR+ release, which is akin to painting a plimsoll line on the outside of the boat so that they could measure how fast it was sinking.

‘The whole context of this CSR+ release was about being able to detect discrepancies between the counter and the middle and back office, the APS systems and such, and highlight where there was a difference between the number of transactions or the balance between the two being different. That’s just building a dipstick instead of actually fixing the hole in the boat. They spent a year, an inordinate amount of time and resource, on this release instead of fixing the problem.’

The only question left unanswered at the end of McDonnell’s three hour session was why he didn’t come forward sooner. Mr McDonnell declined an interview after giving evidence.

---