Ecce Chambers

I attended the morning session of Anne Chambers’ evidence at the Post Office Horizon IT Inquiry yesterday. Chambers is a former senior Fujitsu engineer, currently under police investigation. She is of signal importance to this scandal on the basis that she gave evidence in the Post Office’s High Court civil prosecution of Lee Castleton, the former East Bridlington Subpostmaster whose name consists the first two words of Computer Weekly’s seminal first investigation into this scandal in 2009.

Chambers has never spoken in public about her role before. At the inquiry she seemed the very picture of a chatty, bright and helpful woman. She has a degree in statistics and pure maths and learned everything she knows about computers on the job at Fujitsu.

Whilst the Post Office was prosecuting people over a 15 year period for discrepancies in their Horizon accounts, Chambers was on the other side of the fence, working for Fujitsu’s SSC. Most people at this inquiry seem to think SSC stood for System Support Centre or Service Support Centre. Either way, it was Fujitsu’s top (fourth) line Horizon support function for Subpostmasters, Fujitsu colleagues and Post Office staff who had encountered problems with the Horizon system and needed them properly identified, categorised and fixed.

The Last Word

Before getting fourth line support involved, Subpostmasters with Horizon problems were required to convince first, second and third line Horizon support (and sometimes the Post Office’s National Business Support Centre (NBSC) helpline operators) that their Horizon error was a real IT error and not a mistake (or nefarious activity) by them, their customers or their colleagues. Or a business operation error by the Post Office, someone at the Post Office of one of its business partners.

During the inquiry yesterday afternoon (which I watched on YouTube this morning) we were taken to a document, authored by Anne Chambers, which made it clear that only genuine software bugs should be referred to fourth line support. Pressed on how Subpostmasters, Post Office staff and colleagues were supposed to know an inexplicable error was a software problem, Chambers argued that they should “believe” it to be a software error, having made sure it wasn’t anything else.

Chambers (and her fourth line colleagues) were therefore the last word on whether or not there was a bug or problem with the Horizon system.

A knowledgeable and reliable witness

In her evidence to the High Court during the Post Office’s claim against Lee Castleton for the sum of £22,963.34 (a sum they were spending hundreds of thousands of pounds on legal fees to recover, despite knowing they had no hope of getting the money, as Lee had no assets), Chambers was an expert witness.

Her testimony was summarised by the judge:

“Mrs Chambers examined the questions raised and concluded that there was no evidence whatsoever of any problem with the system. She was unable to identify any basis upon which the Horizon system could have caused the losses. Mr Castleton cross-examined her about complaints from another branch, which he did not identify. She immediately recognized the branch with confidence as being a branch at Callender Square in Falkirk. The problem at Callender Square had, she said, arisen from an error in the Horizon system, but there was no evidence of such a thing at Mr Castleton’s branch. I found Mrs Chambers to be a clear, knowledgeable and reliable witness, and I accept her evidence.”

par 23, Post Office Ltd v Castleton [2007] EWHC 5 (QB) (22 January 2007)

On the basis of Chambers’ evidence, the judge ruled:

the logic of the system is correct, the conclusion is inescapable that the Horizon system was working properly in all material respects, and that the shortfall of £22,963.34 is real, not illusory.

par 11, Post Office Ltd v Castleton [2007] EWHC 5 (QB) (22 January 2007)

Lee lost the case was bankrupted. He and his family have lived in penury for most of the last two decades, whilst Chambers continued working at Fujitsu and is now enjoying a comfortable retirement.

Ms Chambers will not be required to give evidence about Lee’s case this week, but on 2 May we got an insight into how she went about investigating serious errors.

PEAKing

Chambers was taken to an Horizon error log (known as a PEAK) recording some extraordinarily wayward balances at a Subpostmaster’s branch in 2013. By this stage Chambers had already seen Castleton taken apart at the High Court and was aware that Horizon evidence was being used in what she told the inquiry she understood to be a “small number” of criminal prosecutions of Subpostmasters.

By this stage of her testimony Jason Beer KC (counsel to the Inquiry) had established that Chambers understood discrepancies could arise through incorrect declarations, recorded transactions not matching what happened at a branch, withdrawn products, known system problems and unknown system problems.

The PEAK was dated 12 November 2013. It recorded a Subpostmaster calling up to report “doing cash declarations every now and again” which occasionally lead to a “major loss… He states that he loses £2,000 then jumps suddenly to £4,000, one point they lost £8,000 and is always losing money. [Postmaster] stated he has 3 post offices, only happens at this site.”

The Postmaster reported that the problem at his one specific site had been been happening intermittently all year. He told NBSC who sent experienced trainers out to ensure the Subpostmaster wasn’t doing something wrong. The trainers concluded the Subpostmaster was doing nothing wrong and witnessed the error happening. The auditors, who were incentivised by the Post Office to suspend Subpostmasters with discrepancies (another story), came out. They concluded the Subpostmaster was doing nothing wrong and they witnessed the error happening.

Chambers went to work. She delved into the system and reported: “have checked the system figures… and can confirm that all the variants reported since then have been calculated correctly. There are no known issues that would result in the variance being incorrect.”

Chambers closed the ticket with a definitive: “No fault in product”.

The cause of the defect was assigned to “User” – that is, the Subpostmaster.

When Beer asked why, Chambers replied: “Because I was rather frustrated by not – by feeling that I couldn’t fully get to the bottom of it. But there was no evidence for it being a system error.”

Lee Castleton in 2021

Absence of evidence…

The Inquiry chair and retired judge Sir Wyn Williams wondered if it might have something to do with the class of error Chambers had already acknowledged could be designated – unknown system problems.

Chambers conceded: “something was obviously wrong, in that the branch obviously were getting these discrepancies that they weren’t expecting, but all I could see on my side was that they were apparently declaring these differing amounts, and I certainly didn’t know of any system errors that would cause that to happen, or that would take what they were declaring and not record it correctly…. so I felt, on balance, there was just no evidence of a system error.”

No evidence. Williams pointed out that it surely was unlikely to be a user error if both trainers and auditors had recorded the Subpostmaster as inputting information correctly. Chambers replied:

“Well, yeah, I… yes, I don’t know why… I’m not happy with this one. But I still stand by there being no indication of a system error and the numbers that they were recording just didn’t make a lot of sense.”

Chambers admitted to the Inquiry that despite being told the discrepancies had occurred on various occasions at the branch throughout the year, she only looked at the system information behind the most recent discrepancy. She also didn’t check to see if something similar had been reported by any other branch. Her “investigation”, from assignation to conclusion, took one hour sixteen minutes.

Chambers conceded that she didn’t know if it was an unknown system error was affecting the branch, and seemed to accept that her stark conclusion in 2013: “No fault in product” and “user” error, was misleading. She added:

“I don’t think I handled it terribly well. I was frustrated by it and I think that shows… because, you know, it really looked like there was a genuine problem… [but]… There was no sign of it.”

In summary, ten years ago, Anne Chambers appears not to have considered the possibility of unknown system problems, or prompted an investigation more advanced than what she could achieve within the limits of her knowledge and ability. Instead, after a quick squizz at a limited dataset she concluded there was no obvious problem, which meant there was no problem and the cause of the discrepancy was the Subpostmaster.

When criminal and civil prosecutions hang on this sort of evidence, there should be red flags and alarm bells going off in all directions.

It’s not known what happened to the Subpostmaster Chambers reviewed in November2013. Due to the two teams of Post Office staff affirming the Subpostmaster was acting correctly s/he is unlikely to have been held liable for the random discrepancies.

Lee Castleton had no such luck. In 2004, he begged the Post Office to send someone out to have a look at what was going wrong with his machine at Marine Drive in East Bridlington. The Post Office did not bother. They told him there was nothing wrong with the system and he was liable. When the auditors came, Lee thought they would help him get to the bottom of his IT problems. All they did was calculate his discrepancy, confirm he was liable and suspend him without pay. There was no investigation into the problems at the Marine Drive branch in East Bridlington. Instead, when it came to court, Anne Chambers ran the numbers back at SSC and told the High Court that the Horizon system was working properly in all material respects. His Honour Judge Richard Havery QC seems to have had no reason to disbelieve her.

Postscript

The police investigation into Anne Chambers (and other potential criminal activity at Fujitsu and the Post Office) is called Operation Olympos. It has been running since January 2020 and has so far seen no arrests. When I asked, under the Freedom of Information Act, how much Operation Olympos had cost, the police refused to tell me, citing the difficulties of adding up lots of numbers. Operation Olympos only got a cost code to itself in March 2023.


My work on the Post Office Horizon IT inquiry is crowdfunded. If you’d like to contribute, please click on the widget you should be seeing to the right of this text (or below if you’re reading it on a mobile). To find out more before donating, please go to my tip jar web page. All contributors will be added to the ‘secret’ email newsletter, which offers irregular, and at times, irreverent insight into the machinations of the inquiry and the wider scandal. If you’d like to buy my book The Great Post Office Scandal, I would be thrilled – it’s available from all good outlets.

25 responses to “Ecce Chambers”

  1. Mrs Chambers evidence was quite unusual in that she appeared to be very good at her job, with excellent recall even now
    In the later evidence she had Ms Page and Mr Henry asking questions yet she seemed to answer them effortlessly
    She seemed to have done a professional job even though the end result was a miscarriage of justice,though that seems not to have been her fault

    1. Robert Huskinson avatar
      Robert Huskinson

      It should have been obvious to the software support teams that there were day to day issues with the quality and robustness of the software code and also the degree of training that PO staff had been given.
      Believe she was a Graduate in Maths and Economics from a Uk University, so hopefully a fair degree of Intelligence and Intellect. Just because as an individual you cannot prove softwhere error does not mean I was not the cause of the discrepancy. Lee Castletown was financially crucified and her evidence was partially used to do it. In my view Fujitsu did not volunteer any of the staff who coded Horizon and probably new just how vulnerable it was to bugs and errors caused by the unexpected, such as power outages or “comms” issues.
      I had 15+ years implementing and testing advanced Bespoke Financial Software systems and have had plenty of experience with both “user error” and basic reports not “adding up”.
      Think this comes down to the integrity of the individual in a courtroom versus how much do I want to please my employer. Personally she was NOT suitably qualified to say 100% that the errors were NOT system generated.

  2. One of the things which most impressed me about the evidence given by Anne Chambers was her ability to remember details of error logs and required code fixes from nigh on 20 years ago. Compare that to the evidence from many of the POL witnesses who are it seems unable to remember anything, or in some cases (PV) never knew anything.

    It will be interesting to see if Gareth Jenkins comes across anything like as well in his evidence, he may surprise us. One thing i did take from Ms Chambers was that having been unable to find a cause for branch discrepancies she initially had no idea that the postmaster would themselves be treated as liable for the missing money; she also assumed that Post Office itself via the help desk or network staff would then provide support and investigate further, rather than arriving in jackboots to cart off the hapless victim.

    Her internal Fujitsu note on lessons to be learned after her first experience as a witness was insightful and on the money. In summary she came across as competent and helpful , certainly compared to the gross incompetence on show from the likes of POL witnesses such as Singh, Ismay and the majority of the investigators.

    1. Malcolm Davison avatar
      Malcolm Davison

      If a computer expert witness confesses that they were unable to replicate or find the cause of a problem – they can no longer be defined as an expert.

      If such a species exists – they would be have to have been intimately involved in the coding over a long period of time, and be able to recall the design, the upgrades and resolution of past issues. They can only be a senior member of the software team that wrote it. They must be able to simply explain highly complex coding issues to the average Jane or Joe Bloggs, and they must also be able to withstand the rigours of cross-examination (and tedium of the courtroom!).

      So that limits the choice of ‘experts’ dramatically. By definition they are totally partisan – far from independent – and in the public arena they will be obliged to be ‘waving the company flag’ and defensive of their software and brand.

      Independent expert witnesses for bespoke computer software IMHO simply don’t exist.

  3. Chris Palmer avatar

    Sorry to quibble, but SSC was third line support. GDC were fourth line.

  4. I’ve watched her session at the Inquiry and she comes over as diligent, honest, sympathetic, and professional. Her conclusion that she could identify no system bug that could have caused the error was entirely reasonable in the circumstances, and it is outrageous (a) that a court should use that as sufficient evidence to convict the postmaster, and (b) that anyone should accuse her of being dishonest.

  5. Anne Chambers was not “an expert witness” She was supposed to be a “witness of Fact”, whatever that is supposed to be legally. As stated in a previous reply she was level 3 not 4 , which I presume would be rh application development team. From watching her evidence it seems that she was put into the unenviable position of having to testify on the basis that she dealt with, was assigned to, the initial problem reported. As far as I heard, she did all that she could to investigate what was put before her and actually asked the Fujitsu Help Desk to liaise with it’s POL counterpart to get the Post Office to look at how the branch operated. She checked figures in log files, could find no discrepancies, and later in the current phase of investigations worked overnight as did the solicitor to see what the updated files made available by Fujitsu, I presume show. She wrote a follow up report at the time outlining her concerns which seem to have disappeared into Fujitsu upper management and got a “thanks for your thoughts” pat on the head in response. I fully sympathise with her as what her job was, was to check if she could find a cause for the problem reported. Having exhausted her options and found nothing in her investigation, what do you suggest she should have done? Said I can find nothing in my investigation so it must be a fatal error/bug in Horizon so escalate to level 4? On what basis could she justify that? And just remember that every application I encounter in my phone or PC has regular software updates . have a look and see what they are for.. mostly to repair something or roll out something new in functionality or indeed roll out a new version which in turn has software updates to correct discovered “bugs”.

  6. Just been reading the session of Anne Chambers at the Horizon enquiry and
    felt I had to comment.

    I’ve had over 35 years of working in IT and quite a lot of it was investigating /
    solving problems, to write the problem off as entirely down to fraud after only an
    hour and a quarter of investigation is ridiculous.

    Despite the PO trainers concluding that the postmaster did nothing wrong and actually
    seeing the problem happening, also verified by the Post Office auditors themselves she put
    this down to fraud without even considering that it could be an unknown bug.

    If Chambers is an example of the best of the Fujitsu investigators they really need to
    get some better staff – surely they’ve got numerous test systems in place where they can
    replay branch processing to try and replicate the errors ?

    Not sure if she’s just totally inept or wilfully negligent.

    1. She didn’t say it was fraud. She said the only explanation she could find was user error. Essentially she was saying “cause unknown”.

      1. Michael,

        Technically you’re correct, she didn’t mention fraud.

        However, she failed to mention that both Post Office trainers and auditors had been in branch,
        seen the error happen and admit that there was no user error. She did minimal investigation,
        76 minutes, and didn’t even look at the application code or request that someone else investigate further.

        She didn’t testify that it was ’cause unknown’, she did state that it was user error, despite all evidence
        to the contrary.

        As she’s the ‘expert witness’ the court blamed the sub postmaster.

        1. She did hand the problem over to someone else, and they could have done a more independent branch investigation
          She also did a much more thorough investigation later on
          She was part of a team, and the decisions that led others to mis-use her work are for them
          I think she will be assessed to have been an honest and helpful witness, but we will see

  7. SSC is 3rd line support as can be seen in paragraph 4.1 of FUJ00120446 where it says that SSC has responsibilities to 1st, 2nd and 4th line support.

  8. I may be an amateur in this field, but I wonder why Fujitsu did not run an investigative simulation of a normal sub post office operation using standard software and a typical number and spectrum of business processes – had there then been financial discrepancies then any software errors within those processes could have been identified?

    1. We have heard very little in the inquiry about how the system was tested, but we have heard a little about some of the bugs found in the system, and they are typically of the kind (eg rare timing glitches) that tend to arise too rarely to show up in such tests.

  9. Julian Le Vay avatar

    I keep thinking: this story is so absolutely awful, such wickedness and complacency and secrecy had been clearly revealed and documented, soon, surely, someone at the top of the PO or government will stand in front of us and say publicly that that this is among the worst things that the State has ever done to its servants, that everyone involved must be prosecuted, imprisoned, fined, and banned from any office, that the victims must be publicly exonerated and liberally compensated, where necessary, their widows or widowers or children, that across the public sector, reforms must be made to stop the like ever being repeated.

    I keep thinking that….

  10. Never have I encountered a company where they even have 4th line support. That for me would be the development team. All the people on the SSC they’ve questioned did not even have access to the source code and seem more like what should be 2nd line. i.e look at data and try to find a reproducible bug.
    A good company will always blame their own system and think it’s crap and buggy which is so bizarre they’ve not acted the same.

  11. This gets more and more farcical as the days, months and years pass. I’ve worked in IT for 33 years and never worked on a system which has not been full of bugs (maybe the connection is me? ). Common practice when investigating a problem is to try and narrow down the use case in order to recreate the problem. If you can’t do that, you don’t say there isn’t a problem, you say you cannot recreate it. Its like the needle in the haystack; just because you can’t find it, doesn’t mean its not there.
    Thinking about these sub-post masters and they way they have been treated makes me extremely angry; I really can’t imagine how they must feel

    1. Given what you’ve just written about systems usually being full of bugs, why on earth has it been the case since the 1990s that courts assume that any computer evidence they’re given is automatically assumed to be correct?

      1. I completely agree, but what would any judge unqualified in the field in question do? Rely on “expert” witnesses of course as there’s no real alternative to that. And that still leads to false convictions. Computers are a great example of “you won’t understand this m’lud, so just take our word for it”. I worked in IT for many years, and know how much of a “black art” it can seem to outsiders, and judges would be no different to most non- IT people in that respect. The utterly outrageous thing about this whole affair is that there should have been expert witnesses on both sides for fair trials. Without this the judge was relying solely on one (infinitely more powerful) side’s expert evidence, and disregarding the defence of the sub post masters. The fact that so many judges must have known that this was unfair, but went ahead and convicted the victims anyway, is a damning indictment of the UK legal system.

        1. In any other contractual situation the customer would have defended the user against the supplier rather than defending the supplier against the user.

        2. > there should have been expert witnesses on both sides for fair trials.

          An expert witness is a servant of the court, not of either the prosecution or the defence. They aren’t allowed to have a ‘side’. So in general, for one field of expertise, you should only need one expert witness.

          One side might call a second expert, if they disagree with the first expert’s testimony. But it would be hard to find an expert in Fujitsu software, who doesn’t work for Fujitsu.

      2. Andrew Martin avatar

        Contrary to its stated aim of codifying regulation of computer evidence in court – the criminal law review decided to exempt computer evidence from the rigour (and codified) requirements of the Police and Criminal Evidence Act. Note that Home Secretary JACK STRAW would have been well aware of the computerisation plan for the Post Office as would have his Senior Civil Servant (Cabinet Office), soon to be Post Office Chairperson, WIFE , Alice Perkins (almost countless prosecutions and litigations under her domain). Quite a neat little package do you not think??

    2. I agree completely. Microsoft have the most sophisticated software development teams and their products have updates (bug fixes) almost daily even after years of bedding in.
      It beggars belief that Fujitsu didn’t build a test Post Office and run the same sales through and do comprehensive testing. Once the anomalies in the shop were witnessed the whole system should have been analysed and debugged. The PO could always run a manual/paper transaction log as a backup and then any shortfalls could be compared to Horizon.
      Horizon isn’t exactly rocket science – sell some stamps – put money in cash drawer, pay out a pension – take money out of drawer. End of day add up the ins and outs. It’s hardly Oracle. If it cost a billion pounds (plus annual costs) it should have been state of the art and bombproof, not ‘clunky’.

    3. Martin,

      I’d certainly accept that some bugs are inevitable in any complex IT system but your
      statement that “I’ve worked in IT for 33 years and never worked on a system
      which has not been full of bugs” leaves me amazed.

      When I first started in IT as a Trainee Programmer back in 1975 part of a standard
      program specification was a set of unit / program test criteria that had to be met before
      the program underwent further testing. It was the programmer’s role to set up test data
      and ensure that the expected results were produced.

      Once unit / program testing was complete it would move on to system testing /user
      testing then parallel running – all before it could go live.

      One of the bugs mentioned in Horizon was an basic coding error in a formula resulting
      in the wrong value being posted – how this can have undergone even basic program testing
      is beyond me.

      Tony Blair had misgivings about Horizon in 1997, before implementation, but was persuaded
      by a letter from Peter Mandelson’s which stated that “the basic development work has been
      thoroughly evaluated by independent experts who have pronounced it viable, robust
      and of a design which should accommodate future technological developments”

      Now that’s a report I’d love to see although I suspect it’s now in the shredder and nobody
      can remember who produced it.

  12. I have followed this for years Nick, and whilst initially I believed that the PO and Fujitsu were solely to blame for this monumental miscarriage of justice, I have come to conclusion that our judiciary played a leading role in this fiasco.

    Time after time over the years, your reporting has highlighted the complete lack of curiosity by both judges and prosecutors about the large numbers of people being prosecuted and the fact that, pretty well all the defendants described more or less identical problems.

    It would appear that the judiciary worked on the basis that the PO could not be to blame just because it was the PO and that every time the sub postmasters were lying, cheating and committing fraud on a large scale.

    It would therefore seem that after this enquiry is over (will that ever happen?) we then need a Roytal Commission into the actions of the judiciary involved in the Horizon scandal, members should be suject to minute scrutiny and offenders struck off and subject to criminal prosecution. I do not detect ice beginning to form on the edges of Hell so I guess that that will never happen :-).

    Thank you for your continued efforts, you and your colleagues, along with Private Eye and Computer Weekly have restored some of my faith in journalism!

Leave a Reply

Your email address will not be published. Required fields are marked *

Archives

  • 2024
  • 2023
  • 2022
  • 2021


Subscribe For Latest Blog Updates

Tags

Alan Bates alice perkins Alwen Lyons Andrew Winn Andy Dunks Andy Parsons Bates v Post Office BBC Bonusgate CCRC Chris Aujard Clarke Advice False Accounts Fujitsu Gareth Jenkins Grabiner HCAB Horizon Inquiry Interim Report Janet Skinner Jarnail Singh Lee Castleton Lord Arbuthnot Mark Davies Neuberger Nicki Arch Nick Read Noel Thomas Outcasts Creative Paula Vennells Paul Marshall Post Office Rebecca Thomson Receipts and Payments mismatch bug Rob Wilson Rod Ismay Rodric Williams Second Sight Seema Misra ShEx Simon Clarke Susan Crichton Tracy Felstead UKGI

Categories