Senate committee on ABS #CensusFail still points to basic failures on IBM's part

Blame and recriminations were flying around today at the Senate committee’s inquiry into the ABS’s attempt at a “digital first”, online Australian census. Appearing before the committee, ABS still held IBM responsible for the outage that resulted in the online census being unavailable for 40 hours. Although IBM apologised to the Australian public for the outage and offering to pay the federal government for costs incurred in the cleanup, it still insisted that the blame lay with Nextgen and Vocus, the network providers on the project. NextGen in turn countered that IBM was offered protection for DDoS which IBM had declined.

Perhaps the most sensible commentary came from Alastair MacGibbon, cyber security advisor to the Prime Minster. Ultimately, he lay the blame with IBM who was contracted to provide a service, which on the day, it didn’t deliver. He added however that the ABS should have done more due diligence and actually ensured that IBM had done what it said it had.

The testimony did not result in clarifying exactly what happened on the day. The submissions to the inquiry had already detailed the time-line of the 4 DDoS attacks during the day. There was a dispute about the magnitude of the 4th attack which IBM claimed was of “significant size and had the effect of causing the site to become unresponsive and unavailable to the public”. Vocus disputed this in its submission claiming the attack peaked at about 0.5 Gbps which is the size of the majority of DDoS attacks commonly seen but significantly below the peak sizes of 500 Gbps or even 1,000 Gbps reportedly seen recently.

Vocus maintained that it was IBM’s decision to reboot their router that caused the website to be unavailable. IBM admitted in its statement to the committee that it had never tested physically switching the router off and then on again to see if it came back up with its configuration properly loaded. In the event, it didn’t. IBM had relied solely on “simulations” of the router to test the configuration.

Vocus has disputed IBM’s claim that the traffic originating from Singapore was also the sole source of DDoS traffic. Basically neither Vocus, nor Nextgen though very much of IBM’s “Island Australia” plan, a plan that IBM only told Nextgen about 6 days before the census went live. IBM had signed off on the testing of Nextgen’s implementation of Island Australia but it eventually turned out that the testing was inadequate.

For the ABS’s part, Chief Statistician David Kalisch admitted that the ABS had not communicated effectively with the public. This was an understatement. In fact, the entire anxiety and public reaction to the census was largely a factor of the poor communication about the data retention proposals of names and addresses plus the expectation that everyone needed to complete the census on a particular night.

The Canadian online census, held as an example of how to run operations online, actually crashed when Canadians rushed to complete the census online. However, it was brought back up and the public took it in its stride. It is entirely possible that handled differently, the public relations disaster that has been the entire ABS could have been avoided.

What seems clear from the submissions and the commentary presented to the committee is that the ABS decided to use IBM because of a long association with the company and not through an open tender process for the online census project. The site itself was also able to cope with the load of users accessing the site and it didn’t crash on the night through normal use.

What wasn’t discussed was how the ABS plans to use identifying information like name, sex and date of birth to link the census data to other data sets. Although Kalisch had at one point implied that the semi-anonymous statistical linkage key was going to be used, in the submission to the senate inquiry, the ABS denied this was the case and claimed that a totally anonymous cryptographic key was going to be used instead. No further details were requested at the committee meeting today.

The development, deployment and testing of the online census should have been subject to an independent review arranged by an agency unrelated to the ABS. If that had been done, it is highly likely that the inadequate preparation for the inevitable DDoS would have been highlighted. The ABS stated that it had learned lessons from the mistakes of this census and that it would be better prepared for the next one. The Australian public will need to be convinced.