Root of census failures say badly done IBM and ABS. Still down for some.

Despite being told that the site is now operational, the ABS census site is still not accessible for many. The frustration felt by the public over the ABS’s failed census night has been exacerbated by extremely poor communication about highly technical matters.

Everything about the project suggested poor planning and even poorer execution. In the end, despite speculation of hackers and Distributed Denial of Service (DDoS) attacks, the servers were switched off because the ABS and IBM made a bad decision based on false information.

Patrick Gray has reported that according to his sources, IBM and the ABS initially turned down the offer of comprehensive DDoS protection from their network supplier NextGen Networks. Instead, they asked NextGen to simply block all traffic from outside Australia when they were subjected to the first DDoS attack that occurred early on the Tuesday morning.

The system was subsequently hit with an attack from within Australia which caused their firewall to stop working effectively causing it to need a restart. There were initially two firewalls, one serving as the backup for the other. However, after the restart, one of them became inoperative.

At the same time, IBM’s monitoring software falsely interpreted an alarm from the servers that suggested that data was in the process of being sent out to a foreign destination. This was interpreted as a hack of the servers which then caused them to shut down the servers.

This explains the comments by the ABS’s David Kalisch who said initially:

“We have steps in place to counter attacks, [but] this one, there was one breach that did actually get through via a third party … and believe that we’ve plugged that gap,”

The Australian Signals Directorate was called in to investigate because of the suspected breach and even though it proved to be false, still had to complete an investigation.

The question may be asked why IBM and the ABS turned down the offer from NextGen to provide DDoS protection. From IBM’s perspective, this may have been because they believed that the DDoS protection provided by its hosting service would be sufficient.

The eCensus system was built by IBM and hosted on servers run by a wholly-owned subsidiary SoftLayer. The data centre used is likely to have been in NSW at Baulkum Hills.

The ABS is just one of many organisations machines in this shared data centre. This means that the network protection provided by SoftLayer is essentially aimed at protecting the data centre as a whole and not specific individual clients. In fact, this is something that SoftLayer is at pains to stress.

When this protection was obviously not succeeding in preventing DDoS attacks from affecting the service, NextGen was called in.

The net effect of this however was that the ABS and IBM were running the application without specific and effective DDoS protection and they it appears that they never tested the setup under such an attack. The failure of the firewall is evidence of that.

After restoring the eCensus application by Thursday afternoon, the site was still not available for many users. This was because when users try to access the name of the site http://www.census.abs.gov.au, the DNS responsible for translating this address into an IP address was timing out. Again, it is possible that DNS lookups for the ABS are being deliberately blocked if they appear to be from outside Australia, or that the DNS service for this address is being attacked itself by a DNS DDoS attack.

The Australian Census Disaster will become a case study in how not to run an IT project and represents another blow to IBM’s reputation. Worse still, it has now poisoned the whole concept of eGovernment because it will be hard to rebuild the trust of the public that the Australian Government is capable of running any project online. Even the idea of putting NAPLAN testing online is now being questioned

Ultimately, it is clear that all online applications should not be tarred with the same brush. There are millions of sites and applications that function successfully online despite constant attacks and high loads from millions of users. Companies like Apple, Google, Amazon and Facebook prove this every day. Perhaps they, rather than IBM should be the ones to help in the next project?