Daily BulletinDaily Bulletin

The Conversation

  • Written by Robert Merkel, Lecturer in Software Engineering, Monash University

Many Australian Tax Office IT systems have been unavailable for days after a major fault, apparently caused by a problem with a large-scale storage server.

The ATO’s online systems, including its public website and portals for taxation agents, were down for several days. At the time of writing, the ATO reports that most services are now operational but may experience slowdowns.

There were also reports that up to one petabyte of data was affected by the fault. The ATO has reported that no taxpayer data have been lost, although it is unclear as to whether any internal data have been lost.

Outage in a SAN

According to the ATO and media reports, the system outage was caused by a failure in a 3PAR StoreServe storage area network (SAN) made by Hewlett Packard Enterprise (HPE).

These devices contain racks full of hard disks and/or solid-state storage devices to store data on a gargantuan scale, and fast network interfaces to provide that data to the various “application servers” that provide the ATO’s online systems.

The two units purchased by the ATO were reportedly capable of storing up to a petabyte – that’s 1,000 terabytes or 1 million gigabytes – of data each. They would have cost hundreds of thousands of dollars.

While these devices are expensive, they allow IT staff to allocate storage efficiently and flexibly to where it is needed, and thus (in theory) can improve reliability.

image Even Hewlett Packard Enterprise’s state of the art storage system was vulnerable to data corruption. Hewlett Packard Enterprise

Multiple levels of redundancy, made redundant

Entrusting so much of the IT operations of a large organisation like the ATO to a single storage server requires a high degree of confidence that it will function reliably. As such, a number of levels of redundancy are incorporated into this kind of storage system.

As a first protection against a failure of a single disk (or solid-state storage device), data are “mirrored” across multiple physical disks. If monitoring systems detect a failure, operations can fall back on the mirrored data.

The faulty disk can be replaced and the full mirror restored, all without interrupting user operations. High-end systems such as these also incorporate redundancy into their controller electronics.

However, if a major hardware failure occurs, such as a power failure that is not covered by a backup power supply, many such systems have a second level of redundancy. The entire contents of the SAN is “mirrored” to a second system, often in another physical location, and systems switch over to the backup automatically.

According to iTnews, all of this redundancy was made moot by the nature of the problem: corrupted data were being written to the SAN for some reason, and this corrupted data were then mirrored to the backup SAN.

In this situation, all the redundancy within and between the SANs does not help, as the bad data were replicated across the entire system. This is why keeping traditional backup snapshots – copies of data as it previously existed in the system – is so important, regardless of any amount of mirroring.

The ATO appears to have comprehensive backups of the stored data; however, restoring all of it and returning the SANs to an operational configuration has had to be done manually. It is not surprising that this has taken several days to complete.

Assessing the ATO’s response

While it is tempting to pile on to another large-scale government IT failure, a fair assessment should take into consideration the nature of the failure and the ATO’s response.

Firstly, it appears that the ATO heeded one of the key lessons from the Census website meltdown and communicated what was going on to the public effectively. It responded to the failures by providing informative updates on social media and more comprehensive information on a functioning part of its website.

Secondly, it appears that its backup strategy was sufficient to get all systems back up and running without data loss, despite a nearly worst-case failure in their primary storage system.

If its incident response can be criticised, it may have been able to restore services much faster if more of that process had been automated. However, this appears to be a highly unusual incident.

Restoring one set of application data due to corruption caused by the application itself is a relatively common situation. Restoring many different sets of data because of an apparent bug in the storage server is extremely rare.

Furthermore, while few people ever see them, SANs like this are very common devices in data centres. They provide a generic low-level storage service and are expected to provide it highly reliably.

Indeed, HPE markets its enterprise storage systems with a “99.9999% uptime guarantee”, which requires that a device is non-operational for no more than 30 seconds per year.

Over the past few days, the IT staff at the Australian Tax Office have probably had a few sleepless nights. It’s likely that engineers at HPE will have a few more trying to get to the bottom of why their enterprise storage system seems to have failed so comprehensively.

Authors: Robert Merkel, Lecturer in Software Engineering, Monash University

Read more http://theconversation.com/server-down-what-caused-the-ato-systems-to-crash-70396

Australia's smallest fish among 22 at risk of extinction within two decades

arrow_forward

A 3-decade 'moving picture' of young Australians' study, work and life, thanks to LSAY

arrow_forward

In COVID's shadow, global terrorism goes quiet. But we have seen this before, and should be wary

arrow_forward

The Conversation
INTERWEBS DIGITAL AGENCY

Politics

Did BLM Really Change the US Police Work?

The Black Lives Matter (BLM) movement has proven that the power of the state rests in the hands of the people it governs. Following the death of 46-year-old black American George Floyd in a case of ...

a Guest Writer - avatar a Guest Writer

Scott Morrison: the right man at the right time

Australia is not at war with another nation or ideology in August 2020 but the nation is in conflict. There are serious threats from China and there are many challenges flowing from the pandemic tha...

Greg Rogers - avatar Greg Rogers

Prime Minister National Cabinet Statement

The National Cabinet met today to discuss Australia’s COVID-19 response, the Victoria outbreak, easing restrictions, helping Australians prepare to go back to work in a COVID-safe environment an...

Scott Morrison - avatar Scott Morrison

Business News

Link Building Secrets - Comprehensive Guide

Link building has proven to be an effective approach when it comes to promoting your online website. Let's analyze the topic of developing an effective link building strategy for site promotion ...

Julia Smith - avatar Julia Smith

What to Expect from Your NDIS Verification & Certification Audit

The National Disability Insurance Agency administers NDIS (National Disability Insurance Scheme) in Australia. The NDIS Quality and Safeguards Commission governs it. As a welfare support scheme of...

Sarah Williams - avatar Sarah Williams

Why You May Need A Tower Scaffold Hire

When constructing a building, or even a multilevel structure, you must use a tower scaffold to get you into position. What is unique about this type of scaffolding is that you can build it highe...

News Company - avatar News Company



News Company Media Core

Content & Technology Connecting Global Audiences

More Information - Less Opinion