Business Blockbuster - Helpng business to be more visible on Google

Daily Bulletin

Server down: what caused the ATO systems to crash

  • Written by: Robert Merkel, Lecturer in Software Engineering, Monash University

Many Australian Tax Office IT systems have been unavailable for days after a major fault, apparently caused by a problem with a large-scale storage server.

The ATO’s online systems, including its public website and portals for taxation agents, were down for several days. At the time of writing, the ATO reports that most services are now operational but may experience slowdowns.

There were also reports that up to one petabyte of data was affected by the fault. The ATO has reported that no taxpayer data have been lost, although it is unclear as to whether any internal data have been lost.

Outage in a SAN

According to the ATO and media reports, the system outage was caused by a failure in a 3PAR StoreServe storage area network (SAN) made by Hewlett Packard Enterprise (HPE).

These devices contain racks full of hard disks and/or solid-state storage devices to store data on a gargantuan scale, and fast network interfaces to provide that data to the various “application servers” that provide the ATO’s online systems.

The two units purchased by the ATO were reportedly capable of storing up to a petabyte – that’s 1,000 terabytes or 1 million gigabytes – of data each. They would have cost hundreds of thousands of dollars.

While these devices are expensive, they allow IT staff to allocate storage efficiently and flexibly to where it is needed, and thus (in theory) can improve reliability.

image Even Hewlett Packard Enterprise’s state of the art storage system was vulnerable to data corruption. Hewlett Packard Enterprise

Multiple levels of redundancy, made redundant

Entrusting so much of the IT operations of a large organisation like the ATO to a single storage server requires a high degree of confidence that it will function reliably. As such, a number of levels of redundancy are incorporated into this kind of storage system.

As a first protection against a failure of a single disk (or solid-state storage device), data are “mirrored” across multiple physical disks. If monitoring systems detect a failure, operations can fall back on the mirrored data.

The faulty disk can be replaced and the full mirror restored, all without interrupting user operations. High-end systems such as these also incorporate redundancy into their controller electronics.

However, if a major hardware failure occurs, such as a power failure that is not covered by a backup power supply, many such systems have a second level of redundancy. The entire contents of the SAN is “mirrored” to a second system, often in another physical location, and systems switch over to the backup automatically.

According to iTnews, all of this redundancy was made moot by the nature of the problem: corrupted data were being written to the SAN for some reason, and this corrupted data were then mirrored to the backup SAN.

In this situation, all the redundancy within and between the SANs does not help, as the bad data were replicated across the entire system. This is why keeping traditional backup snapshots – copies of data as it previously existed in the system – is so important, regardless of any amount of mirroring.

The ATO appears to have comprehensive backups of the stored data; however, restoring all of it and returning the SANs to an operational configuration has had to be done manually. It is not surprising that this has taken several days to complete.

Assessing the ATO’s response

While it is tempting to pile on to another large-scale government IT failure, a fair assessment should take into consideration the nature of the failure and the ATO’s response.

Firstly, it appears that the ATO heeded one of the key lessons from the Census website meltdown and communicated what was going on to the public effectively. It responded to the failures by providing informative updates on social media and more comprehensive information on a functioning part of its website.

Secondly, it appears that its backup strategy was sufficient to get all systems back up and running without data loss, despite a nearly worst-case failure in their primary storage system.

If its incident response can be criticised, it may have been able to restore services much faster if more of that process had been automated. However, this appears to be a highly unusual incident.

Restoring one set of application data due to corruption caused by the application itself is a relatively common situation. Restoring many different sets of data because of an apparent bug in the storage server is extremely rare.

Furthermore, while few people ever see them, SANs like this are very common devices in data centres. They provide a generic low-level storage service and are expected to provide it highly reliably.

Indeed, HPE markets its enterprise storage systems with a “99.9999% uptime guarantee”, which requires that a device is non-operational for no more than 30 seconds per year.

Over the past few days, the IT staff at the Australian Tax Office have probably had a few sleepless nights. It’s likely that engineers at HPE will have a few more trying to get to the bottom of why their enterprise storage system seems to have failed so comprehensively.

Authors: Robert Merkel, Lecturer in Software Engineering, Monash University

Read more http://theconversation.com/server-down-what-caused-the-ato-systems-to-crash-70396

Business News

The Growing Demand for Durable Plastic Packaging Supplies Across Australia

Why Do Australian Businesses Need Reliable Plastic Packaging Supplies? Businesses need packaging that can withstand everyday handling, storage, and distribution without creating avoidable delays. Pla...

Daily Bulletin - avatar Daily Bulletin

Office Cleaning Melbourne: Choosing the Right Scope for Your Office

A well-planned office cleaning Melbourne arrangement starts with a clear understanding of the premises and the people who use it. An office may include private work areas, shared desks, reception, mee...

Daily Bulletin - avatar Daily Bulletin

How Businesses Can Prepare for Unexpected Power Outages

Power failures can occur quickly. The supply may be interrupted by bad weather, issues with the electricity network, damaged equipment or local construction work. This can be just a hassle for a fam...

Daily Bulletin - avatar Daily Bulletin

How Chatswood Tutoring Can Fit Into a Student’s Weekly Routine

Create a Predictable Weekly Structure A consistent weekly routine can make study easier to manage when students are balancing several subjects. Chatswood tutoring can provide a fixed learning point a...

Daily Bulletin - avatar Daily Bulletin

The Rise of Digital Marketplaces in the Australian Trade Sector

For decades, the Australian trade and construction sector operated almost entirely on word-of-mouth recommendations and local community networks. Small business owners typically relied on local newspa...

Daily Bulletin - avatar Daily Bulletin

How Immigration Lawyers Can Help

Introduction Visa decisions can shape employment, family life, study plans, travel, and future residence. A small omission can lead to delay, added expense, or refusal. Immigration lawyers assess l...

Daily Bulletin - avatar Daily Bulletin

How Industrial Drying Equipment Supports Efficient Processing

Many industrial processes require moisture to be removed from compressed air, products or process materials before they move to the next stage. Excess moisture can affect equipment performance, produc...

Daily Bulletin - avatar Daily Bulletin

Practical Ways a Whiteboard Can Improve Workplace Communication

Effective communication helps teams stay organised, share ideas and keep track of important information. While digital tools are now common in many workplaces, a whiteboard continues to provide a simp...

Daily Bulletin - avatar Daily Bulletin

Designing Eco-Friendly Custom Water Bottles for Your Next Event

The Evolution of Sustainable Event Merchandise Event planning has undergone a massive transformation over the last decade. Gone are the days when organizers could hand out cheap, single use plastic...

Daily Bulletin - avatar Daily Bulletin

The Daily Magazine

How Much Life Does Your Roof Have Left? Recognizing When Replacement Becomes Necessary

A roof can age quietly for years. From the street, everything may look reassuringly ordinary, even w...

How Australian Drivers Can Make Their Cars Harder to Steal

Most people don't think much about car theft until it happens to them, or to someone they know. Th...

The Short-Term Rental Rules Australian Owners Can’t Afford to Guess At

A holiday let looks simple from the outside: a spare property, a listing, some guests. Underneath ...

The Risks Of Removing Ear Wax Without Professional Guidance

Ear wax is an essential component of the hygiene process since it acts as a barrier between the ea...

Plumbing Call-Out Fees in Perth Explained

When you hire a plumber to fix a problem with a dripping tap, leaking pipe, or whatever it might be...

Workshop Tool Maintenance: A Practical Guide to Drills, Sanders and Table Saws

A drill that strips every screw head. A sander that leaves swirl marks however long you work the pan...

How Architecture and Engineering Firms Can Improve Project Profitability

Project profitability remains one of the most persistent challenges for architecture and engineering...

The Blossoming World of Fresh Flower Delivery: An In-Depth Exploration

In recent years, the demand for fresh flower delivery in Sydney has soared, driven in part by the ...

How Penrith SMEs Can Keep Stock and Equipment Working for the Business

For Penrith SMEs in 2026, efficient inventory starts with reliable stock data: what is on hand, wher...