Crossed wires: ISPs are already struggling to retain our metadata

Today we learnt that 84% of internet service providers (ISPs) in Australia have not met the deadline set by the federal government for them to start collecting metadata. And 61% are asking for some exemption or variation in the requirements specified in the legislation.

This suggests the metadata retention legislation is proving to be more complicated to enact than the federal government has previously suggested.

Not surprisingly, much of the source of the difficulty seems to be a lack of clarity as to precisely what metadata is to be collected.

When metadata collection was first mooted, the goal was to provide intelligence and law enforcement agencies with an internet “phone book”, in the words of David Irvine, the former head of ASIO.

In other words, the intelligence agencies wanted sufficient information to link identity to internet traffic in the same way as a telephone book links a name to a phone number. If they intercepted someone downloading material from a terrorist website, they wanted to be able to work out who was doing the downloading.

This is not as simple as it sounds. There is no long lasting identifier, such as a phone number, for the internet. Data on the internet consists of chunks of data called packets with the source and destination identified by IP addresses.

However, IP addresses are a scarce resource, which are usually allocated to individual users as they need them. When no longer in use, they are reallocated to someone else. Keeping track of who had what IP address at what time would have met the goal of providing an internet phone book.

More than a phone book

If that was all that was to be collected, metadata retention might still have been controversial, but would be comparatively simple for the ISP to implement. It would have been, as was originally claimed, retention of data that was already being collected.

Unfortunately, the legislation requires retention of a lot more than just the mapping between user identity and IP address. Data to be retained includes the source, destination, date and time of a communication.

It includes volume of data uploaded and downloaded and, in the case of mobile communication, which cell tower the user was connected to. It is quite a long way from an internet phone book.

The decision to collect more than a very basic set of data has made the legislation much more complex than it might otherwise have been.

The authors of the legislation decided they did not want to be too specific about what would be collected. Specifying exactly what data item was to be retained for every possible service would be a huge job and would have been out of date very quickly.

So rather than specifying (for example) source and destination email addresses being retained, the legislation talks about retention of the source and destination of “a communication”. Helpfully, examples of “a communication” are listed in the legislation and include email, SMS, chat and voice service but the list is not intended to be exhaustive.

Unfortunately the consequence of this approach is that the task of clarifying what data actually is to be collected has fallen onto the ISPs. This is not a trivial task.

As John Stanton from the Communications Alliance points out:

There are a thousand different nuances that I’ve seen flying around as to what needs to be retained in respect of a particular service.

Sorting out these issues is a big task that for some small ISPs is proving very difficult.

ISPs can apply for exemptions and be given up to another 18 months to implement the legislation, although the application process seems quite difficult and slow.

Laurie Patton, CEO of Internet Australia, has described the legislation as “fundamentally flawed” and called for an immediate review, with the aim of making the requirements much more easily understood and so help compliance. If 84% of ISPs are struggling to comply then that sounds like a very reasonable suggestion.

Philip Branch does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond the academic appointment above.