Call for independent watchdog to monitor NZ government use of artificial intelligence

New Zealand is a leader in government use of artificial intelligence (AI). It is part of a global network of countries that use predictive algorithms in government decision making, for anything from the optimal scheduling of public hospital beds to whether an offender should be released from prison, based on their likelihood of reoffending, or the efficient processing of simple insurance claims.

But the official use of AI algorithms in government has been in the spotlight in recent years. On the plus side, AI can enhance the accuracy, efficiency and fairness of day-to-day decision making. But concerns have also been expressed regarding transparency, meaningful human control, data protection and bias.

In a report released today, we recommend New Zealand establish a new independent regulator to monitor and address the risks associated with these digital technologies.

Read more: To protect us from the risks of advanced artificial intelligence, we need to act now

AI and transparency

There are three important issues regarding transparency.

One relates to the inspectability of algorithms. Some aspects of New Zealand government practice are reassuring. Unlike some countries that use commercial AI products, New Zealand has tended to build government AI tools in-house. This means that we know how the tools work.

But intelligibility is another issue. Knowing how an AI system works doesn’t guarantee the decisions it reaches will be understood by the people affected. The best performing AI systems are often extremely complex.

To make explanations intelligible, additional technology is required. A decision-making system can be supplemented with an “explanation system”. These are additional algorithms “bolted on” to the main algorithm we seek to understand. Their job is to construct simpler models of how the underlying algorithms work – simple enough to be understandable to people. We believe explanation systems will be increasingly important as AI technology advances.

A final type of transparency relates to public access to information about the AI systems used in government. The public should know what AI systems their government uses as well as how well they perform. Systems should be regularly evaluated and summary results made available to the public in a systematic format.

Read more: Avoid the politics and let artificial intelligence decide your vote in the next election

New Zealand’s law and transparency

Our report takes a detailed look at how well New Zealand law currently handles these transparency issues.

New Zealand doesn’t have laws specifically tailored towards algorithms, but some are relevant in this context. For instance, New Zealand’s Official Information Act (OIA) provides a right to reasons for decisions by official agencies, and this is likely to apply to algorithmic decisions just as much as human ones. This is in notable contrast to Australia, which doesn’t impose a general duty on public officials to provide reasons for their decisions.

But even the OIA would come up short where decisions are made or supported by opaque decision systems. That is why we recommend that predictive algorithms used by government, whether developed commercially or in-house, must feature in a public register, must be publicly inspectable, and (if necessary) must be supplemented with explanation systems.

Human control and data protection

Another issue relates to human control. Some of the concerns around algorithmic decision-making are best addressed by making sure there is a “human in the loop,” with a human having final sign off on any important decision. However, we don’t think this is likely to be an adequate solution in the most important cases.

Read more: Automated vehicles may encourage a new breed of distracted drivers

A persistent theme of research in industrial psychology is that humans become overly trusting and uncritical of automated systems, especially when those systems are reliable most of the time. Just adding a human “in the loop” will not always produce better outcomes. Indeed in certain contexts, human collaboration will offer false reassurance, rendering AI-assisted decisions less accurate.

With respect to data protection, we flag the problem of “inferred data”. This is data inferred about people rather than supplied by them directly (just as when Amazon infers that you might like a certain book on the basis of books it knows you have purchased). Among other recommendations, our report calls for New Zealand to consider the legal status of inferred data, and whether it should be treated the same way as primary data.

Bias and discrimination

A final area of concern is bias. Computer systems might look unbiased, but if they are relying on “dirty data” from previous decisions, they could have the effect of “baking in” discriminatory assumptions and practices. New Zealand’s anti-discrimination laws are likely to apply to algorithmic decisions, but making sure discrimination doesn’t creep back in will require ongoing monitoring.

The report also notes that while “individual rights” — for example, against discrimination — are important, we can’t entirely rely on them to guard against all of these risks. For one thing, affected people will often be those with the least economic or political power. So while they may have the “right” not to be discriminated against, it will be cold comfort to them if they have no way of enforcing it.

There is also the danger that they won’t be able to see the whole picture, to know whether an algorithm’s decisions are affecting different sections of the community differently. To enable a broader discussion about bias, public evaluation of AI tools should arguably include results for specific sub-populations, as well as for the whole population.

A new independent body will be essential if New Zealand wants to harness the benefits of algorithmic tools while avoiding or minimising their risks to the public.

Alistair Knott, James Maclaurin and Joy Liddicoat, collaborators on the AI and Law in New Zealand project, have contributed to the writing of this piece.