Explainer: why we shouldn't be so quick to trust energy modelling

As is so often the way with political debate about new policies, much of the rhetoric around the federal government’s proposed National Energy Guarantee (NEG) is couched in terms of “the modelling”.

According to modelling released last week by the government, the plan will save households an average of A$120 a year while increasing the stability of our energy system and reducing greenhouse emissions.

Read more: Energy ministers’ power policy pow-wow is still driven more by headlines than details

But opponents have called for the NEG to be modelled alongside alternative policies, such as an emissions trading scheme or the clean energy target proposed by Chief Scientist Alan Finkel.

But models, while a useful tool, are not infallible. They’re only as good as the assumptions on which they are based – and the proprietary nature of most models makes it difficult to draw direct comparisons between policies. It’s worth keeping in mind what models can – and can’t – tell us.

What are we modelling?

To start with the basics, these models are simulations of the commercial operation of the National Electricity Market (NEM), which covers most of the country (excluding Western Australia and the Northern Territory) and delivers 90% of Australia’s electricity.

Read more: The National Electricity Market has served its purpose – it’s time to move on

The NEM is generally used to refer to both the physical structure of the grid (the power stations, interconnectors and so on) and the behaviour of market participants (generators).

Generators offer bids every five minutes with the volume and price of electricity they’re willing and able to supply, which are accepted (or not) by the Australian Energy Market Operator.

To simulate this bidding, a model must include a representative profile of demand for electricity during each five-minute period of a year in each NEM region, together with an estimate of the total electricity used in that year.

For greater accuracy (and complexity), the model must also simulate scheduled maintenance, the closing of old power plants, and investment in new ones (all in response to market signals). The modelled bidding behaviour may also be influenced by short- and long-term weather, solar and wind forecasts.

Complete simulation of the operation of the NEM requires the model to include a representation of the physical structure of the grid through which electrical energy flows. Detailed operational information is added to this, such as the running costs and technical performance of each power station, which are known with reasonable certainty.

But to say anything about the future, it also requires the model to make assumptions about economic factors such as the capital cost of new builds, the prices of coal and gas, future interest rates and so on.

It’s these economic assumptions that can make models differ radically from one another, and from the eventual reality.

Models in Australia

Currently in Australia there are four or five models that claim to simulate the NEM wholesale market. Each is the property of a separate commercial consultancy business.

Typically, all the policy work on the electricity market over the past decade or more has been informed (if that is the right word) by one or more of these models.For example, modelling for the Warburton review of the Renewable Energy Target was done by ACIL Allen Consulting; modelling for the Finkel review by Jacobs Group (formerly SKM MMA); modelling for the Victorian Renewable Energy Target by ROAM Consulting (now part of Ernst & Young); and most recently, modelling for the NEG by Frontier Economics.

Given the complexity of the system being modelled, it’s not surprising models differ in how they represent the NEM and its operation. But the proprietary nature of the competing models makes it virtually impossible for an outside observer (or even a client) to know what the differences are, let alone understand how that may affect the result.

One way to test the impact of different assumptions on a policy is to give the same set of questions and starting information to two modelling companies, and compare their results.

This is very rarely done, but we can look at the example of the Rudd government’s Carbon Pollution Reduction Scheme, which was modelled by both SKM MMA and ROAM Consulting. Surprisingly, the two models produced broadly similar results – but there was a margin of error between them.

It’s worth noting that the NEG, the government’s current proposed policy, has modelled household savings of A$120 per year – which could fall entirely within this margin of error.

Read more: Infographic: the National Energy Guarantee at a glance

Another way to test a model’s assumptions is through “sensitivity analysis”, which is carried out by the company doing the modelling. This is simply testing with assumed values for a key factor, and comparing the results.

In the case of the NEG modelling, a key assumption is that the cost of financing new energy infrastructure would be around 3% cheaper with the NEG than without it. But the sensitivity analysis found if there’s no difference in financing, about half of the A$120 savings disappear.

This assumption rests on the policy providing stability, which reduces the risk to investors and makes raising capital easier and cheaper. It seems reasonable to assume the presence of a national-level energy policy provides better outcomes than having none at all, which is what the model tested.

But this is why opponents want the NEG modelled in comparison to other policies, which may provide better outcomes. The government, of course, is refusing to give its detractors ammunition by crunching the numbers on rival schemes.

One way to move past this bickering would be for the government to use public funds (perhaps through the Australian Renewable Energy Agency) to fund open-source models, which are then used to test public policies.

Allowing all interested parties to investigate and validate the model would go a long way towards restoring public trust in what has otherwise been largely a short-sighted and self-interested debate over the future of Australia’s energy market.