How better data would improve the electricity market

The Australian Electricity Regulator is investigating whether wholesale electricity generators in New South Wales are bidding “in good faith” in the electricity market. Good faith means price changes are the result of real problems, such as weather or machinery failure, rather than market manipulation.

The reason the regulator doesn’t already know the answer to this question is that the market is opaque – the data are not easily workable for analysis. In our ongoing research into the electricity market we have run into the same problem. We’ve only been able to make headway by applying big data tools.

Although not bidding in good faith isn’t illegal in the Australian electricity market, it is reminiscent of fradulent bidding in United States Treasury bond markets in the 1990s. We can learn from how this scandal was dealt with – brokerage firms were required to provide data that could be clearly monitored. This increased transparency and led to lower costs.

How the electricity market works

The way the electricity market works is that each generator offers the market operator (AEMO) a set of ten prices and the volumes of electricity it is willing to supply at each of those prices. Generators do so for every half hour of the day, approximately a day ahead of time. AEMO aggregates the offers from all the generators, forecasts demand and works out the price and amount of electricity it orders each generator to dispatch.

Things may change between the time the offer is made and the actual dispatch time. It might be hotter or colder than expected, or a machine might need maintenance. So there are good reasons for generators to change their offers before dispatch. But there is also an opportunity for generators to change their offers in order to influence prices.

Read more: The National Electricity Market has served its purpose – it’s time to move on

Prices in the wholesale electricity market are determined every five minutes but the generators are paid an average of these five-minute prices for all the electricity they dispatch in a half-hour. If one five-minute price in the half-hour is increased the average price for the whole half-hour also increases and the electricity generators receive more money.

Both AEMC and our own work have shown that generators change their offers most often in the first and/or last five-minute periods of the half-hour. Economic theory tells us this is the most profitable strategy and there is no reason to believe it is the result of weather changes or machine maintenance. It’s one of the reasons for the proposed move to five-minute settlement in the future.

The data obscure the market

AEMO receives data from the generators and provide this information for public scrutiny. The data are easily available but not particularly transparent for analysis. This prevents outsiders from monitoring the market behaviour of individual electricity generators.

When we first started looking at the data it seemed impossible to track how individual electricity generators behave. The spreadsheets had inconsistent headings, unnecessary data, empty rows and multiple data dumps in each file.

After becoming more familiar with the data we tried to apply some machine-learning algorithms to cluster the generators and to see if there are commonalities. But again the data hindered progress and this became very time-consuming.

Eventually, Ali built a three-dimensional model, allowing us to observe the strategic behaviour of generators and analyse this with statistical software. But none of this has been straightforward – everything from the way in which the dates are formatted to the alignment of columns and rows of data has to be carefully considered to extract the useful information.

As with the current AEMO data, the initial datasets in the US Treasury bond markets were difficult to work with. However, with the new data academics and regulators worked together, revealing problems with how prices were set in these markets and improving transparency

The ability to monitor market behaviour in real time has made it more efficient and less costly for participants. This has widespread benefits, such as improving the returns for superannuation funds. This trend has continued since the move was made to electronic trading, which makes the data even more accessible.

Read more: Australian electricity must get more competitive: here’s how

The latest AEMO report strongly supports a review of the market structure, but flags it as a medium-term or longer-term issue.

This is an urgent issue, which needs to be examined in conjunction with the physical infrastructure issues of the electricity market. The physical and financial networks of the market are intertwined, and tackling them one at a time is unlikely to lead to the best outcome.

Improving transparency in the electricity market will both change behaviour and make real-time regulation and oversight possible. There will be some cost to this transition and we do not yet know the size of the benefit from doing so. But it is necessary and important, not to mention easier than building expensive new infrastructure.

A new interconnector to South Australia is expected to cost up to A$2.5 billion. Keeping NSW’s Bayswater and Liddell power stations operational could cost up to A$900 million. These are but two of the investments required to match electricity supply to demand.

Generators and electricity companies have no incentive to provide data in an easily accessible form. It is much more convenient for them to dump data daily, minimising costs while still technically complying with any transparency requirements. This should change.