Politicians are inflating the evidence used to justify tax increases

Many government policies are flawed because the economics research upon which they are based suffers from high bias and low credibility.

Governments rely upon economics research to identify those policies that produce public benefit. However, a new survey we conducted with our fellow researcher, John Ioannidis, shows economic studies frequently report effects to be much larger than they actually are, leading to inflated claims about policy effectiveness and public benefit.

Australian policymakers face an additional hurdle, as the economic evidence base is thin for many policy issues relevant to Australia.

Read more: The 10 stuff-ups we all make when interpreting research

We investigated more than 64,000 economic estimates reported in 6,700 international economics studies, finding many lack statistical power and thereby lack firm scientific credibility. Similar problems exist in science in general.

The typical reported economic effect is inflated by 100%. In other words, over half of economic research results are reported to be twice as large as they actually are. Worse, we found one-third of economics research results are exaggerated by a factor of four or more, so that potential policy effects are made to appear to be four times too big.

We also investigated whether economics research tools have the power to answer the issues in question. We found the typical economics study has a low probability to detect the relevant effects.

What policies are based on shaky economic evidence

One example of where evidence and policy in economics fall out is on the effect of unions on business performance.

The global evidence base consists of 301 studies. Of these, only four use data from Australian companies. What reliable conclusions can be drawn when the evidence base is so small? Extrapolating from the US or European studies can be misleading given their different methods of determining wages, working conditions and labour market regulations.

Another example is the effectiveness of taxes in curbing alcohol consumption. The National Alcohol Strategy flags the use of a minimum price on alcohol and tax rises to reduce consumption, stating that:

There is good evidence that higher alcohol prices decrease both alcohol consumption and alcohol-related harms…

However, a closer look at the evidence does not support this, because studies typically exaggerate the effectiveness of taxes in reducing alcohol consumption.

A survey of 114 studies conducted around the globe suggests that a 10% increase in the price of beer will reduce consumption by about 4%. However, when this evidence is corrected for bias, any tax increase will be only half as effective as planned.

In the case of consumption of spirits, the 14 available Australian estimates suggest that a 10% increase in the price of spirits will reduce consumption by about 6.5%. When corrected for bias, we found taxes reduce only about half as much alcohol consumption.

But the studies found these taxes are good for raising tax revenue for government!

Another example is the conservation of water. Studies imply that a 10% increase in price would reduce water consumption by 4%. Based on this, renowned experts call for higher prices to conserve water in Australia.

However, when we correct this evidence for bias, the same 10% increase in the price of water would now reduce water consumption by only 1%. This doesn’t mean that prices should not be used to ration water. However, it does mean that increasing water prices will be less effective in curtailing water usage than governments expect and that other means need to be used as well.

So, what can policymakers do?

Politicians need to be especially cautious of the results reported by any single economics study and not put too much faith on a reading of the reported research. Instead, careful reviews of the evidence base need to be undertaken, especially meta-analyses, that explicitly make allowance for likely bias. In lieu of more research, prudent policy should be based on conservative, estimated lower values.

The low responsiveness of policies based on price suggests that while well functioning markets are necessary, they may often not be sufficient for effective policy. Many policy concerns require changing norms and behaviour instead. But this is much more difficult than raising taxes.