Support for democracy and trust in politicians is falling. We hear a lot about evidence-based policy as a way to stem this decline, but less about how that evidence should be generated.
One idea that may generate the type of evidence that will help make more informed decisions appears, paradoxically, fairly unpopular with the punters.
Perhaps the problem is that not enough has been done to explain to the public what this idea - carefully testing new policies on small groups first - might mean in practice.
In a new paper just released, we show that we may still be a long way off adopting this practice.Mark Esposito/AAP
There is an emerging view that there should be much greater use of evaluations of public policies, including randomised controlled trials (RCTs), to test the effectiveness of new policies before they are rolled out. This applies particularly to policies or programs for which there is limited or no evidence about their likely impact.
RCTs have been around for years in medicine and other sciences, and are increasingly being used by small and large companies to test products and services. Conceptually they are simple, although implementing one can be complex. A RCT involves selecting a sample from a population of interest and randomly dividing them into two groups (using the equivalent of a coin toss). One group is given an intervention (that is, a program or policy) and the other is not. If the RCT has been done properly, the differences in the outcomes of the two groups tells us the impact of the intervention being trialled.
There are other ways to try to measure causation, and some are necessary when an RCT isn’t possible. However, Shadow Assistant Treasurer Andrew Leigh argues in his new book Randomistas that:
Researchers have spent years thinking about how best to come up with credible comparison groups, but the benchmark to which they keep returning is the randomised controlled trial. There’s simply no better way to determine the counterfactual than to randomly allocate participants into two groups: one that gets the treatment, and another that does not.
While there is strong support within the policy and research community on the important role of trials and evaluations, we know far less about what the general public thinks about how policies should be implemented and to what extent they should be trialled before widespread introduction.
In a survey undertaken as part of the ANUPoll series, we ran an online survey experiment that measured the level of support for trials in general and RCTs in particular. We also looked at the factors that influence that support, and whether there is a causal relationship between expert opinion, party identification and support for an RCT.
That is, we ran an RCT on RCTs.
As part of the survey, we asked respondents to “consider a hypothetical proposal to reform” in one of five policy areas (school education; early childhood education; health; policing; support for those seeking employment). We then asked “which of the following approaches do you think the government should take?”:
- Introduce the policy for everyone in Australia at the same time
- Introduce the policy to everyone, but do it in stages
- Trial on a small segment of the population who need it most, or
- Trial on a small segment of the population chosen randomly,
We found that more people want new government policies rolled out without testing - except for jobless support.
Authors: Nicholas Biddle, Associate Professor, ANU College of Arts and Social Sciences, Australian National University