What the numbers say (and don't say) in the same-sex marriage survey

More than 12 million people took part in the same-sex marriage postal survey with the results showing that 61.6% voted Yes and 38.4% voted No. But not everyone voted, so what can we draw from the results?

Based on the figures from the Australian Bureau of Statistics (ABS), which conducted the voluntary survey, the result is seen as a win for the Yes vote.

But right now we can’t make any broader interpretation of our nation’s overall view of same-sex marriage. That would require linking these data with follow-up surveys and more detailed analysis.

Read more: Same-sex marriage survey by the stats: a resounding ‘yes’ but western Sydney leads ‘no’ vote

The numbers straight up

The ABS reports that more than 79.5% of voters enrolled with the Australian Electoral Commission submitted their form with a clear vote cast for either side of the debate.

Simply put, this means that slightly fewer than half of all eligible voters (49.0%) voted Yes and a little less than a third (30.5%) voted No. The rest did not vote or their preference was unclear.

If the survey was a two-party election, it would be a landslide result for the Yes vote. According to the ABS:

All states and territories recorded a majority Yes response. 133 of the 150 Federal Electoral Divisions recorded a majority Yes response, and 17 of the 150 Federal Electoral Divisions recorded a majority No response.

So does this mean that we now know what all Australian eligible voters think about this issue? Can we conclude that a majority of eligible voters are in favour of same-sex marriage?

What the numbers can’t tell us

The number one point to understand about these results is not what they are telling us, but what they are not telling us. It is tempting to believe that the survey outcome means that 61.6% of Australians are in favour of same-sex marriage, and 38.4% against. This is not true.

While the response rate was high, we know that 20.5% of eligible voters didn’t take part.

Given Australians’ attitudes to voting in elections, such a high response is to be expected. The latest figures from the Australian Election Study Update report that 70% of Australian voters favour compulsory voting, and 80% would vote even if it was voluntary. This seems to accurately reflect the response rate of the current marriage equality vote.

But all we are really able to say is that these figures (61.6% Yes and 38.4% No) are representative of those eligible voters who chose to vote, rather than being representative of all eligible Australians. These two populations may be very different.

This may seem a subtle point, but it is an important one when trying to understand the survey results in the context of informing government policy, which is the intent in this case.

It is also important to get the right interpretation when reporting the results in the media, as marriage equality is a sensitive issue for many people. This point was recently raised by the Statistical Society of Australia, which was particularly concerned that the survey results could be misinterpreted.

When running surveys it is common to use statistical techniques that would allow the survey results to be extended to the wider population. But the government decided this was not needed in this case.

A look at the non-voters

If all eligible voters who did not vote, or who did not provide a clear vote, responded No to the survey question, then the Yes vote would be 49.0% and the No vote 51.0%. If all non-voters voted Yes, then the Yes vote would be 69.5% and the No vote 30.5%.

These extreme cases are unlikely, and the true opinions of the non-voters will be somewhere in between. For the Yes vote to reach 50.0%, only 185,843 non-voters (or 5.7% of non-voters) would need to cast a Yes vote. Intuitively it may seem reasonable that this could be the case, and it might seem highly unusual for this not to occur.

But is there any principled way to determine how the non-voters could have voted?

This is a standard question that could have been answered well if the same-sex marriage postal survey had been conducted as a proper statistical survey. But the government direction made it clear that this was not required.

Louise Ryan’s recent article described the issues involved in conducting a good statistical survey in the context of the marriage equality vote.

One such statistical survey was conducted last week by The Guardian as the marriage equality survey closed. It estimated a 64% Yes vote and a 31% No vote for all eligible voters.

Read more: The marriage equality survey is won, but the battle against discrimination continues

Follow-up surveys such as these, which incorporate demographic information such as the age distribution of voters, can be combined with the ABS figures to provide a more in-depth analysis than presented here.

This would allow the attitudes of the non-voters to be known in more detail, and would then allow us to be more certain about the outcome of the marriage survey.

From the ABS figures, we can see that more older voters returned surveys than younger voters. Of the younger voters, more females than males voted.

One concern is that voters of different ages may have voted differently. Follow-up statistical surveys may help clarify this issue.