Polling is more complex than Fox News boss Roger Ailes wants you to know

The first Republican presidential debate of the 2016 election cycle will be held on August 6, more than 15 months before the election and almost five months before the Iowa caucus.

Because of the large field of official candidates, Fox News will select just 10 to take part in the debate. The remainder of the field will be invited to a debate “kiddie table” that will be held earlier that day.

Fox News has released the following criteria for trimming down the list of official candidates:

Must place in the top 10 of an average of the five most recent national polls, as recognized by Fox News leading up to Aug. 4 at 5 p.m. EST. Such polling must be conducted by major, nationally recognized organizations that use standard methodological techniques.

Jon Stewart’s quip may not be far wrong: “Ah, so basically they’re going to look at the polls and [Fox News boss] Roger Ailes is going to pick whoever he wants.”

Which five polls?

Between the day Donald Trump officially entered the race though August 4, 18 national polls were conducted by dedicated polling firms for major national news sources. Polls were also conducted by major universities.

These polls differed on several important aspects: sample size, voting qualification and telephone use. Each of these differences affects which candidate gets the most support. These differences also raise questions of how – or if – they can be averaged meaningfully. This is the reason behind the Marist Institute of Public Opinion’s decision to suspend its Republican polling until after the Fox News decision.

“It’s a bad use of public polls,” Lee Miringoff of Marist told Politico.

Sample size and precision

The 18 national polls we identified have sample sizes ranging from 300 to 815. As a general rule, larger sample sizes provide estimates that are more precise. At the top end of the range, a sample size of 815 corresponds to a margin of error of approximately plus or minus 3.5%; a sample size of 300, to plus or minus 5.7%.

This precision is important for the candidates vying for one of the 10 debate positions. Of the 710 sampled in the July 28 Quinnipiac poll, for example, 3% supported Rick Perry and 6% supported Ben Carson.

This tells us that Carson has more support than Perry in this particular sample. However, it does not follow that Carson has more support among the entire population of Republican primary voters. The margin of error is too large to come to that conclusion. What we can say based on the basis of this poll is that we are 95% confident that Perry’s actual level of support in the entire population is between 0 and 6.5%. Since this interval contains Carson’s estimate, these two candidates are in a statistical tie.

So, the first problem with the method proposed by Fox News is that we can’t say with any certainty which candidates are more popular.

Voting qualification

The second issue is one of who will be polled.

If the polling firm is attempting to estimate the candidate support in the general population, then that firm should contact a sample of all US adults. If, however, the firm wants to estimate support within those who will vote in the Republican primaries, then the polling firm should ask only Republicans who will vote in the primaries.

These two populations are very different and will lead to very different results.

In the 18 polls, four different populations of Republicans and Republican-leaning independents were sampled: all adults, registered voters, likely voters and primary voters.

Different polling firms used different populations. For instance, CBS News uses “primary voters.” Fox News uses “registered voters.”

The different sampled populations can give a different ordering of candidate preference. Which of these four populations is of most interest to Fox News? Fox News isn’t saying.

Telephone use and accuracy

The third issue centers on biases in sampling.

Research we conducted on polling in the 2012 election uncovered a probable reason pollsters predicted a victory of Republican candidate Mitt Romney.

The pollsters were relying too heavily on responses to landline telephones, creating a bias against cellphone-only households. Cellphone-only households, according to the Centers for Disease Control, tend to be younger, poorer and urban. Excluding these groups from polling biases the results away from their preferences.

This is the first election where the majority of polling firms are explicitly contacting cellphone numbers in addition to landlines. However, several polls still fail to contact cellphone users. Skipping these people severely biases the estimates.

The upshot is that those Republican candidates who appeal to the younger crowd are disadvantaged in polls that do not explicitly used cellphone numbers in addition to landline numbers.

A question of averaging

One last thing Fox isn’t making public is how they intend to average the five polls.

Will they just average the candidate support predictions? Will they weight the predictions based on the sample sizes? If so, which statistical model will they use?

The choices they make can affect who is on stage for the main event.

Take the example of Mike Huckabee and Marco Rubio. Simply averaging their support estimates puts Rubio ahead of Huckabee. If the estimates are weighted using their sample size, Huckabee is ahead of Rubio.

Since this just swaps the candidates between the #4 and the #5 spots, it does not affect who is in the debate. It does affect, however, what podium a candidate occupies, with those with highest poll numbers literally getting “center stage.”

Of course, these averaging issues can also affect the all-important #10 spot, which is currently a toss-up between John Kasich and Rick Perry.

The final 10 are…

The upshot of this insecurity is that Fox News has wide latitude in this first debate. Could Fox News decide the debate lineup based on something other than the polls? Arguably, is Perry more entertaining on the stage than Kasich? Also of note are the multiple interviews that Perry has given Fox News over the course of the past week.

As it currently stands, we propose that the following nine are safely in the debate: Donald Trump, Scott Walker, Jeb Bush, Marco Rubio, Mike Huckabee, Rand Paul, Ben Carson, Ted Cruz and Chris Christie.

The #10 spot most likely goes to John Kasich, who holds a slight edge over Rick Perry.

The remaining six major candidates should not wait for the phone call, unless it is from a polling company.

The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond the academic appointment above.