Nicholas Jacobs is the Goldfarb Family Distinguished Chair in American Government at Colby College, where he also serves as the inaugural director of the Bram Public Policy Lab. 

I love a good poll as much as the next person.

It’s why I’ve relied on them throughout my research and teaching. Surveys offer a rare glimpse into attitudes that are otherwise difficult to observe, and in competitive races they can help orient both journalists and voters to what appears to be unfolding. And this Senate race in Maine — it is competitive. I’m itching for clarity.

Polls matter beyond our general academic curiosity. They actually shape the race and our expectations. The findings out of the University of New Hampshire about Graham Platner’s meteoric rise in the Democratic primary have already begun to shape how observers are talking about the Senate race, subtly altering expectations about competitiveness and early advantage. No doubt, donations will follow the topline finding.

But a word of caution is warranted. Polling in Maine is unusually difficult. And yes, you can simply refuse to “trust the polls,” but let me also suggest you don’t have to even go that far: just look at what the pollsters are and are not telling you each time they report results.

Most anyone who cares about polling results knows a few things to check, none more important than the all important margin of error. It offers a useful reminder that polls estimate rather than measure, and that even well-executed surveys contain uncertainty.

Try telling me who’s ahead with just a few dozen people and you’ll see a margin of error in the double-digits; everyone knows know you might as well stop reading. But a small margin of error only reflects precision, not representativeness — and a survey can be statistically tidy while still overlooking meaningful variation within the electorate.

You can get a representative snapshot of what Maine, on average, thinks with a modest sample — about 1,000 of our neighbors. Yet that is rarely what readers or campaigns are focused on in moments like this. We are not just asking what “Maine” thinks. We are asking what primary voters, independents or late-deciding voters think. And that is where interpretation becomes harder.

As attention shifts to those subsamples, the number of respondents quickly shrinks and the margin of error widens. That mechanical inflation is familiar and usually reported. What is discussed far less is whether those smaller groups meaningfully reflect the diversity of voters they are meant to represent — geographically, politically and in terms of engagement with the race. Because, as is often the case, the initial goal was not to survey, say, young people in Maine, but all people in Maine. That distinction creates problems.

When looking at subsamples, the relevant question is not simply how large the margin of error becomes, but how much confidence we should have that the subsample itself captures the electorate we care about. One way researchers evaluate this is by looking beyond sample size to how heavily responses must be weighted and adjusted to reflect that diversity — a process captured in what survey methodologists call “design effects.”

When those adjustments are substantial, the survey contains less independent information than the respondent count suggests, meaning apparent precision can mask deeper uncertainty about how accurate the estimates really are.

Again, the latest UNH survey in Maine offers a useful illustration.

Buried in the methodology statement, the researchers report a design effect of 2.3 and note that they did not adjust their margins of error for what is a pretty major acknowledgement that their sample, however large, needed some help in representing the broader Maine electorate. Put plainly, a design effect of 2.3 means those 1,120 likely voters function statistically more like a sample of about 500 — making the apparent precision of the results considerably overstated.

If the effective sample size is cut substantially, the true uncertainty around candidate support widens. What was a margin of error of about ±2.9 grows quick, to ±4.5. Of course, this might mean that Platner’s lead over Collins in the general election is higher than what the poll estimated, but it also means that, in this case, his lead could be as small as two points.

Specific to the one finding that is drawing substantial media attention, it also means that Platner’s “advantage” among Maine independents is a statistical fantasy. That is because once you start looking at sub-samples, the “penalty” that a design effect has on a poll’s margin of error is even greater.

To begin with, there are only about 164 independents represented in the full sample — a testament to the large design effect, because the poll seems to have captured way more partisans than proportionally exist in the state. The baseline margin of error for that group, to begin with, is ±7.

And then once weighting and design effects are taken into account, the effective number of independent respondents becomes smaller still — in this case, giving us estimates that have an equal chance of being 12 points higher (Platner leads with 59% of independents!) or 12 points lower (Collins has a 15 point advantage!). We just don’t know.

Now, I realize this may sound like unwelcome news to those eager to read the poll as
confirmation of a decisive shift in the race. I look forward to the emails I will receive telling me my “academic caution” is masquerading as excuse-making for Sen. Collins.

But, if anything, the statistically rigorous takeaway remains quite interesting. The same issue with independents I describe above (an ever-shrinking sample size) is just as true for analyzing the subset of Democratic primary voters. Even after accounting for the design effect here, functionally inflating the margin of error on the Democratic primary, Platner’s lead is unequivocal.

Even the most generous read, given that uncertainty, gives Mills just about a third of Democratic primary voters in the survey. The margin may be less precise, and there are still questions about whether the poll captured the broad swath of likely voters, but the signal is unmistakable: he is a credible and competitive challenger.

Statistical caution does not weaken that conclusion, even as it tempers claims of an inevitable victory for one candidate or party.

Platner’s emergence is real. So is the uncertainty surrounding everything beyond it. Acknowledging that uncertainty, though, is the difference between careful interpretation and wishful thinking. And when uncertainty is translated into premature conclusions, the narrative can begin to influence the election before voters do.