5050 Fifty Fifty Politics ← Go to Homepage
Media & Information

Why the Same Poll Produces Two Opposite Headlines

Fifty Fifty Politics · Background & Data
A poll is released. One outlet reports that support is climbing. Another reports that it has stalled. Both are describing the same survey, and both can be technically accurate at once. The explanation is not usually dishonesty. It is that a poll is a set of choices, and a headline reports one of them.

The margin of error is wider than it looks

A poll showing a candidate at 47 percent with a margin of error of plus or minus three is not reporting 47 percent. It is reporting that the true figure is probably somewhere between 44 and 50.

That range is wide enough to support two opposite headlines. Approaching half, or stuck in the mid-forties, both sit inside it. Neither is inaccurate, and a reader shown only one has no way to know the other was equally available.

What a margin of error actually covers One result, reported two ways A candidate polls at 47%, margin of error ±3 47% 44% 50% HEADLINE A “Support climbs toward half” HEADLINE B “Support stuck in mid-forties” Both headlines sit inside the same interval. Neither is inaccurate.

Comparisons multiply the uncertainty

The problem compounds when two numbers are compared. If support was 44 percent last month and 47 this month, the headline writes itself: support is up three points. But each figure carries its own margin of error, so the change between them is less certain than either number alone.

A three-point move in a poll with a three-point margin is, statistically, close to nothing. Coverage rarely says so, because a headline reporting that nothing measurable happened is not a headline anyone runs.

Who was asked changes the answer

Polls report on different populations, and the difference is not cosmetic. All adults, registered voters and likely voters produce systematically different results, because the groups differ in composition.

Likely voter models are the most consequential and least visible. Each pollster decides who counts as likely using its own method, and those methods are not standardised. Two polls fielded on the same days with the same questions can differ mainly because they disagreed about who would turn up.

Four decisions made before you see a number Four decisions made before you see a number Who was asked All adults, registered voters, or likely voters. These give different answers. How it was worded Small changes in wording move results by double digits. Which number is led with The topline, or a subgroup that moved more. How it was weighted Raw responses are adjusted to match the population. Methods differ.

Question wording moves results by more than most people expect

Survey researchers have documented for decades that small changes in wording produce large changes in response. The best-known example comes from long-running General Social Survey data: Americans express very different levels of support for spending on assistance to the poor than for spending on welfare, despite the two describing similar programmes.

Order matters too. A question asked after several related questions gets different answers than the same question asked first, because the earlier items have primed what the respondent is thinking about. None of this is manipulation. It is an unavoidable property of asking people things.

Which number gets led with

A poll release typically contains a topline result and dozens of crosstabs breaking it down by age, region, party and more. Every one of those is a candidate headline.

When the topline has not moved but one subgroup has, the subgroup becomes the story. That is a legitimate editorial decision, and it is also how a survey showing broad stability produces coverage about dramatic change. The number in the headline is rarely the most important number in the release. It is the most interesting one.

House effects are real and mostly boring

Individual pollsters show consistent small leans relative to the average, a pattern known as a house effect. It is tempting to read these as bias, and occasionally that is right, but usually the cause is methodological: how a pollster reaches respondents, whether by phone, panel or text, and how it weights the result.

The practical implication is that comparing two polls from different firms tells you less than comparing a single firm's poll to its own previous one. A firm's trend is usually more informative than its level.

How to read a poll in two minutes

Find the margin of error and treat the reported figure as the middle of a range rather than a point. Check who was sampled, and whether the population is the one the headline implies. Look for the exact question wording, which reputable pollsters publish. Note the field dates, because a poll taken before a major event is measuring a world that no longer exists.

Then, if the number matters, look for a polling average rather than a single survey. Individual polls have house effects and random variation. Averages smooth out both, and they are the closest thing to a reliable reading available to a general reader.

One last check is worth the time: find out who paid for the poll. Surveys commissioned by campaigns, trade associations or advocacy groups are not automatically unreliable, and many are conducted by reputable firms to normal standards. But a sponsor chooses which questions get asked and which results get released, and a poll showing an unhelpful result is simply never published. That selection happens before any of the methodology matters.

The underlying habit is the same one that helps with fact-check ratings and AI summaries: the verdict is a compression, and the information is in what was compressed.

Sources
Browse All Blogs