A recent poll putting the Liberals and Conservatives at roughly even strength in Alberta, but with a 10% margin of error got me to thinking about the accuracy of such polls, and what a 10% margin of error means. It certainly sounds quite high, but does it mean that the poll is useless? I wanted to know.

So I whipped up a spreadsheet in Excel to simulate polls. 100 people polled represents a 10% margin of error, so that's the sample size I simulated. In Alberta, there's an approximate support level of about 20% for parties other than the Liberals and Conservatives, so I set that and consequently started at 40% Liberal and 40% Conservative "true" support. I ran 20000 simulated polls in each test with varying "true" differences in support. These are the results:



So that means that if you poll 100 people, and the true levels of support are 39-41-20 (Con-Lib-Oth for example), my simulation gave the leading party a leading poll result 56.6% of the time. In other words, with a 10% margin of error, a 2% lead in true support was accurately reflected more than half the time. A 4% true lead was accurately reflected 65.6% of the time. a 10% lead in support, a level equal to the margin of error, was accurately reflected 86.2% of the time. The poll inspiring this investigation reported a 40% lead vanishing to nothing. So I simulated a 40% lead too. This lead was NEVER falsely reported, even with a margin of error of 10%.

So, even though the accuracy at low margins is poor, even with a 10% margin of error and a true lead of only 2%, the poll results as far as showing who's leading and who's trailing are probably correct.

Just something to think about whenever you read poll results.

   

The only poll that ever matters is the one where the votes are cast. I especially distrust polls when they say something I like.

   

Yea, but polls measure real things and have real results and it's good to understand them.



Me too. I always worry it's gonna make the other side fight harder.