Nate Silver can suck it
 

I kid Nate Silver, I kid.

Ok, let's talk about polls while we still have a chance! Yeah! I'm starting a new thread here to give the prez odds thread a break, and also to draw more attention to myself.

As a jumping off point, let's look at some things wonderboy Nate Silver has to say in his explanation/defense of his model:

http://fivethirtyeight.blogs.nytimes...-the-favorite/

In reaction to some "pushback" against his projection as Obama as a solid favorite, he responds:
So far, so good. And he's right, you do need to justify yourself if you are not going to take the polls at face value. But he's missing half of that equation. You also have to justify yourself if you ARE going to take the polls at face value. What I'm going to argue is that there is no "face value" to a poll. The impression is that a poll is what it is ("it's scientific!"), and if you are going to tweak it, then you better have a good reason. That's true, but again you also need a good reason to accept it as-is since they are tweaked by the pollster and we need to know their reasons in order to accept them. Or maybe they are not tweaked by the pollster and they let the chips fall where they may, but I will argue that that is a distinction without a difference. (I think they SHOULD tweak the samples generally because it is necessary to correct for sampling errors.)

So then Nate goes on to say that it is possible that most of the polls could be biased in one direction. He expands on this in the next day's post:

http://fivethirtyeight.blogs.nytimes...ically-biased/

I recommend you read both of those if you are interested, but the newer post can be summed up by it's title: "For Romney to win state polls must be statistically biased". And that's absolutely true, so an even shorter summary might be "Duh." And his main argument against them not being biased is recent history of them being more-or-less correct. He does claim, by the way, that his model explicitly accounts for this possible bias (whether it is for or against a particular candidate). From the second article:
It is unclear what his actual method is here -- it sounds like he is not really talking about adjusting the accuracy of individual polls (I don't think), but adding a measure of uncertainty to his simulations that creation his projections. I think he does some sort of Monte Carlo simulation, but I haven't followed him enough to know. Anyway, from what I've read of him (which is fairly lightly) I gather he does not dig into individual polls and simply creates his weighted average without considering the poll internals. But really I'm not sure -- certainly he doesn't seem to talk about adjusting polls individually, and I've seen him be dismissive of the idea of digging into individual polls because he doesn't weight any one poll very much and seems to assume that bias errors will generally cancel themselves out in the average. (I've got his book sitting here -- if he turns out to be right about everything maybe I'll read it.) But in this latest article he is acknowledging the idea that most of the polls could in fact be tilted in one direction which would bias his projection also, but since he doesn't really believe it he isn't going into depth on the ways he could be wrong. So let's cover some of that ground. But despite the title (which is mainly for forum user "Jake"'s benefit), this post actually isn't about Nate Silver other than a starting point indicating what the general consensus on the polls out in the world is, or at least in liberal circles. Anyway, let's now leave Mr. Silver aside.

(cont...)

(...cont)

Ok, so polls: art or science? In every interview with a pollster that I've seen, they are asked this question, and they all answer MOSTLY art or at least HALF art. That's the pollsters talking, so let's just put aside right away that polls are these objective scientific snapshots -- they aren't. They try to be of course, but the means to get there is highly subjective and involves human beings making judgement calls about what to include and why, etc. And that's fine, that's the best we can do, but just realize that.

Taking polls at "face value" or interpreting them with our own tweaks and added complexities: is there a fundamental difference? No.

Is it "valid" to interpret/tweak a poll (in general)? Yes.

Is it "valid" to take a poll at face value? Only if you've already got reason to trust the implicit or explicit model that underlies the poll.

And this is the key, all polls make assumptions about the makeup/structure of the population they are polling -- they have an underlying model, all of them. Even if the pollster doesn't "weight" any group or trait in particular and it is up to random chance, THAT IS STILL A MODEL, just one that wasn't explicitly chosen. And in political polling, there is really no such thing as just leaving it up to random chance because it is non-random whether people respond to the poll or not. There is no particular reason to believe a random sample is representative of the voting population as a whole. In fact, there are lots of good reasons to think otherwise. (This is going to be long enough already without going into detail about that here, but maybe in the subsequent discussion.) Pollsters know this, so many of them do introduce weights among the traits and groups in order to hopefully better capture the true population. (Or in some cases unfortunately, in order to deliberately skew the results one way or another for their client. We'll ignore that possibility and assume all these big league pollsters cited in the media are actually trying their best.) But it is "valid" to weight a poll sample in some manner. And that's where the most of art of the thing resides. None of them explicitly reveal all of their methods, but some of them do in broad strokes, usually something along the lines of looking at the region in question, using the registered voter percentages and census figures to come up with a target sample population, and then tweaking that by historical turnout figures for those groups in statewide elections. So that approach attempts to do the arty part in a more or less mechanical "scientific" way. Others may be more aggressive by making assumptions about what they think the difference in turnout will be this time and adjust for that, or that have a whole sideline activity that constantly tries to keep up with the electorate, current enthusiasm, etc to generate precise estimates of turnout based on current conditions. (Rasmussen does some of that -- he has a moving dynamic model, and a pretty good one it seems.) In any case, the resulting target sample is just an educated guess. I'll get back to this.

The other big remaining piece of the "art" side of the puzzle during the final month or so of the campaign is the "likely voter screen". Once that wire starts getting close, the pollsters all switch to "likely voter" models which attempt to focus in on people that will actually cast ballots rather than just anyone that is registered. This is pure art, and there is nothing much to be said about it, except for the early voting problem which I'll get to in a second. Putting early voting aside, they ask a few questions about enthusiasm, have you voted before, etc, to determine if you are a likely voter. In general, the tighter the screen, the more it favors Republicans because there are many more registered Democrats than Republicans (thanks to Motor Voter laws and other such measures that have gotten lots of people to register to vote that often don't care about voting), but those registered Democrats turn out in fewer numbers than Republicans. But most "unlikely" voters are self-screened by not answering polls. Otherwise we'd have to conclude that all likely voter screens are total hogwash because they typically let through 80-90% of the poll respondents, which is obviously far far above the actual turnout gap between registered and actual voters -- a level of 70% is record-breaking in reality. So really we have no idea how accurate these screens are, but the looser they are, the more they skew Democrat.

EXCEPT in the case of early voting -- if you say you voted already, there is nothing else to ask -- you automatically make it through the screen. And early voting favors Democrats because that is a big strategy for them. So with early voting added in, the polls will tend to skew Democrat no matter how tight the screen is. Everybody that claims to have voted is through the screen, so now if for the remaining people you make it too loose you are letting in too many more Democrats (that won't really vote) vs Republicans (who will), but if you tighten it up then your not-yet-voted Democrat numbers will be better but you start turning away Republicans that will in fact vote as "unlikely". So early voting skews Democrat, and heavily. There doesn't seem to be a strategy by the pollsters yet to deal with this -- the ones that don't shoot for particular target sample proportions anyway. They need to adjust but if they've taken an anti-adjustment stance then what we tend to see is the "+D" numbers go up with the early voting. We are definitely seeing this effect in Ohio in some of the polls -- I suspect when they are way off they will come up with something for next time as early voting is becoming more and more prevalent. For instance, if we look at the last three PPP (Public Policy Polling) polls in Ohio, we see them go from D+4 to D+6 to D+8. Significantly, Obama's lead over those polls doesn't go up! It is actually down a point in the last poll from the first, but with a +4D greater advantage in the last poll from the first. What does that tell you?

Other polls seem to more tightly control their samples and you don't see this creeping +D with the early voting.

Here's a quote from Battleground Watch discussing the early voting problem:

See this blog post for some more detail about how that skew works and the difference in strategy between the campaigns on early voting: http://jhpolitics.com/2012/10/how-ea...ing-the-polls/


(cont...)

(...cont)

Ok, moving on, so what about the science part? What CAN we take away from polls at face value?

Well, the direct questions and answers: If they ask who you are voting for, or your age, or what party or ideology do you consider yourself, there is no reason to question those answers. Sure, people can lie, but that's got to be a pretty small number. So that's the actual "opinion of the people" part of the poll (which the turnout numbers are not), but we want to use a whole bunch of such q&a sessions to predict things like who's going to win and by how much. But because we know that the population of poll respondents does NOT represent the entire voting population at large (and the margin of error does not address this question, see my previous post about polling cats & dogs in the prez odds thread), that's where the art comes in and we start weighting our samples and looking at the main variables we are interested in (who are you voting for) cross-referenced with other traits like party id, sex, age, etc. And how we weight those samples controls the "top-line" results of the poll. (e.g. Obama 47, Romney 45). Because the opinion part of the polls are very consistent within regions and groups. For instance, if you break it down by (self-identified) party id, which correlates highest with breaking for one candidate or the other, the results are always around a 90-10 split for the "my party" guy and then the independents break where ever they break for that particular region and set of candidates. And you will generally see strong agreement in all the polls within these categories, even if their top-lines are completely different. (Independents vary the most, but the self identified groups are always between 88-92 for their guy. 88% percent represents a very UNENTHUSIASTIC response to a party candidate within that party.) Close to the election you don't see significant changes in those numbers, so the top-lines result is ultimately controlled almost completely by the makeup of the sample.

It is easiest to pick on party id as *the* factor to look for here, and that's the one people always talk about (which is exactly what I'm going to do as well), but realize there are an infinite number of ways to slice up this data.

Ok, so all of the above boils down to this: the top-line poll results you see are FAR more "turnout estimate models" than they are representative of "the opinion of the people". So just remember when comparing polls with vastly different results what the real differences are. If you drilled into those polls you'd find near identical numbers under the Republican, Democrat, and Independent categories, and the reason they have different top-line results is because the proportions of Republicans, Democrats, and Independents varies wildly between them. So on the science side, we've got numbers that are essentially fixed and that everybody agrees on, and so the difference between the polls is almost exclusively on the art side. (This is not necessarily true over time, but right near the end of the election it is.) So what are we looking at when we see a poll result? An estimate of the TURNOUT in the *opinion* of the pollster. That's it! But their opinions are not (usually) wild guesses -- they are based on the census, voter registrations, historical turnout, and their likely voter screen. But please realize how sensitive these numbers are. Just one percentage point from one party's column that should be in the other party's column has a major effect because they all vote 90-10 for their guy. So polls with a 2 point spread, that means nothing. The exit polls aren't even that accurate, often off by 1.5pt or more for each candidate. (Early voting is now a problem for exit polls too, but it is factored in.) The level of precision people ascribe to these polls is simply not there.

Which is why we make averages, right? Yes, that does improve things, but if all the DRI (Democrat-Republican-Independent) mixes in each poll are systematically off in most of the polls in one direction, then that error will still be baked into the average rather than canceled out between them, which is the usual justification of the averaging in the first place.

So what I do is to separate the art and science parts of the poll by making two weighted averages.

The first is the percentage votes for each candidate by each party id sub-group. So I'll look at several polls, and maybe one of the will have Dems voting for Obama:Romney at 90:10, and another will have 92:8, and another will have 89:11 and then I'll weight those by sample size to get an estimate of how self-labeled Democrats are going to vote, and then do the same with Republicans and Independents. These numbers I really pay attention to, they are not opinions (of the pollster), and it doesn't even matter if the pollster is biased -- these numbers should still hold as long as the pollster isn't just fabricating data (which some of them clearly do, but mostly small-timers).

Then I take a separate weighted average of the DRI mix used in each poll to get a grand average of all the polls turnout estimate. And then I can put those two resulting sets of averages back together and do the math to come up with a weighted poll of polls (that is still using the internal models of the pollsters). If the pollsters are accurate in their models, this will give a very good projection. But I'm also free to ignore the DRI average and substitute what I might feel is a better turnout estimate and create a different projection that way -- substituting my art for theirs. Not all polls give out all the information I need to calculate the averages (or you have to pay for it), so I don't use Rasmussen and some others.

So the first question is: are the pollsters turnout estimates (DRI mix) any good? Well, that depends on whether you believe the exit polls -- it gets murky when you start comparing one poll to another to check for accuracy, especially since in states with early voting adding them to the exit poll is tricky business, and the turnout model of the exit poll could also be flawed. Whaddya gonna do? (And as we are about to find out, the polls turnout projection and the reported turnout on the exit poll are wildly different.) I've concentrated most of my analysis on Ohio and Colorado. If Romney wins those, he wins. I think it is safe to say that if he wins Ohio in defiance of the polls and wins Colorado also (which I actually think might be tougher) then it is a pretty good bet he is also going to win all those other swing states where he is tied or leading and so will have at least 275 electoral votes. Both Ohio and Colorado are heavy early voting states.

Let's look at 2008.

According to my average of the 2008 polls taken just before election day in Colorado, the pollsters projected DRI was 40:38:22. (Once again that's 40% Dems, 38% Reps, and 22% Inds) What did the exit poll say? 30:31:39. Not even close to the projection, and that is a trend I see in most of the 2008 data -- the polls projections have way too many Democrats and far too low Independents compared to the exit polls. And for Colorado anyway, that independent number of the exit poll is more plausible as more people in Colorado (where I am) do think of themselves that way, but they lean Democrat. Independents leaned Democrat in Colorado even towards Kerry in 2004, and are projected to so again this year which is one reason I think Colorado may be tougher for Romney than Ohio even though it appears he is polling better here. Anyway, even though the projection mix and the exit poll mix are very much different, the projections made using the pollsters' model in 2008 was actually spot-on. I usually normalize the numbers as if all votes went to one of the two main candidates, so I'm not looking at exact vote numbers. My normalized 2008 projection for Colorado using the polls turnout estimate: 54.7-45.2. Actual (normalized) results: 54.4-45.6. WOW that's close. That's closer than the exit poll itself predicted using its own DRI mix and vote-by-party id percentages. It predicted 53.3-46.7, which over-estimated McCain. Still, that could just an accident. What if I plug-in the exit polls reported DRI mix with my calculated numbers for how each group would vote? Then I get 55.1-44.9, which over-estimates Obama. Of course, if you are doing this before the election, you don't have an exit poll to give you a DRI mix -- you either use the pollsters or tweak it or base it on the previous election or just make one up.

2008 in Ohio pollsters projected DRI mix: 45:39:16. Again, independents so low. Exit poll reported mix: 39:31:30. That looks closer to reality for 2008. The pure poll projection for 2008 Ohio: 52.2-47.8. Actual results: 52.3-47.7. Wow, again. Predictions don't get closer than that. (Again, these numbers are normalized to sum to 100%.) I'm going to have a lot of trouble arguing against using the pollsters numbers for 2012, aren't I? They nailed this thing in 2008, at least in CO & OH. (Actually, I nailed it using their numbers, but anyway.) Exit poll prediction was: 53.6-46.4, over-estimating Obama. And using the exit DRI mix with the poll numbers for how those groups would vote: 53.9-46.0, over-estimating Obama even more. But to repeat, there is no exit poll before the election so that's just a curiosity. However, we would like to have some historical numbers to use in order to help guide us as we come up with a turnout projection for this year, so it would be nice if the exit poll numbers were accurate. No way of knowing.

And now 2012.

In Colorado, my current DRI projection from the polls is: 34:34:32. Not as rosy for Obama as 2008's 40:38:22, but in Colorado independents lean Dem so it still points to an Obama win: 50.7-49.3 (normalized), although a thin one. (He won by nearly 9 points normalized in 2008.) But certainly not a lock this time. What if we use the 2008 exit poll turnout numbers (30:31:39)? Then we get dead solid even -- Obama ahead by only a 1/3 of a normalized point.

So what happens in Colorado this year? By the pollster's numbers, a thin Obama win. By the 2008 exit polls, a tie, uncallable. Move any of the turnout projections towards Romney at all, and Romney wins. But Colorado is very tough -- population is expanding with people moving from other states, mostly to the urban areas. (Denver public school district is currently the fastest growing the nation.) Those new people probably lean Democrat, but conservative stronghold Colorado Springs is growing as well, and then we've got places like Aspen and Vail full of rich people and all the rural areas that always go red. And, enthusiasm for Obama *is* down overall if not necessarily among hard-core Democrats. Among independents it is definitely way down from 61-39 to 53-47. And enthusiasm for Romney is much higher than McCain among Reps: from 89-11 to 94-6. 94 is about as high as it gets for in-party numbers, and 89 is pretty bad actually. So I think the best scenario for Obama is that any possible (but not assured) favorable changes within the population in the state are canceled out by his lower popularity and Mitt's much better showing. Remember the exit poll in 2008 actually had 1 point more Republicans than Dems and with independents being higher than either, and over-estimated McCain. Let's assume the Dem-Rep mix was a rounding error and give 1 pt back to the Dems from the Reps. That gives us 31:30:39, which would have predicted the 2008 election spot-on. Using that mix for 2012, we get a projection of 51.0-48.9, almost exactly the same as the current polls projection of 50.7-49.3. Different roads to the same result.

Here's where the art really takes over. We have seen that the pollster's turnout projections compared to the exit polls turnout estimates are VASTLY different -- not even close. And they are all just polls. So at this point anybody inclined to argue simply that "pollsters know what they are doing, we should just go by their estimates" doesn't have a leg to stand on. You can't argue that pollsters are always right when their own polls are wildly differing on the equivalent figures. So is the sampling from an exit poll more representative of the voting population than pre-election sampling? Common sense says yes, and the exit polls have much larger samples than the normal pre-election polls. But still they could be skewed depending on where you do your exit polling, and how you account for early voting which means you are adding telephone samples back into the mix. Polling is more difficult than ever -- these figures just can't be that precise. Anyway, I choose the exit polls as more plausible and less arbitrary than the pre-election turnout estimates. You may disagree, and start throwing "historical accuracy" at me, but certainly exit polls historical record at predicting elections is at least as good pre-race polls, right?

Ok, so if a 31:30:39 DRI mix picked the 2008 election spot-on, and that same mix predicts a 2 pt win by Obama this time, what does that mean? I don't think anybody is going to argue that this election is going to be MORE favorable turnout-wise for Obama than last time, even in Colorado where maybe the Democrat population has increased. So Obama's best scenario is a thin win in Colorado. If he wins big in Colorado (he did win by 9 last time), then I am totally out to lunch and I apologize you've read this far. But we've got to ask how plausible is it that turnout will even be equally as favorable to Obama this time? Not likely, I think. To get an idea of what high Republican turnout in Colorado might look like, consider than the 2004 exit polls were 29:38:33 -- R+9. Such things are possible in this state. If turnout is like that, Romney wins by 7 points. But, that's probably not plausible either. Obama is an incumbent and still well-regarded among his base, not a lame-o loser like John Kerry who wasn't well-regarded by anybody. But similarly, enthusiasm for Romney is well above that for McCain. Does it approach incumbent George Bush 2004 levels? Maybe, but he is the challenger so let's assume not. Independents are tough to figure -- they haven't gone over to Romney en masse (they went for Kerry even), but they are way less sweet on Obama. That could just mean the ones that voted for him last time have largely decided just to stay home. I'm going to say independents are going to be closer to a 1/3 of the electorate this time rather than the very high figure of 39% reported last time. So let's lower independents, add some to Repubs, and also add some to Dems, but less than Repubs. Remember these are percentages so D's and R's can go up relatively simply by I's staying home. Basically I'm assuming that D's will turn out in the same numbers, R's will turn out in much greater number, and the independents will turn out less. How about 33:34:33? That gets us absolutely tied -- a re-count situation with Romney winning by 100 votes or something. These are not absolute predictions but just plausible scenarios -- more plausible I think than what the raw polling is giving us. I'm trying to be very conservative and be generous to Obama -- he's the incumbant and incumbants are hard to beat. But seriously, the range of the plausible goes from a very thin Obama win to a big Romney win. But the most likely is Romney by a point or two at least. But I do find Colorado hard to figure.

Objection: "But you're just pulling numbers out of your ass!" Well, yeah, that's the art side, and the pollsters do the same. That's my point. ALL TURNOUT ESTIMATES ARE PULLED OUT OF ASSES! So we look at history and current conditions, etc to guide us. But in the end there is always something arbitrary about them.

On to Ohio. It seems like it should be a slam-dunk for Romney, given that we are talking about Obama being tied in Colorado which he won by *9* last time, greater than the national average. He won Ohio by only 4 or 5 pts (depending on where you look -- I actually see at least 3 different results for Ohio 2008 from different sources), well less than the national average. And everybody agrees that Romney is winning Ohio independents handily. So what's going on? How could Obama possibly be winning Ohio?

Using all the pollster numbers gives us a projection of Obama winning by 3 points: 51.5-48.5. Using the 2008 exit polls, we get Obama by 4.5 points: 52.3-47.7. But the exit polls for 08 also over-estimated Obama in 08, so if we dial back the Dems a bit to give it a 37:32:31 DRI mix, that would have hit 2008 spot-on. If we compare the pollsters turnout DRI mixes from 08 to 12 it looks like this: from 45:39:16 to 38:31:30, which like in Colorado actually ends up slightly rosier than a slight tweak to the exit poll to get it in line. Their projection actually looks much more normal this time as the independent number isn't absurdly low anymore. Anyway, using my modified 2008 exit poll DRI mix, the projection is now Obama by 2 points: 51-49.

So again we ask, is it really plausible that the turnout in 2012 will be just as favorable to Obama as it was in 2008? Once again, let's see what a high Republican turnout in Ohio might look like. The 2004 exit poll reports 35:40:25. If we use that, Romney wins by 6. But, as in Colorado, probably more realistic is to split the difference while still favoring Obama. Moving the turnout rates only a single point from 2008 towards Romney to 36:33:31 gets us a absolute tie. Isn't it reasonable to think that the shift towards Romney will be at least that much? If so, Romney wins.

I would like to sum up now, but I'm sick of writing this, so we'll cover any confusion in later discussion.

And now a final wrinkle -- the big-ass storm. Did it help Obama? Probably. Will it tip it in his direction if it needed tipping? Maybe. I've got no answers for that, and there isn't enough time for the polls to really reflect it. But a bunch of very late polls moving towards Obama (or vice-versa) could be the real deal. I had expected the election to have obviously broken for one guy or another by now, so that still can happen very late storm or not, reflected in the polling or not. Gallup suspended national polling early due to the storm. Hard to say what the final effect will be, if any, since the hardest hit areas are already going for Obama. It did kind of stop the Romney campaign in its tracks and take him off people's radar. Who knows?


Here's some more crap to read on similar subjects:

http://www.redstate.com/2012/10/31/o...ewed-unskewed/


Discuss.

That final link I provided to redstate.com is definitely worth reading. (I hadn't actually read it until just now.) I could have just pointed to that.

Hmmm.....
 

Put a little thought into this, have we? :lol:

I'll never look at polls quite the same way again....... ;)

Is there a Cliff Notes version???

Why would you want one?

McLaughlin in that redstate article makes the very good point that these state polls are the *only* thing pointing to an Obama win as every other fundamental and historical indicator points strongly to Romney. An Obama win in this situation would truly be totally unprecedented.

I would have had to start this thread a week ago to type all of that. :cool:

I did!

tldr

Not tl;mr? Aww...

Great post GameTheory! :ThmbUp:

Given all this information, betting on Obama with all those different places that Lamboguy quotes at the recently listed odds sure looks like a huge underlay, even if he somehow does end up winning.

Gametheory:
I read your entire Post, er uh, Posts, and I read the redstate.com article. Both were informative and enlightening and I found very little to disagree with. I would also give caution to going against Silver since the Redstate article gave him a lot of credit for being a baseball statistical whiz. Both you and the article hinged your pro-romney argument on an enthused Republican turnout and a depleted Democratic turnout. This will surely be somewhat true, but how much? I am slightly critical of Silver also, in that unCommon sense tells me that anytime a candidate polls so close nationally and in razor thin state polls then he has a good chance of winning. Obama should be the favorite;he is certainly trending better, but I would only go 60-40. This 83% favorite rating that Silver gives Obama is stupid. Much of your model and the Redstate model compares the drastic differences between the 2004 and 2008 turnout rates. I am pretty sure the turnout rates for Republicans will be somewhere between these 2 extremes.
Thanks for the hard work and the informative posts. I always appreciate a
thread such as this that is based on sound and reasonable assertions.

This has been the really odd thing to me. I have a complete distrust of all media. I don't believe hardly any of it anymore. So the polls in the states are an enigma to me. I don't think you can buck the trends of history in a national election. There is tons of data to fall back on. If I was a betting man When it comes to elections I would go with the national historical trends. But nothing is the same as it used to be. The level of corruption is beyond anything I could have ever imagined. There are no real jounalists anymore. Nothing is the same. It could all be disinformation. In fact I am sure it is. Who the hell knows what is really going to happen.

When I start to think the historical trends are reliable I have to remind myself that the mainstream media is so invested in Obama that it may be completely impossible for Romney to win. The media controls the ill-informed at a level that could lead to there never being another Repub Prez. We may have finally crossed the threshold of the takers out numbering the makers. It could really be that simple.

If so, Ayn Rand's world from Atlas is just around the corner.

If you aren't familiar with Rand.....check it out.