How Did Pundits Get it So Wrong

The size and shape of 1994 Republican victories came as a big surprise to pundits and political scientists. Despite savvy and sophisticated surveys, most of us wildly underestimated Democratic losses. Of twenty-one pundits who made predictions on election eve, only four guessed Republicans would control the House of Representatives. Predicting the Republicans' net gain in House seats, every one guessed too low. Zero for twenty-one.

Make that zero for twenty-two. One of us, (David King, a political scientist) studied hundreds of congressional races and also came up with a too-low prediction.

The other (Richard Zeckhauser, a decision theorist) paid little attention to the campaigns and announced that the pundits had a good chance to miss the mark. Were there a parimutuel on net seat gains, he would have bet on the long-shot outcomes: small Democratic losses or a Republican landslide. It is not that the pundits had no chance of being right. But they had a much better chance of being wrong than they probably realized.

The table shows predictions of twenty-one pundits, recorded on November 6, 1994. The next day, voters gave Republicans a majority in the U.S. House.

In Reality....	52
Morton Kondracke	46	PUNDIT Average	32
Bob Novak	42	Mark Shields	32
Mary Matalin	41	Fred Wertheimer	31
John McLaughlin	40	Bill Schneider	30
Marty Tolchin	36	Ron Walters	30
Frank Luntz	36	Ralph Reed	29
Fred Barnes	36	Eleanor Clift	28
Charles Cook	35	Margaret Carlson	28
Chris Matthews	34	Jack Germond	26
Doug Bailey	34	Ronald Lester	26
Al Hunt	33	Alan Abramowitz	25

What tripped up the pundits, and why aren't decision theorists surprised? There are at least three reasons, and they are worth remembering for those who must make or rely on predictions.

First, there is a "herding instinct" among decision makers, including so-called experts. Academic studies going back many decades demonstrate this. In experiments, individuals cluster when asked to estimate the length or weight of an object.

Moreover, public experts -- such as our pundits -- get another benefit from clustering: they gain protection in case their guesses are way off. "After all, everyone else got it wrong." There is safety in a herd, a phenomenon widely known in relation to Wall Street analysts of corporate earnings.

Second, election outcomes are related to each other in a particular way. Consider two toss up races. If a Republican wins the first, it is more likely that the Republican in the other race will win as well. In the language of the statistician, there is a positive correlation in the error terms associated with initial predictions.

Pundits frequently overlook this point. Their expertise is expressed through detailed analyses of individual races: Whom did the newspapers endorse, what are the personality flaws, what do the polls say, etc. To the extent that voters are moved by national concerns, such as the Contract with America, or Bill Clinton's Mideast trips, there will be common factors influencing all races.

In a sense, the country did not run 435 little races for the U.S. House in November, 1994. The country ran one big election. Little race-by-race predictions were swamped by national trends, and that kind of national impact on local races is more common than many suppose.

The problem of correlated errors across races is worse for pundits now than it once was because the news media have helped nationalize politics. With television, most voters know substantially more about the personal and political life of the president than of their senator or representative. That image affects elections.

National trends are also more sweeping today because regional and economic barriers are going away. Not long ago, it was reasonable to think of America as a patchwork of blocs, like the farm bloc, the rural South, and the industrial heartland. They muted national trends. Today industry is found virtually everywhere in the U.S., the service sector is omnipresent, and farmers comprise a tiny fraction of the electorate. Regional values are converging. Congressional districts are becoming more alike, and that makes for more volatile predictions if pundits miss a national trend.

Third, pundits make point estimates, not probability estimates, and that is a big problem. "The Republicans will gain 32 seats," one might say. But if we want to know the likelihood of a Republican landslide or, on the opposite extreme, a mere modest Democratic loss, then we want to know how likely these outcomes might be.

A sophisticated prediction might be: "On average I expect the Republicans to gain 36 seats, but I believe there is a five percent chance they will pick up more than 42, and an additional five percent that they will gain fewer than 28. With such information, assuming it was well calibrated, we would recognize whether a landslide was perceived to be likely, unlikely, or virtually impossible.

Decision analysts have long pushed for prediction methods that ask individuals to define some critical points in their distribution of outcomes, not just the mean. The National Weather Service reports probabilities. Pundits should too. Even if they did, they would likely guess in too narrow of a range.

In a typical decision analysis experiment, individuals might be asked to guess numbers like the population of Finland, or the paper diaper consumption of the U.S. Then they would be asked to define their surprise points, or their 1st and 99th percentiles. These are amounts such that the individual thinks it is only one percent likely that the true value lies above an upper or lower surprise point. They should be surprised two percent of the time, on average.

Experience has shown that individuals assess such distributions far too tightly. As a rule of thumb, people are surprised roughly one-third of the time. Pundits are people too, and we expect their election predictions to be no different.

What pundits might describe as a severe long-shot, 100 to 1 against, might happen fifteen or twenty percent of the time. Something like that probably happened to the 1994 election predictions, but it is harder to articulate a probability distribution than the quick grunt demanded on "The McLaughlin Group."

Pundits are prognosticators, not seers. Their predictions would improve, we believe, if they were forced to place bets in some parimutuel system. 1994 would have been an exceedingly profitable year for individuals who recognized that herd instincts, correlated errors, and overly tight distributions were sure to be found in the predictions of experts.