Followup to the previous post. Nate’s extensive article is too wide-ranging to paraphrase or summarize, but is a great overview of the problems of forecasting.

And some choice followup commentary:

Dan M.Grove, OK Nate, I really think you overstate your case. I’ll give an easy counter-examples to your statement that narrow theories are better than broad theories: the standard model of physics. From it, all weak intereactions and all of quantum electromagnetism can be derived. And classical electromagnitism has been derived from quantum electrodynamics. These theories have been verified millions of time. They are the basis for our understanding a wide range of technology, from electromagnetism to computers to lasers to quantum optics. You rightly point out that medical papers are often not reproduced. That is because they only need a 95% confidence level (or 2 sigma) to be published. And, since null results are rarely published, its easy to have 19 random unpublished result, and 1 random published one. When charm was found, it was published with a 5-sigma statistical signal. It was reproduced immediately. These are broad ranging theories that have been well identifed. If you want a political science result to be verified, it should be something that isn’t just something that can be restated N different ways, that has stable results when you change the question slightly. In particular, it is a big plus for the theory if you offer a skeptical colleauge the right to reset the question and then recomute the results. Then the results should have less than a 1 in 100 chance of being found randomly. 1 in 1000 would be much better.

Nate Silver, Brooklyn, NY Dan, You make some excellent points. In particular, one of the things I found very problematic when I began to examine the elections “fundamentals” models is that they were not very robust to small change in assumptions. Replace an economic variable with one that is normally closely correlated with it, and you will get a substantially different result in certain elections. But I think one needs to be careful about drawing analogies between the physical and the social sciences. One of the things that characterized Tetlock’s hedgehogs was that they saw the political system as more analogous to a noncomplex (perhaps even Newtonian) physical system than the foxes did. This can sometimes cut the other way as well. For instance, there are some criticisms of global warming forecasts that would be reasonably compelling if they were tantamount to social science predictions, but don’t work as well when the causality of the greenhouse effect, etc. is relatively well understood (although, I certainly don’t claim that global warming forecasts are above criticism or without their problematic elements). Then again, it’s interesting that a lot of Bayesian probability theory really originated with Laplace, who thought that even though the mechanisms understanding the universe were extremely regular, our ability to measure and understand them precisely might not be.

Richard, NY Nate, I think you are confusing complexity with uncertainty in formulation. The weather/climate system is immensely complex and involves a massive number of interactions and feedbacks. Most of those interactions however are reasonably well understood and can be derived from the laws of physics. Social science models on the other hand are complex but also subject to fundamental lack of understanding of the basic interactions involved. This manifests itself in the parametric sensitivity you mentioned. Weather models are not subject to anywhere near this degree of parametric uncertainty even though they are proably more complex. Indeed the largest numerical models in the world are weather/climate models.

Dan M.Grove, OK Thanks for your reply Nate. You are absolutely right that facile comparisons between physical sciences and social sciences are extremely dangerous. But, when you included medicine, I wanted to point out that broad statements can have enormous predictive power. Ecconomics and social sciences are causally dense. So, it is hard to make broad quantitative statements. Still, I don’t think that the attempt to make fields like interenational relations more like science by so limiting the scope of one’s study to make it barely useful is the answer either. People like Huntington still provided insight, even thought they weren’t quantitative. It’s interesting that you mention Laplace because physicists talk about the Laplacian ilusion; since QM shows the inherent indetermancy of physics. Indeed, some measurable properties cannot exist apart from measurement (e.g. electron spin at N degrees). Finally, while it is hard to make general, robust, high probabability statements in the field of political science, it is not impossible. It’s just that most folks in the social sciences, and many in medicine, alas, think they’ve done it when they haven’t. Part of it is the way statistics are improperly treated. Being one of the first scientists who learned his craft when Monte Carlos were reasonably priced, I understand something of the pitfalls and the ways around them. So, I agree, most of the supposedly precise general statements in the social sciences aren’t…but a few are.

Kris NedzynskiWarsaw, Poland Nate, I would be extremely eager to hear your opinion on Jude Wanniski’s approach to prediction in politics. I believe he was a true genius, largely still waiting to be appreciated. He applied Hayek’s “dispersed knowledge” concept (actually it can be traced back to Aristotle) to politics. Through that lens he was analyzing American and global political events, often with amazing accuracy. For instance he warned Senator Jesse Helms in 1998 that if the US government would not start to study origins of terrorism instead of merely defending against it, “terrorist mind will succeed in taking two towers completely”. Wanniski’s theory was laid down in kis “The Way The World Works”, but these may serve as a brief summary: http://tnij.org/q72t http://tnij.org/q72o http://tnij.org/q72p

DMC, Lucerne, Switzerland Are you aware of the “Good Judgement Project?” http://goodjudgmentproject.com/ In brief, these academic researchers asked teams of volunteers to make predictions about a range of international events. The teams varied in the training they received. Results of the predictions were scored in a rigorous way that benefits those who are aware of the uncertainties in their own predictions. I participated as a forecaster in the first year of the project, and found it a very interesting exercise. I am very much looking forward to seeing the papers that will document the results and conclusions.

valleyforge, Valley Forge, PA Political science is a soft science precisely because it is immune to reliable modeling and hence prediction. Humans are not billard balls or even charmed quarks and the society we’ve created is not governed by immutable laws. Hegel, Marx, and Asimov’s fictional psychohistorian Hari Seldon may have believed in deterministic history but the “great man” problem will always defy the models. For illustration just look at the decisive influence that one man’s vote, Anthony Kennedy’s, has on national policy for 312 million people. Or the many unexpected inventions and discoveries, and even disasters, that changed the course of history. Such black swan events by definition cannot be predicted but have tremendous consequences. Judging political scientists who seek merely to explain by the accuracy of their theories implied predictions is inappropriate as it expects that their field of inquiry is ultimately knowable when it is not. Conversely political scientists who have the temerity to make inadequately-qualified predictions deserve the ridicule they get.

Gyre, Pennsylvania I’m not so sure that predictions can’t be made without statistics. I wasn’t at all surprised when the Mali coup occurred, it fit the pattern of coups across the world for the past sixty years. A dangerous separatist conflict, a past history of coups in the region, a fairly poor country and the perception that the civil leaders couldn’t handle it. Admittedly I didn’t predict the coup surviving as long as it has, but predicting the coup itself isn’t so bad. In the interest of fairness to numbers, political scientist Jay Ufelder has a post on using certain numbers and criteria to predict the nations most likely to have coups. http://dartthrowingchimp.wordpress.com/2012/01/30/assessing-coup-risk-in-2012/ Also political scientist Daniel Drezner has an article in response to Stevens. He’s not impressed. http://drezner.foreignpolicy.com/posts/2012/06/25/when_a_stupid_op_ed_produces_some_smart_debate

Richard, NY Some context from the physical sciences might be useful: Weather forecasts have been probably the most successful predictions in the context of large unavoidable uncertainty. It is interesting to look histocially at their skill and the reasons for its slow improvement over decades. Initially (50s-70s) this occured due to better model formulation. Such models were very firmly grounded in well established physics. Later (80s-present) it was due to more complete observational data for prediction initialization. So the lessons would appear to be: 1. Work out the basic rules governing the dynamics of the system of interest. 2. Once step 1) has been achieved, hit the system with tons of data. My sense as a physical scientist is that social/political scientists have a lot of trouble with point 1) and resort to point 2) almost in desperation. Personally I would think a lot more about the basic behavioural dynamics that set political opinion. Economic factors are pretty clearly important but not completely explanatory…..

wheelers, denver Nate, you are part of the solution. the network of wicked problem solvers. http://pressthink.org/2012/06/covering-wicked-problems/