I. Standards of Evidence and Proof
In epistemology, as in life, it is helpful to set one's aspiration level high, but not too high. On the one hand, the early pioneers of the scientific approach to world politics were probably too optimistic (Richardson, 1960). We will never have clock-like Newtonian laws of world politics that predict the waxing and waning of great powers or the trajectories of arms races. On the other hand, post-modernist debunkers of scientific approaches have probably gone too far (Ashley, 1988). Predicting political events is not as hopeless as predicting the shape, color and size of the clouds in the sky next week. That said, however, there are good reasons for supposing that there is limited potential for discovering powerful laws that support accurate long-term forecasts (Almond and Genco, 1978; Jervis, 1992b).
A. Grounds for Scientific Pessimism
(1) the tape of history runs only once. Psychologists have grown accustomed to the inferential luxury of control groups that allow them to assess whether the hypothesized cause really made a difference. In world politics, the control groups exist--if "exist" is the right word--only in the imaginations of political observers who try to reconstruct how events would have unfolded if the hypothesized cause had taken on a different value in an alternative world. Could we have averted World War II if Churchill rather than Chamberlain had confronted Hitler at Munich in 1938? Could we have triggered World War III if Kennedy had followed his more hawkish advisors and launched air strikes against Soviet missile sites in Cuba in 1962? Would the newly industrializing countries of the 1970's and 1980's have grown even more rapidly if their governments had pursued less interventionist economic policies? Time-machines fantasies to the side, there is no way to rerun history and experimentally manipulate the presence or absence of a "key" personality, event, or policy.
Whereas experimental control is simply impossible, statistical control is possible in principle, but often deeply problematic in practice. For many categories of questions--such as the role of nuclear weapons in preserving the long peace (1945-1991) between the U.S.A. andU.S.S.R.(
Gaddis, 1993)--there are too many confounding variables and too few degrees of freedom to disentangle competing casual claims. Judicious selection of comparison cases and meticulous process tracing of decision-making records can sometimes tip the scales of plausibility in these statistically indeterminate cases (George, 1980; Khong, 1991), but any causal inferences will still ultimately rest on counterfactual assumptions about what would have happened in possible worlds in which the hypothesized independent variables took on alternative values from those in the actual world. The dependence on thought experiments is not, however, adequately appreciated (Fearon, 1991) and is regarded by many as embarrassing, undercutting the validity of all casual claims about world politics. This reaction is understandable but exaggerated. To say that debates over security issues ultimately reduce to competing counterfactual scenarios is not to concede that anything goes. It is possible to articulate standards of evidence and proof even for counterfactual arguments (Tetlock & Belkin, 1996). Among other things, good counterfactual arguments should have clearly specified antecedents, consequents and connecting principles, should not rewrite massive stretches of history, should rely on connecting principles that are consistent with well-established theoretical and empirical generalizations, and should contain the seeds of testable hypotheses in the actual world.
(2) many plausible causal candidates. When scholars perform thought experiments to explore what might have happened under this or that contingency, they often discover a plethora of possibilities. Half a dozen schools of thought may stand ready to advance their preferred causal candidates and to assert confidently that they know how events would have worked out if the hypothesized causes had taken on different values. These causal candidates are drawn from the full spectrum of levels of analysis. It is instructive to observe their interplay in two actual controversies: the debates over blame for World War I and credit for the Asian "tigers."
(I) World War I. For nearly a century, scholars have debated the origins of the ironically labeled "war to end all wars." Some claims invoke "butterfly-effect" counterfactuals that, in the spirit of chaos and complexity theory, stress the role of quasi-random contingencies in shaping events (cf. Gaddis, 1993). A classic example is the precipitating event--such as the assassination of Archduke Ferdinand in June 1914--which can be easily "undone" by simply positing that the driver of the royal carriage possessed a map of Sarajevo and did not make a fateful wrong turn that gave the Serbian assassins who had just botched the job a remarkable second chance to do it right. Other claims invoke cause-effect generalizations drawn from traditional disciplines. Some psychobiographers suggest that German foreign policy would have been more prudent if Kaiser Wilhelm had not been so insecure (perhaps because of his withered arm) and determined to assert his manhood (Kohut, 1982). Students of crisis decision-making suggest that time pressure and information load promoted simplification and rigidity of thought (perhaps preventing policy-makers from generating complex compromise agreements that might have averted war--Holsti, 1972). Students of military doctrine and organization assign blame to the widespread "cult of the offensive" that led key planners to believe that the side which mobilized first possessed a decisive strategic advantage (Snyder, 1984). Students of identity-politics trace the conflict to the inherent instability of multi-ethnic empires such as Austro-Hungary and to the resulting power vacuum. Neorealist analysts of international systems point to the inherent instability of multipolar balances of power and the threat to that balance posed by the rapid growth of German industrial and military strength. In this view, World War I was a conflagration waiting to happen. The Sarajevo assassination was but one of countless sparks that could have easily set off the same underlying conflict.
(ii) The emergence of the "Asian tigers”. Almost no social scientists in the 1950's predicted that the astonishing growth rates of East Asian economies in the late 20th century. But almost all social scientists in the 1990's can generate four or five reasons for the inevitability of those same growth rates. Advocates of cultural explanations can point to the "work-ethic" character traits inculcated by Confucianist family values: hard work, in-group loyalty, and willingness to subordinate immediate gratification of individual desires to long-term group goals (Pye, 1985). Advocates of strategic trade can point to government subsidies and nurturance of "infant industries" in high-growth-potential sectors of the economy (Johnson, 1993; Tyson, 1993). Advocates of neoclassical economic theory can point to intense competition within many of these "infant industries" and to tax laws and low-inflation macro-economic policies that encourage high savings rates that, in turn, provide large pools of low-interest investment capital (Friedman, 1992). Defenders of authoritarian government as an essential transitional phase toward prosperity can point to the importance of suppressing unions and maintaining low wage rates in labor-intensive industries as an early source of comparative advantage in international trade.
What do World War I and the economic miracles of East Asia have in common? In each case, we find a long list of distinct but interrelated causal candidates, each of which rests on difficult-to-test assumptions about what would have had to be different to alter the observed outcome. In each case, an event that virtually no one predicted appears, with benefit of theoretical hindsight, hopelessly over-determined (Fischhoff, 1975; Hawkins & Hastie, 1990). Finally, in each case, people often find it very difficult to recapture the sense of uncertainty that prevailed before the historical outcome was known. We exaggerate the degree to which we “knew it all along.” Our capacity to assimilate known outcomes from the past to favorite causal schemata vastly exceeds our ability to predict unknown outcomes in the future (cf. Dawes, 1993).
(3) the interrelatedness of potential causes. In the ideal thought experiment, we manipulate one cause at a time and gauge its impact. Part of what makes "assassination" counterfactuals so popular among chaos and complexity theorists is that it seems so easy to rewrite one or two trivial details of history, to hold all else constant, and to observe big effects. All we need to do is to suppose that Lee Harvey Oswald was not quite so good a marksman and most of us share the intuition that we would have a rather different list of American presidents from 1963 onward. The thought experiment seems to illustrate the "sensitive dependence on initial conditions" of major events (McCloskey, 1991).
Thought experiments are, however, more problematic than this laboratory model suggests. Causes rarely exist in isolation from each other. When we tamper with one potential cause, we almost always alter a host of others, thereby creating confounding variables (Jervis, 1996). For instance, if we counterfactually posit slower German growth rates in pre-1914 Europe, we simultaneously change the entire geopolitical calculus. Perhaps Britain would no longer perceive Germany as the power most likely to achieve European hegemony but rather would see France as the primary threat (as in Napoleonic times) or Russia (as after 1945). A shift in the domestic political or economic condition of one state may have far-reaching ramifications on alliance structures. Game theorists have been most explicit in modeling these sorts of "ripple effects" by mapping out the best responses available to other players in the event that one player (for whatever reason) deviates from the equilibrium path (Bueno de Mesquita, 1996; Weingast, 1996). To reach determinate counterfactual conclusions, however, these game theorists must make heroic assumptions both about the rationality of the players and about the assumptions that the players themselves make about each other's rationality (necessary for identifying equilibrium strategies via backward induction).
If world politics is best represented as systems embedded within systems and if it is generally inappropriate even to imagine manipulatingan
hypothesized cause in isolation from the causal network within which it is embedded, we confront an extraordinary dilemma. For the more densely interconnected the potential causes, the less possible it becomes to trace the impact of any change even after the fact, less still to predict it before the fact. From this standpoint, the most popular research strategy for disentangling cause and effect in world politics--the comparative case method--is deeply, perhaps fatally, flawed. Searching for several cases that are similar except for one "independent variable" is systematically misleading. Not only is there no random assignment to conditions, there is the problem of path-dependence (Jervis, 1996). Our location in the historical flow of events is consequential. What we do now is shaped by what happened earlier. These earlier branching-point events have taught us particular lessons and values. For example, when we compare the consequences of pursuing deterrence versus reassurance policies (a popular comparison in political psychology), we need to factor into our intuitive causal model the variety of reasons why people found it reasonable to resort to deterrence in certain cases but to reassurance in others. Deterrence may be an effect as well as a cause--a sign in itself of how far the relationship had already deteriorated. Interpretive controversies of this sort surface frequently in security debates.
The strong form of the "system-effects” argument leaves us teetering on the brink of policy nihilism, with no way to tell what might have happened if we had listened to one or the other faction in a policy dispute. Of course, the strong form of the argument may be too strong. In the cosmic matrix of causal interconnections, most entries may be close to zero (cf. Pattee, 1973), in which case the indeterminacy problems are less acute. The only sure fact is that no one knows for sure. Firm opinions must be based on metaphysical guesses.
(4) counterfactuals are often politically controversial. A fourth factor complicates efforts to make sense of world politics. Most debates over counterfactual scenarios engage partisan political motives. For instance, defenders of the Reagan defense build-up of the 1980's argued that, without it, the Soviet political establishment would never have accepted as radically a reformist leader as Gorbachev--a leader whose policies of glasnost and perestroika "led to" the disintegration of the Soviet state (Pipes, 1993). Critics of the Reagan administration argued that the defense build-up was either an irrelevancy or an impediment to Soviet reformers and--in either case--an egregious waste of national treasure (Garthoff, 1994; Lebow & Stein, 1994). For our purposes, the key point is this: one can neither sensibly defend nor compellingly criticize a policy initiative without making assumptions about how things would have worked out differently if the government had done something else.
Short of adopting a stance of radical agnosticism toward all policy initiatives, something few social scientists are willing to do, there is no avoiding the daunting task of setting thresholds of proof for counterfactual thought exercises. One must judge the likelihood of making Type I versus Type II errors (e.g., concluding that deterrence works when it does not versus concluding that it does not work when it does) and the risks associated with each error, all the while under partisan pressure to tip the scales of plausibility in one direction or another.
(5) too much is at stake to confess ignorance. Putting political pressures aside, human beings find it dissonant to acknowledge that they are making extraordinarily consequential decisions with little knowledge of the likely consequences. Insofar as people need to believe that they are masters of their destiny or, at least, that they live in a comprehensible and predictable world, they will bolster their confidence in preferred counterfactual scenarios and downplay rival scenarios. They will also tend to dismiss indeterminacy arguments as nihilistic carping. Indeed, they will find it aversive to be reminded of the soft counterfactual underbelly of their belief systems. Who wants to be told that deeply held convictions rest on speculative assumptions about what might have happened in imaginary worlds?
In sum, there are powerful temptations--cognitive, emotional, and political--to claim to know more than one does or even can about world politics. Part of the problem is how few causal claims can withstand systematic scientific scrutiny. There are usually many plausible accounts and imperfect means of gauging their relative credibility. Part of the problem is the magnitude of the policy consequences and the awkwardness of confessing how little one knows when so much is at stake. The scholarly study of world politics seems to require a superhuman capacity to tolerate ambiguity and to resist the siren calls of moral-political advocacy.
This argument might be mistaken for a counsel of despair, perhaps even a postmodernist critique of the quest for causal laws in both social psychology and history (cf. Gergen, 1978). It is one matter, however, to acknowledge candidly the difficulty of the task and quite another to abandon it altogether and replace it with a hermeneutic agenda. Awareness of epistemological traps and political temptations helps to calibrate realistic aspiration levels for research. The classic goal--well-defined covering laws that allow us to deduce past events and to predict future ones with great accuracy (Hempel,1965)--
is outside our reach now and perhaps forever. A more modest goal--cumulative multi-method research programs that allow us to identify recurring patterns in the past and to anticipate some future events with greater than chance accuracy--is within our reach. Indeed, the proof is the research literature.
B. Grounds for Scientific Optimism
Given the scale of the indeterminacy problems in world politics, how can researchers build a persuasive case for a particular hypothesis? This chapter argues for a three-pronged strategy in which: a) researchers draw upon middle-range theoretical generalizations that rest on cumulative empirical work in other fields as sources of causal hypotheses; b) researchers make good-faith (reasonably value-neutral) efforts to test the applicability of these generalizations to various facets of world politics; c) researchers remain vigilant to the possibility that the transfer of knowledge is not one-directional and that "basic" psychological findings may have to be seriously qualified or even reversed by political, economic, and cultural moderator variables.
With respect to the first requirement, the wide assortment of "basic-process" chapters in the volume should serve as evidence that cumulative bodies of work exist on such relevant topics as judgment and choice (Dawes, Handbook Chapter) bargaining and negotiation (Pruitt, Handbook Chapter) social identity maintenance and intergroup conflict (Brewer & Brown, Handbook Chapter) and perceptions of fairness within and across group boundaries (Tyler & Smith, Handbook Chapter). Prima facie relevance is not enough, however, to clinch the case. Thoughtful theorists deny the relevance of laboratory studies on one or another of the following grounds: a) selection arguments (to make it into high-level political roles, one must be a lot more mature and rational than the typical sophomore participant in laboratory studies); b) motivational arguments (policy-makers are especially motivated to make rational decisions because the stakes are so high); c) accountability-constraint arguments (even if policy-makers were prone to the same effects as laboratory participants, they work within complex institutional systems of checks and balances that prevent those effects from being translated into policy).
Each argument raises subtle issues concerning boundary conditions for experimental findings. Here the key point is the necessity of obtaining independent evidence that the hypothesized psychological processes are indeed operating in the political world. The crucial question becomes one of multi-method convergence: Do we obtain similar functional relationships between the conceptual independent and dependent variables studied in laboratory and real-world settings? Consider three examples:
1) laboratory studies have revealed that people tend to shift into "simpler" modes of cognitive processing as information load, time pressure, and threat exceed some optimal point (Gilbert, 1989; Kruglanski & Freund, 1983; Streufert and Streufert, 1978; Svenson and Maules, 1994). Archival researchers, relying on content analyses of political statements, have concluded that similar processes have occurred in numerous international crises (Raphael, 1982; Suedfeld, 1992a; Wallace & Suedfeld, 1992; Walker & Watson, 1989, 1994).
2) laboratory studies of bargaining and influence reveal that threats often either do not work as intended or even backfire. One recurring theme is the relative effectiveness of some version of the tit-for-tat strategy that protects one from exploitation but leaves the door open for reconciliation. Qualitative case studies and quantitative event-analytic studies of international relations often converge on strikingly similar conclusions (Leng, 1993), as do computer simulations that pit all logical combinations of strategies against each other (Axelrod, 1984).
3) laboratory studies of judgmental biases reveal a host of mistakes that people apparently make in drawing causal attributions, estimating relationships among variables, revising views in response to new evidence, and attaching confidence to their judgments (Fiske & Taylor, 1991; Nisbett & Ross, 1980). Many case studies of the foreign policy decision-making process suggest that policy-makers fall prey to the same effects (Jervis, 1976; Lebow, 1981; Snyder, 1984; Vertzberger, 1986; Wirtz, 1991).
When it arises, and it does frequently in this chapter, multi-method convergence is generally taken as an encouraging sign. The working assumption is that theoretical generalizations that pass radically different tests stand a better chance of capturing robust regularities than do generalizations whose support is confined to one genre of research. Controlled experiments reassure us of the internal validity of the original causal claim whereas field methods--case studies, content analysis, event analysis, codifying expert judgment--reassure us that the hypothesized process holds up in the hurly-burly of world politics.
There is much to recommend this methodological division of labor and I have indeed endorsed it elsewhere (Tetlock, 1983). But there are good reasons for caution. First, convergence is sometimes spurious. Field researchers may conclude that policy-making that only superficially resembles a laboratory analog is the product of the same underlying process. For instance, policy makers may appear to rely on crude rules of thumb in drawing lessons from history (a finding consistent with laboratory work on analogical reasoning -- Gilovich, 1981). Policy makers may, however, actually possess a farmore subtle
grasp of the situation. They may be using simple historical arguments such as "no more Munichs" or "no more Vietnams" to achieve impression management goals: to rally support from wavering political constituencies and to pre-empt potential critics. In a similar vein, policy makers may not be unaware of value trade-offs or of contradictory evidence but may find it politically useful to refuse to acknowledge them. Distinguishing perceptual-cognitive from impression management explanations is often a tricky judgment call even in controlled laboratory settings (Tetlock & Manstead, 1985); the indeterminacy problems are obviously more severe in historical case studies.
Political psychologists, nonetheless, frequently make such judgments. Holsti (1989) made such judgments in assessing whether crisis-induced stress really impairs policy reasoning or whether policy makers are trying to influence the calculations of other national leaders by persuading them that such impairment has occurred (see Schelling, 1966, on the rationality of occasionally appearing irrational). Stein (1991) made similar sorts of judgments in assessing whether leaders of states that challenge deterrence are allowing motives and wishes to inflate the perceived odds of success or whether they are trying to intimidate the status quo power by persuading it that the challengers have no choice (given their public commitments) but to persevere with confrontational policies. Fischhoff (1991) and Sagan (1985) grappled with a similar dilemma in the domain of nuclear command, control and communications systems. How can we assess whether decision makers responsible for operating these systems overestimate their ability to simultaneously avoid Type I errors (falsely conclude that an attack is occurring) and Type II errors (falsely conclude that an attack is not occurring) or whether these decision makers are self-consciously promoting a functional fiction that the system is in sound working order (so that the public will not panic and that adversaries will not test the system) .
In short, it is no simple matter to distinguish true from spurious multi-method convergence.
There is also a second reason for caution. Multimethod divergence sometimes occurs. It is easy to identify real-world exceptions to most laboratory-based generalizations. Decision makers sometimes draw flexible, multidimensional lessons from history (Neustadt and May, 1986), confront trade-offs even in highly stressful situations (Maoz, 1981), and display a willingness to change their minds in response to new evidence (Breslauer & Tetlock, 1991). Multimethod divergence of this sort does not, of course, mean that one set of findings must be right and the other wrong. When different research methods yield different results, there are lots of possible explanations. How people think may depend on a variety of boundary conditions: individual differences in intellectual capacity, cognitive style, and interpersonal style, cultural background, and institutional variables such as the nature of the decision making task, small groups processes and role and accountability relationships. Each class of variables--by itself or in combination with others--may explain inconsistencies in the evidence.
Finally, a third cautionary comment merits mention. There is inevitably an element of subjectivity in judgments of multimethod convergence and divergence which creates potential for both the appearance and reality of political bias. There are no hard and fast rules that specify how quickly one should conclude that "convergence is specious" or "divergence constitutes falsification." When the hypotheses at stake are as politically consequential as "conservatives are more prone to cognitive biases than liberals” (Kanwisher, 1989) or "deterrence typically does not work" (White, 1984), the stage is set for epistemic mischief on an epic scale. Most social scientists are well known to be liberal in their political sympathies (Lipsett, 1982) and the suspicion inevitably arises -- justifiably or not -- that they are selectively raising and lowering standards of evidence to favor certain hypotheses (cf. Tetlock, 1994). Under these inauspicious circumstances, the scholarly community can best protect its reputation for reasonably fair and value-neutral scholarship by affirming its commitment to a level field for hypothesis-testing. "Turnabout” thought experiments should become routine mental exercises in the peer review process in which skeptics are encouraged to make the case, for example, that "if liberals instead of conservatives displayed a certain judgmental tendency, we would have been more reluctant to label the tendency a bias" or "if the predictions of the conflict-spiral rather than deterrence theory had been refuted, we would have more vigorously challenged the logical adequacy of the test". The key questions become: Have the rules of evidence been "stacked against" unpopular points of view? Do we require exceptionally strong evidence to challenge popular hypotheses and accept remarkably weak evidence to reject unpopular hypotheses? Do we, in brief, fall prey to the same theory-driven biases of thought of which we accuse others? And what scientific accountability mechanisms can we create to check such biases and adjudicate claims of epistemic double standards in so methodologically eclectic a field as world politics?
Notwithstanding these qualifications, multi-method triangulation still provides the soundest basis for causal claims in world politics. In the following sections, I shall be explicit about the evidential basis for various claims, giving extended attention to those that enjoy multi-method support.
0%
Author: Philip E. Tetlock
