Friday, 3 June 2016

Experiments in political science and the Cartwright critique

Over the course of the last couple of years, the political science discipline has twice hit the headlines for scandals linked to "field experiments." Maybe this isn't surprising: such experiments have become incredibly fashionable. Success in an academically fashionable endeavour can bring large rewards, and it's certainly plausible this has created incentives making fraud or poor judgement more likely.

To the extent that bad behaviour reflects incentives, one can always try to to police against it more vigorously. But changing incentives may be more effective. In this spirit, I'd like to encourage political scientists to stop being so damned excited by experiments and offering such big reputational rewards for them!

As a reason to calm down, consider some arguments (or great lecture version) from the brilliant philosopher of science Nancy Cartwright. Experiments (of the presently fashionable sort) rely on the logic of randomly assigning groups to "treatment" and "no treatment," so that any difference in outcome between the two groups can confidently be ascribed to the treatment. Yay, science!

Cartwright's core insight is that what such experiments can establish is only the role of a particular link in what might be a complicated, and highly context-dependent, causal chain. To build on an example she uses: Suppose you had a set of toilets, and assigned each of them randomly to have the lever attached to its side pressed or not. On completion of the experiment, you could confidently assert the relationship "lever pressing leads to water release." This formulation, though, would entirely obscure the point that these levers only release water because they are part of a mechanism to open a chamber supplied with water by pipes, etc.

Thus, if you went off to deploy your exciting new experimental result to solve California's drought by having everyone push the levers attached to the sides of their toasters, you'd be disappointed by the results. As Cartwright says (p.102), "Once stated this is an obvious and familiar point," but nonetheless one too often overlooked. This she effectively demonstrates with empirical examples of the disappointing performance of 'experimentally validated' policy interventions in new contexts.

So why is it that experimentalists are overlooking this obvious and highly consequential point?  [I'm not going to defend in detail the claim that they are, but will assert that the discussions about 'external validity' from experiment evangelists are not nearly searching enough.]  Let's use a little notation to make the argument more compact: the causes of an outcome O of an experiment are the experimental intervention I (such as lever pushing) + the rest of the mechanism M.  

So the question becomes, why the emphasis on I rather than M?

  • A lot of the methodological backdrop for political science experiments is drawn from experimental medical trials. In these, the common features of human organisms are regarded as similar enough that M will function in the same way. This assumption can be criticised even in a medical context, but for social scientists the issue is orders of magnitude more significant. 
  • Unlike pressing a lever, field experiments in political science are difficult to organise and often quite expensive. After all that effort to demonstrate the role of I, it's hard to remember that the M is important too.
  • I will often have been chosen precisely because it's the aspect of a broader mechanism that is easiest to manipulate. If the effects of M cannot be assessed via randomised controlled trials, then experiment absolutists will deny the possibility of making any meaningful claims about those effects. They haven't faced up to the fact that this means that they will never have any basis sanctioned by their own methodological precepts to assert that the results of one experiment have any generalisable implications whatsoever.

Whatever its origins, the mania for measuring the effects of interventions, and the corresponding neglect of the causal import of the context of these interventions, strikes me as very bad thing for many reasons, on which I hope to expand on another occasion.

PS: Cartwright's is not the only impressive critique of experimentalism on offer; I especially recommend Dawn Teele's edited volume. But so far, the critiques don't seem to have made much of a dent in the popularity of field experiments. Political science as a discipline seems to have an almost congenital need to affirm its 'scientific status'. But we should be suspicious of anything we need so much. How much did that ring really help Gollum?

No comments:

Post a Comment