What Do Experiments Prove?

Craig Holdrege

From In Context #3 (Spring, 2000)

This past summer three scientists from different labs described a remarkable set of experiments bearing on the hereditary basis of mouse behavior (Crabbe et al. 1999). Such research is fraught with difficulties. As a commentary accompanying the report by John Crabbe and his colleagues states, “No sooner has one group of researchers tied a gene to behavior when along comes the next study, proving that the link is spurious or even that the gene in question has the opposite effect” (Enserink 1999).

Crabbe and his colleagues wondered whether subtle differences at laboratories might be affecting experimental results. At three different labs — Crabbe's lab in Portland, Oregon, one in Edmonton (Alberta), Canada, and one in Albany, New York — the researchers went to painstaking lengths to establish and carry out exactly the same experiments. They used mice that were from the same genetic strain, exactly the same age, and raised under the same conditions. The mice were subjected to the same set of six behavioral tests — for example, learning to swim to a visible platform. The tests were done in the same sequence and each lab began each experiment on the same day and at the same local time.

In some experiments there were only small differences in the results at the different labs, while in others there were remarkable site-specific effects. For example, mice in Edmonton were generally more active than the ones at the other labs, and a particular strain of mice was very responsive to cocaine in Portland, but not at the other sites. A genetically engineered strain of so-called “knock-out” mice (with no receptors for the neurotransmitter serotonin) was more active in a maze experiment in Portland, less active in Albany, and proved in Edmonton no different from the control mice with intact receptors. These knock-out mice exhibited, however, about the same preference for alcohol at all three labs and did not differ from normal mice in this respect.

An Unexpected Result

This last result surprised the researchers because previously in Crabbe's lab the serotonin receptor knock-out mice drank much more alcohol than normal controls. He had repeated the experiment four times and come to the conclusion that the lack of serotonin receptors is related to addiction. He was on his way to finding an “addiction gene” the new experiments gave no indication of a predisposition for alcohol addiction among these mice. Were the mice in some unknown way slightly different from the ones he had used a few years before? Had seemingly insignificant features of the experimental conditions been unwittingly altered? No one knows. But for the time being the alcohol addiction gene has disappeared from the canvas.

Surprisingly, this kind of stringent replication of experiments is rarely done. As one scientist put it, “it's the kind of study that needs to be done, but nobody wants to be doing. You're looking into something that people would like to believe is not a problem” (quoted in Enserink 1999, p. 1599). The “problem” is that one cannot completely control and standardize life. Laboratory experimentation always creates artificial conditions, and the more one can control these conditions, the greater the likelihood of uniform results.

For example, inbred mice strains are used in behavioral tests because members of a strain are typically more uniform biologically and behaviorally than normal populations of mice. Of course, in nature inbred strains don't exist; they are an artifact of the laboratory setting. In one of the tests mice could drink either water or alcohol, giving them a clear “choice” that could be exactly registered and quantified. One could therefore attain precise results. But the question is, how much do these results tell us anything beyond themselves? Mice in nature are never confronted with such a situation. Does such an experiment help us to understand non-inbred mice in non-experimental conditions?

The alcohol preference test showed the most uniform results of all the tests, and unlike the other tests, it involved a bare minimum of handling by the researchers. In all the other tests there were more interactions between people and mice, making strict standardization very difficult. Crabbe related that one experimenter in Edmonton was highly allergic to mice and wore a respirator while carrying out the experiments. “That looks weird to us; it may look strange to a mouse too” (quoted in Enserink 1999, p. 1600). Only by minimizing all possible unwanted interactions can one approach the ideal of “the controlled, standardized experiment.” But the more this ideal is attained, the more isolated it is and the less it tells us about anything except itself. A mouse's life is full unexpected situations—the flooding of a tunnel during a rainstorm or the new cat on the block. Its life has little to do with experimental isolation.

Experiments Are Always Interactive

Once we come to the conclusion that the more controlled the experiment, the less it tells us anything pertinent relating to a natural situation, we begin to greet all announcements like “experiment proves link to cancer...” with a large dose of skepticism. We ask about the context: Was it an animal experiment? Under what conditions? Was the experiment trying to mimic natural conditions or was it very narrowly defined and controlled?

What, then, can we learn from experiments? We can learn that their results are context dependent. Knowing this, we will not naively transfer the results to another situation. It may be at first discomfiting to discover that experiments don't tell us about “the way things are.” But it can also be liberating to be freed from the illusion that scientific laws exist in a kind of metaphysical independence from the conditions of their discovery. We begin to look at experimentation as a method of interacting with phenomena in a precise way. What we find has validity, but it is not “the” truth. As Rudolf Steiner aptly put it, “We need to understand that every truth is valid only in its place, that something is true only as long as it is claimed under the conditions in which it is originally based” (Steiner 1899, p. 395).

The more experiments we carry out, circling a phenomenon and investigating it from different angles, the better we can see the modifications that occur with changing conditions. Our understanding of the phenomenon grows. But we also grow, gaining inner flexibility that helps us to expect the unexpected and appreciate the uniqueness of a new situation, as well as to modify our previously gained knowledge to fit the new situation. In this way experimentation becomes a way of training flexibility in thought and judgment, which, in the long run, is much more essential than believing one has “proven” something through an experiment.

References

Crabbe, John C. et al. (1999). Genetics of Mouse Behavior: Interactions with Laboratory Environment. Science 124: 1670-1672.

Enserink, Martin (1999). Fickle Mice Highlight Test Problems. Science 124: 1599-1600.

Steiner, Rudolf (1899). Ernst Haeckel und die `Weltraetsel', in Rudolf Steiner, Methodische Grundlagen der Anthroposophie (Dornach, Switzerland: Rudolf Steiner Verlag, 1989, pp. 391-402).