Experimental design

Careful consideration must be given to experimental design to avoid confounding effects and biases, and to ensure that experimental treatments can be detected.

 
 
Experiments can provide important insights into causation, but proper insights require proper experimental design.
 

Experimental Design

Observational associations between variables do not necessarily imply causation. Associations can be caused by a third ‘lurking’ variable. Experiments provide an efficient way to test for causation, but the quality of our inference will depend on the quality of our experimental design. Care needs to be taken in the design of experiments, from the allocation of treatments to the way in which we perform our manipulations.

When performing experiments we need to make sure that we are manipulating only the hypothesised agent and that subjects and observers are blind to treatments.
 

Experimental Pitfalls

When we perform experiments we need to be sure to manipulate only the putative agent of causation. In some cases, this manipulation requires other vehicles or procedures that we need to control for with a ‘sham’ manipulation. We also need to beware that humans are particularly bad subjects and observers and we need systems to blind humans from the treatments that are being applied to themselves and other subjects.

Additional Resources

Sampling and Statistical Power - StatsTree handout

Replication - StatsTree handout

Whitlock & Schluter - The Analysis of Biological Data

Chapter 14: pages 427-448 [Sapling]

 

Introduction to experimental design

Intro: Discussion of “control” and “treatment” groups, blind and double-blind experiments.

 

Describing main effects and interactions

Advanced: What interaction effects are and how to interpret them.

 

Verhoeven et al. 2005. Implementing false discovery rate control: increasing your power

This paper explains how to use ‘false discovery rate control’ to control for inflated type-I error rates that might otherwise be inflated by repeated tests of the same hypothesis.

Factorial designs

Intermediate: Description of a factorial design experiment.

 

Power analysis

Intermediate: How to determine the minimum sample size required to see an expected treatment effect in your experiment.

Review Questions

 

  1. Identify which goal of experimental design (i.e. reducing bias or limiting sampling error) is aided by the following procedures. [Credit: Whitlock and Schluter, 2009]

    (a) Using a genetically uniform animal stock to test treatment effects

    (b) Using a completely randomized design

    (c) Grouping related experimental units together

    (d) Taking the response measurements while unaware of the treatments assigned to experimental units

    (e) Using a computer to randomly assign treatments to experimental units within each block

  2. What is the justification for including extreme doses well outside the range of the exposures encountered by people at risk in a dose-response study on animals of the effects of a hazardous substance? What are the problems with this approach? [Credit: Whitlock and Schluter, 2009]

  3. Diet restriction is known to extend life and reduce the occurrence of age related diseases. To understand the mechanism better, you propose to carry out a study to look at the separate effects of age and diet restriction, and the interaction between age and diet restriction, on the activity of liver cells in rats. What experimental design should you consider employing? Why? [Credit: Whitlock and Schluter, 2009]

The Next Steps

Confused?

Let’s move down the tree and review these concepts.

Ready to Move Forward?

Let’s move up the tree to the next topic.