When you design an experiment, consider the following items:
The Experimental Procedure Often, the experiment is so simple that you'll assume you understand the procedure, but making it explicit helps to find spurious effects and sampling biases.
An Example of a Data Table Lay out your data table with independent and dependent variables in columns. This shows you what is being varied and what data are being collected. Now consider the rows. How often will data be collected? At the end of each trial? At regular intervals? After significant events? Do you need a variable to time-stamp your data? If you can put hypothetical numbers in the table, so much the better, because then you can consider an example of your analysis.
An Example of Your Analysis A common complaint from statistical consultants is that their clients collect reams of data with only a vague notion of how it will be analyzed. If you think through the analysis before running the experiment, you can avoid collecting too much or too little data and you often can find a better experiment design. These benefits are especially likely when your experiment is intended to find interaction effects, because these experiments inherently involve combinations of conditions, which complicate analysis and confuse humans.
A Discussion of Possible Results and Their Interpretations
What do potential results mean? It is very helpful, especially in complex
designs, to sketch hypothetical results and see how you would explain
them. Consider, for instance, a factorial experiment with two
independent variables, 
and 
, and a continuous
dependent variable y. You plan to calculate the mean of y in
each of four conditions defined by the combination of two values of
each of and . A useful trick is to plot some possible
outcomes and see whether you can interpret the results, as shown in
Figure 3.12. In case A, clearly has an effect
on y because mean y values are higher at 
than at
at both levels of . Similarly, mean y values are
higher at 
than at 
`, irrespective of the values
of . In case A, and have
independent effects on y. In contrast, in cases B and C, the
effect of on y depends on the value of or vice
versa. Your interpretation of cases B and C might be that the
influence of on y is controlled or ``gated'' by .
The point is that sketches like Figure 3.12 help you
consider the possible outcomes of your experiment, often raising
possibilities that you wouldn't have considered otherwise, and help to
determine whether the experiment design can actually produce the
results you are looking for.
Figure 3.12 Rough sketch of potential outcomes of a two-factor experiment.
Now, what was the question, again? Experiment design is a seductive activity, made more so by easy statistical analysis. You can design an experiment and an analysis that do not actually answer the question you initially asked. You should double check the potential results against the question. Do they answers the question? Nothing prevents you from changing the question if, while designing the experiment, you find one you'd rather answer, or one that better summarizes your research interest. Still, check whether the experiment answers your question.