Experimental Design

From Scientificmetho

"To minimize individual differences by reducing the effects of extraneous variables, in order to and draw predictions of behavior from a experimental stimulus."


One Factor - Two Level Designs

Ok, the simplest experimental design we have is a design with a single independent variable and two levels of that variable. The two levels in the simplest design would be one group getting the independent variable (the experimental group) and one group getting no treatment (the control group) but being treated in an otherwise similar fashion (placebo).

The independent variable can be tested between groups as in the above example, or within groups. If it is tested between groups, it could either be a manipulated (again, as above) or a subject variable. If the independent variable is manipulated, the design will either be called an independent groups design if random assignment is used to create equivalent groups, or a matched group design if matching is needed. If a subject variable is being investigated, the groups are by definition a nonequivalent group. Terman’s Termites, members of a longitudal study designed to see if intellectually gifted people (such as me) lived better lives than people of average intelligence (such as you), were subjects involved in an nonequivalent study. Matching was used to reduce the nonequivalence as much as possible, such as picking people of the same age groups, same percentage of gender, same racial groups.

The repeated measures design is a within group design, used when the independent variable is tested within subjects, or when every subject in the study experiences both levels of the independent variable. A typical example would be food taste testers. This design has the benefits of requiring less subjects than between group designs, since every participant will experience all levels of the independent variable. This design also reduces the errors that occur when comparisons are made between groups, since the exact same subjects are being compared to themselves! However, this design suffers from experimental confounds that the between groups designs do not - dealing with the confounds that prior experience with the independent variables cause, such as proactive interference - where the learning of new material is hindered by similar old material previously learned. Another confound involves the effects of order of presentation.

Now, to test your knowledge of experimental design, try out the decision tree below.


Just hold your mouse over the title and try it out.

Analyzing single-factor, two level designs

To determine whether differences found between the conditions of a two level design are significant, or due to chance, one runs an inferential analysis. If the measured scales are nominal, a Chi Square is run. If the dependent variable is measured on an ordinal scale, another inferential test is run. If the dependent variable scales are either interval or ratio, and the participants are randomly assigned to groups, or the variable being studied is a subject variable, a T Test for independent/nonequivalent groups can be run. If the independent variable is a within-subjects/repeated factor, or if the two groups of people matched, a T Test for dependent groups (or correlated groups) is run.

A T test basically lets us know if measured mean of two groups differs in a statistically significant way. For example, if an experimental group is averages one headache per year while on a medication while the control group suffers an average of 6, 403 headaches per year, the T test will most likely uncover a significant difference between the groups.

One Factor - More than 2 levels

The advantage of using multilevel designs is that they enable the researcher to uncover nonlinear effects, such as uncovering a threshold effect for a drug. For example, if we use only two levels in a design - such as 1mg or 3mg, the linear graph line illustrating the speed of effect of the drug, moving directly up from 1mg to 3mg may be illusionary. By testing other levels of the drug, 2mg, 4mg, we may see that the graph line moves straight up to 2mg, but levels off before 3mg - revealing that the drug threshold is at 2mg, not 3mg. Fortunately, drug companies have long known this.

By using more levels in an experiment, more than one hypothesis can be tested within a single experiment. Remember the experimental confounds of single-factor designs discussed above? Multilevel designs in within-subject, repeated measure experiments can allow for the use of counterbalancing, or ensuring that presentation order does not influence results. In other words, the more levels in an experiment, the more permutations in order presentation are possible.

The results of single-factor, multilevel designs are tested by a procedure of one-way, inferential analysis of variance known as an ANOVA. (ANalysis Of Variance.) One-way means independent variable. The ANOVA reveals an Fisher score, or F score, which points to statistical significance. Post Hoc tests are used to collapse the levels so that we can uncover for which level of the independent variable there is a significant finding.

Multi Factorial Designs

Multi factorial designs test more than one independent variable at once, but usually keep the number of IVs to 4 or less, so that the statistical end of the research does not become unwieldy.

A factorial design is described with a numbering system that simultaneously identifies the number of independent variables and the number of levels of each variable. A 2 x 3 factorial design has two independent variables, the first has 2 levels and the second has 3. You multiply the levels of the variables to count the number of experimental conditions. In this example, there are 6 experimental conditions.

With multi factorial designs, we get both main effects and interactions. Main effect is used to describe the overall effect of an independent variable. In a study with 3 independent variables, there can be up to 3 but no more than 3 main effects. Determining the effects of any one independent variable involves collapsing data over all levels of the other factors, to isolate its effect.

However, the distinct advantage of factorial designs over single-factor designs lies not in their ability to uncover several main effects at once, but in their potential to show interactive effects. Interaction is said to occur when the effect of one independent variable depends (ironically enough) on the level of another independent variable. Interactions often provide the most interesting results in a factorial study. Since the world is more complex than most single-factor experiments can explain, the interactive nature of multi factorial designs allows us to uncover experimental effects where single-factor designs would find nothing of significance.

An example:

Ronald Fisher performed research on improving wheat farming. Single-factor experiments using new strains of wheat showed that no one genetic strain outperformed another.

Next, he turned to using experimental fertilizer - using another single factor design and analysis, he found no significant difference between fertilizers on wheat growth.

Fisher recognized the complexity of the situation called for a complex research design. He therefore decided to use a multi factorial design - combining the independent variables of genetic strains of wheat (Type I and II) and experimental fertilizer (A, B).

Fisher again found no significant difference for genetic strain of wheat or experimental fertilizer. Just as he was about to grab a shotgun and kill all his co-workers, he thought up a statistical analysis that could examine interactions in his experiment - an inferential analysis that we call the ANOVA. Fisher found significance in one of his "Fisher" ratios. (We now just call them F ratios.) He found that level 1 of the genetic strain interacted with level 1 of the experimental fertilizer.

Main Effects and Interactions Table

In a simple 2 x 2 factorial design, there are 8 permutations of effects:
1) Main effect for factor A
2) Main effect for factor B
3) Main effects for A and B
4) Main effect for A plus an interaction
5) Main effect for B plus an interaction
6) Main effects for both A and B plus an interaction
7) An interaction only (Just as Fisher found)
8) Null hypothesis

Varieties of Factorial Designs

Varieties of factorial design are similar to the single-factor designs, but include the addition of more designs. This is because while single-factor designs must either be a between-subjects or within-subjects model, factorial designs can be both. This leads to a mixed factorial design. If the factorial design includes both a manipulated and a subject variable - an interaction between person and environment, then this is a P x E factorial. Theorists, such as Kurt Lewin, who feel that an understanding of human behavior requires knowledge of the interactions of a person with his environment favor this model.

To see for yourself how factorial designs work, try out this factorial design decision tree.


Go on, just move your mouse over the title below.

Analyzing factorial designs

Factorial designs are interpreted using N-way ANOVAs, where the N stands for the Number of independent variables. A 3 x 5 x 5 factorial would have an N of 3 (remember, 3 independent variables are represented here, the number refers to the levels of the independent variable.)

In a factorial ANOVA, more than one Fisher score will be generated one for the main effects, and one for each interaction. In a A x B factorial, with two independent variables, there will be 3 F scores. In a factorial with 3 independent variables, A x B x C, there will be 7 F scores. In a A x B x C x D factorial, with four independent variables, there will be 19 F scores. This shows us why few experiments use more than 4 independent variables.

Personal tools