Experimental Design

Front 1

Experimental Design

Back 1

An experiment deliberately imposes a treatment on a group of objects or subjects in the interest of observing the response. This differs from an observational study, which involves collecting and analyzing data without changing existing conditions.

Front 2

Key Advantages of Experimental Design

Back 2

* Internal validity, because experimenter can rule out any other rival explanations.
* Complete control
- Wide variety of manipulative or statistical controls
- Systematic variation of the independent variable(s) (manipulations) to create homogeneous treatment groups
- Random assignment of subjects to treatments eliminates many threats to internal validity

Front 3

Experimental Design Weakness

Back 3

* Lacks in external validity (i.e. generalizability) because the environment in which it is administered is sterile and not subject to any external real-world influences

Front 4

Experimental Design Uses

Back 4

Useful in situations in which experimenter wants to examine the effects of a "treatment". The treatment is administered to the experimental group and compared against the control group who doesn't receive the treatment. For example, a psychiatrist might want to measure the effectiveness of a drug. It could be useful in HCI to determine how users respond to various web interfaces. It could be used by a sociologist to see how effective a study program is.

Front 5

Experimental Settings

Back 5

Lab, Natural Settings, Field Experiment, or Simulation

Many criticisms of experiments are due to the laboratory setting and not to the experimental strategy itself

Front 6

Laboratory Procedures

Back 6

* Manipulation Checks
- To assess effectiveness of experimental manipulation
-Temporal placement of manipulation checks
* Construct Validity
- Manipulation and confounding checks
- Multiple different operationalizationsof IVs in same experiment
* Dependent Variables
- Response biases: responses due to subject being aware of being observed or in an experiment
- Use unobtrusive measures
- Multiple measures of DVs using different methods (observation, paper & pencil, etc)

Front 7

Types of Experimental Designs

Back 7

Two broad categories: signal enhancersor noise reducers.

Hint 7

* The signal-enhancing experimental designs are called the factorial designs. In these designs, the focus is almost entirely on the setup of the program or treatment, its components and its major dimensions. In a typical factorial design we would examine a number of different variations of a treatment.
* There are two major types of noise-reducing experimental designs: covariance designs and blocking designs. In these designs we typically use information about the makeup of the sample or about pre-program variables to remove some of the noise in our study.

Front 8

2x2 Factorial Design

Back 8

2 Factors, 2 Levels
setting and time in instruction are factors

Front 9

Null Case

Back 9

No difference between setting and time in the graphs

Regression or 2-way ANOVA

Front 10

Main Effects

Back 10

first one there's no effect of setting
second one there's no effect of time

Front 11

Interaction Effects

Back 11

Front 12

2x3 Factorial Design

Back 12

2 factors, 2 levels, 3 levels - 6 conditions

Don't need graph for main effect, but you do for interaction

Front 13

2x2x3 Factorial Design

Back 13

2x2x3
3 factors (dosage, setting, treatment)

Main effects
3 2-way interactions
1 3-way interaction

beyond 3-way interactions, it is difficult to do

Front 14

2x2x3 Factorial Design

Back 14

2x2x3
3 factors (dosage, setting, treatment)

Main effects
3 2-way interactions
1 3-way interaction

beyond 3-way interactions, it is difficult to do

Front 15

Incomplete Factorial Design

Back 15

Certain levels of treatments
don’t make sense and are not
included

Hint 15

Contrast with:
Fully Crossed Factorial Design
A design that includes the
pairing of ALL combination of
factor levels

Front 16

Statistical Analysis for Factorial Designs

Back 16

Front 17

Why Manipulation & Confounding
Checks?

Back 17

* Manipulations may not be strong enough to allow
for a meaningful test of the theory
* Manipulation designed to vary one particular IV may also affect other IVs in the study
* Other latent variables (i.e. confounding variables) may be inadvertently affected by a manipulation
* The experimental procedures might create demand artifacts
* The design of the manipulation may suggest that the generalizability of the findings to real-world situations

Front 18

Manipulation Checks: Assessing
construct validity

Back 18

* Manipulation check: test to see whether manipulation worked as intended, i.e., whether the manipulations are related to direct measures of the latent construct they are intended to alter (convergent validity)
* Confounding check: test to see whether manipulation did not inadvertently manipulate other related variables (e.g., by manipulating image you also manipulate quality) (discriminant
validity)
* Reliability check: The manipulation and confounding test measures must reliably measure
the intended constructs.