Study your flashcards anywhere!
Download the official Cram app for free >
 Shuffle
Toggle OnToggle Off
 Alphabetize
Toggle OnToggle Off
 Front First
Toggle OnToggle Off
 Both Sides
Toggle OnToggle Off
 Read
Toggle OnToggle Off
How to study your flashcards.
Right/Left arrow keys: Navigate between flashcards.right arrow keyleft arrow key
Up/Down arrow keys: Flip the card between the front and back.down keyup key
H key: Show hint (3rd side).h key
A key: Read text to speech.a key
63 Cards in this Set
 Front
 Back
Research design typically has two categories:

1. experimental design (now).
2. correlational design (next). 

True Experiments:

1. state at least 1 hypothesis about IV’s effect on DV(s)
2. have at least 2 levels of an IV 3. assign participants to conditions randomly 4. use specific control procedures for testing hypotheses 5. use controls for internal validity threats 

Variance…

Defined, as before, as SS/df…
Goals for experimental design: answer questions by testing causal hypotheses control variance to increase internal validity be confident that the variation between experimental conditions is due to the manipulated IVs 

Variance…
Systematic betweengroups variance 
necessary for determining causal differences
good and bad types, both change participant scores in 1 direction. experimental var + extraneous variance 

Variance

The only way we can be confident about our decision of rejecting the null hypothesis is if we’ve used proper control procedures,
e.g., automation, doubleblind, deception, elimination, constancy, build a variable into the design 

Nonsystematic withingroups variance

due to random factors (e.g., participant fatigue, equipment fluctuation), individual differences
chance error affects Ss in both directions, so effects tend to cancel out each other (e.g., high and low energy participant effects tend to balance out)—but this doesn’t mean that chance error should be ignored 

FRatio and Variance

How do A and B relate to each other? When we compare groups in a statistical analysis, we take a ratio of the two, which we call “F” (we’ll use this in ANOVA) F=BtwnGroups Var / W/inGroups Var


We want “F” to be large, due to

maximizing betweengroups variance,
controlling extraneous variance, and minimizing error variance (use large N, matched Ss, reliable tests). 

Variance…
Conceptually, “F” is broken into this ratio of variances: 
experimental + extraneous + error / error
Error is included in both numerator and denominator because it will always be present (G&R: “Suppose there are no systematic effects. In this case, both the numerator and denominator represent error variance only, and the ratio would be 1.00.” Misleading! F is often BELOW 1.00. Use “1.00” as a conceptual guideline.) 

Experimental variance—MAXIMIZE!

use at least 2 IV levels (at least trtmt and ctrl)
use “manipulation checks” (selfreport measures of manipulation given—e.g., did you use the strategy procedure we gave you?) 

Extraneous variance—CONTROL!

Equate trtmt & ctrl grps before start of experiment
(i.e., completely equal) 2. Random assignment (any differences have an equal chance of appearing across groups) 3. Elimination & constancy (Elim—remove a particular age group; Constancyuse only 20yrolds) 

Extraneous variance—CONTROL! cont.

Build the variable into the design
(e.g., if sex differences exist for spatial ability, use “sex” in experimental design of spatial study) 5. Match participants across groups (e.g., age, IQ, SES) 6. Use participants as their own control (WithinS design or repeated measures design; use a dependent means ttest if two times of measurement are used. 

Error varianceMINIMIZE!

chance variation due to individual differences, random fluctuations in test instruments, etc.
1. Use reliable test instruments consistent tests will minimize error, have better chance of measuring “something” 

Error varianceMINIMIZE!
Use matchedSs or WS designs 
Ss serve as their own control; persons are more consistent within themselves than across others


Nonexperimental designs

These designs DON’T have all the 5 characteristics of a true experiment [EPF  1GP  1GPP  PPNC]
examples: ex post facto; 1grp, posttest only; 1grp, pretest posttest; pretest, posttest, natural controlgrp design 

Ex post facto  “after the fact”

Group  Event  Measurement
Present observation is made Past info not observed Causal inferences difficult to make Problem: No controls for confounding factors e.g., therapy situations; Do abused persons become abusers? Maybe there is a link, maybe not. 

1group, posttest

1group, posttest only
manipulation is made Group  Trtmt  Posttest e.g., Memory training given to a group (no pretest, no ctrl!) Why did scores look good/high? Problems: Expectancy, Hawthorne Effect, placebo, history, maturation, regression to the mean? 

1group, pretestposttest

1group, pretestposttest
Group Pretest  Trtmt Posttest Compare Pre and Post scores e.g., better situation for memory training experiment Problems: maturation, regression to the mean, placebo, history 

Pretest – posttest, natural control group
[PPNC] 
Pretest – posttest, natural control group
Grp 1 Pretest  Trtmt Posttest Grp 2 Pretest no Trtmt Posttest Compare posttest scores Use naturally occurring groups e.g., sororities, schools, organizations; no random assignment is used 

Pretest – posttest, natural control group…

e.g. Differences in assertiveness due to training
Sorority 1  qaire assert training Posttest Sorority 2 – qaire – no training Posttest Compare posttest scores 

Pretest – posttest, natural control group…

Problem: How do we know that preexisting differences weren’t present in the two groups? More social cohesiveness, more experience with training, etc.
Better to make random assignment within sororities to ensure group equivalence. This way, even if we’re using preexisting groups, we can control for other differences between them. 

V. Experimental Designs

have the 5 characteristics of a true experiment [RPOC, RPPC, MLRB, Solomon]
A. Randomized Posttest only Control Group design R Grp 1Trtmt Posttest // R Grp 2 No trtmt Posttest 

Experimental Designs

Compare posttest scores
Random assignment controls threats to internal validity Random selection controls threats to external validity Using control groups protects regression to the mean, history & maturation; instrumentation 

Experimental Designs
Randomized Pretestposttest Control group design 
R Grp 1 Pre T Post
R Grp 2 Pre NT Post Compare posttest scores Better than the natural control group design just mentioned; RANDOMIZATION occurs! better than the randomized posttest only control group design; PRETEST occurs! ***Problems: Receiving the pretest affects Ss somehow; sensitizes Ss to study purposes; “interaction effect” 

Multilevel, completely randomized, betweensubjects design

extension of earlier designs; 3 or more conditions, pretest not always included
e.g., Give different amounts of practice for solving a cognitive task: 0 practice trials for Group 1 5 trials for Group 2 10 trials for Group 3, etc. Controls for validity threats via random assignment 

D. Solomon’s 4group design

controls for interaction of pretest and trtmt effect
very extensive, welldeveloped most imp. comparison is between Grp 1 & 2’s posttests R Grp 1 Pre T Post R Grp 2 Pre Post (history & matur) R Grp 3 T Post (interact’n; grp 1) R Grp 4 Post (maturation) e.g., attitudes towards tobacco use; T = videotape 

Conditions where exp. designs with a single variable are not appropriate

A. When randomization is NOT possible
e.g., gender of participants 

Conditions where exp. designs with a single variable are not appropriate

B. Variables rarely affect behavior singly; mostly more than 1 variable affects behavior simultaneously
e.g., speed, attention, memory, motivation, strategy use affect cognitive performance 

Conditions where exp. designs with a single variable are not appropriate

Variables may INTERACT with each other
e.g., Differences between levels of factor 2 for a level of factor 1 may be different for the other level of factor 1. (sound familiar?) hence, the need for factorial designs… 

Experimental research design distinctions
A. Dependence 
A. Dependence
1. Independent groups = Betweensubjects designs (Different persons are in each group) 2. Correlated groups = Withinsubjects designs (Same or matched persons are in each group) 

B. Single/multiple IVs

1. Single variable designs = univariate (1 IV)
2. Factorial designs = multivariable (2 or more IV’s) e.g., 2 x 2 factorial design 

SingleVariable, Correlated Groups Designs

Correlated groups designs do not use free random assignment to conditions
One type of correlated group designs is withinsubjects, or repeated measures designs Another type is called matchedsubjects designs An extension of the withinsubjects design is a singlesubject experimental design; we’ll discuss 3 types of these shortly 

Withinsubjects designs

Withinsubjects designs involve the repeated measurement of the same person
Each person is given all levels of the IV, like this Working Memory task example, where each person is given 3, 4, 5, 6, and 7 words to retain... 

Withinsubjects designs…

The analysis involves comparing differences between the correlated groups. For this example, one would conduct a dependent
means ttest on the number of items recalled. To be sure the IV affects the DV, one should counterbalance the order of exposure to conditions This protects against sequencing effects such as practice and carryover 

Withinsubjects designs…

Strengths:
No preexisting differences between groups, since the groups are the same people; this helps protect internal validity WS design eliminates error variance due to individual differences Saves time; bring in only 1 set of participants Need fewer N 

Withinsubjects designs…
Strengths 
No preexisting differences between groups, since the groups are the same people; this helps protect internal validity
WS design eliminates error variance due to individual differences Saves time; bring in only 1 set of participants Need fewer N 

Weaknesses:

Sequence effects can be problematic; Permanent changes are not reversible (e.g., surgery, knowledge/attitude change)
Two types of sequence effects: Practice effects; can create positive, enhanced change, or create negative change, such as fatigue; not necessarily due to a particular order of experiences Carryover effects; a previous trial influences the next, because of the nature of the previous trial; effects may not be “even” for all orders of presentation 

Controls for sequence effects:

Present conditions randomly to participants, to “even out” the confounds across conditions
Systematically present the conditions by complete counterbalancing, so that each condition appears in a particular order Latinsquare designs; this presentation order involves partial counterbalancing the conditions such that each condition appears only once in a row, and once in a column. Other controls include:  holding a variable constant (by training to a criterion before the experiment begins)  allow a break to control fatigue  skip the WS design and use a BetweenSubjects design instead 

MatchedSubjects Designs

Matched subjects involves using different persons in each condition, but these persons share characteristic(s) based upon an important consideration of the study’s goals
E.g., age, visual acuity, SES, GPA, gender 

MatchedSubjects Designs…

Participants each get only one level of the factor
Use a dependent means ttest if only 2 IV levels are used Use a repeated measures ANOVA if more than 2 IV levels are used With either analytic procedure, keep track of who is linked to whom! 

MatchSubjects Designs…
Strengths include: 
Strengths include:
the ability to use a smaller N with a given statistical power level, relative to a BetweenSubjects design no sequence effects [smaller n req but usu req oversampling!] 

MatchedSubjects Designs…
Weaknesses 
Greater effort for the researcher to find an effective matching characteristic, then find the right matches for each participant
Usually requires oversampling 

Singlesubject designs

Experimental (an IV is manipulated), used mainly in clinical settings; good for treatment evaluation
We use these when we want to measure change within a person And, obtain information otherwise lost in group comparison (ex. if average effect cancels out even though indiv increases) 

Singlesubject design Types

Types [ABAR  MB  SSRTS]
ABA + [reversal] Multiple baseline Singlesubject, randomized, timeseries designs 

Singlesubject designs
A. ABA + [reversal] 
Baseline, Treatment, No treatment, Treatment
*Not good if detrimental to health to stop suddenly. 

Singlesubject designs
Multiple baseline 
Use if we don’t want to return to a negative
Baseline (danger, injury, ethics) Use treatment on different behaviors successively E.g., Teacher wants to see child do better in math and reading and have less disruptive behavior. Start only reading help, then reading and math help. 

Singlesubject, randomized, timeseries design

Randomly pick a point in which a reinforcement or change is given to subject
If change occurs, it’s unlikely due to chance, maturation, or history. *Randomly start stimulus change in 6th trial, 2nd, etc. 

Introduction to ANOVA

You must use if it you desire the comparison among 3 or more means…(not multiple ttests)
Types of ANOVA 1. Betweensubjects ANOVA (like what you’ll be doing in lab) 

Withinsubjects ANOVA

Or repeated measures ANOVA, where we desire to know how an individual changes over time, or across conditions.
e.g., give an “Attitude Towards Pres. Bush” scale (ATPBS), at 3 time points 

How are the (between) groups defined?

Groups are defined by factors—those independent variables we manipulated (like amount of reward given), or are known to differ (like gender)
We require you to manipulate both factors in your final project, although in practice outside of 306, one might decide to use preexisting variables as factors, like gender, sorority, school, car ownership, educational level, etc. A general way to describe these designs is as follows: Threeway ANOVAthree factors etc. 

A more specific way to describe these designs is as follows:

[Design 'way' defined by number of numbers]
2 x 2 ANOVA > a twoway design, each factor having 2 levels // 3 x 3 x 4 ANOVA > a threeway design, first & second factors have 3 levels; third has 4 levels. // 2 x 2 x 3 x 3 ANOVA ? a fourway design, first & second factors have 2 levels, third & fourth factors have 3 levels 

A even more specific way to describe these designs is as follows:

Oneway betweensubjects ANOVA manipulating 4 levels of stimulus brightness
2 x 2 betweensubjects ANOVA, manipulating reward (low, high), and punishment (no, yes) 3 x 3 x 4 betweensubjects ANOVA, manipulating reward (low, medium, high), punishment (low, medium, high), and stimulus brightness (very dim, dim, somewhat bright, very bright)And, a 2 x 2 x 3 x 3 (why do this kind of design??)… 

Determine the characteristics of the comparison distribution…

F varies with df for the numerator and denominator
F is positively skewed, since the distribution is a distribution of variance ratios…variances can only be positive 

comparison distribution cont (X – GM, X – M, M – GM)

All Ftests are onetailed, but with an omnibus Ftest, we don’t express “direction” of expected differences (even if we have one!). The Ftest doesn’t indicate which means differ other than the largest and smallest means.
Ftests can only be positive The “direction” of expected effects must be examined at the multiple comparison step. 

Multiple Comparisons

When the omnibus Ftest (oneway ANOVA F) is statistically significant, we may want to know which means are statistically different; so far, we know only that at least two means differ.
We can do multiple comparisons, in which several pairs of means are tested 

Multiple Comparisons
So, powerwise, it’s best to be able to conduct planned comparisons 
Planned, ‘a priori’comparisons [B&F]
e.g., Bonferonni e.g., Fisher’s LSD Easier to achieve a statistically significant results because they’re planned tests Unplanned, ‘posthoc’ tests [S&T]  e.g., Scheffe Ftests (*Most conserv)  e.g., Tukey (*next most conserv)  avalue is cut into many pieces, so it’s more difficult to get a statistically significant result; e.g., alpha of .05 / 5 tests = .01 

Multiple Comparisons

With Fisher’s test, one can make up to 3 planned comparisons without needing to adjust alpha (no matter how many means you have available).
Therefore, be selective about which means you want to test specifically. 

Multiple Comparisons

As hypothesized, the results of Fisher’s LSD tests showed that both verbal (Mv = 30.00, SDv = 2.00) and verbalpictorial (Mv&p = 40.00, SDv&p = 2.00) teaching methods were superior to no teaching method (Mzip = 26.00, SDzip = 2.00) , F(1,6) = 6.00, p < .05, F(1,6) = 73.50, p < .01, respectively; In addition, the verbalpictorial method was found to be statistically significantly better than the verbal method alone, F(1,6) = 37.50, p < .01.”


Multiple Comparisons

With Fisher’s LSD test, we compare 2 means at a time, but the pooled variance is MSW , the variance estimate from all groups
This pooled variance is more stable and powerful than the pooled variance from ttests 

Tukey– conservative, and
good if # of groups is large. Scheffe – more conservative; F critical is adjusted to by a (k1) multiplier. Fisher’s LSD  pretty liberal 
Bonferroni – conservative; adjusts alpha for comparisons.
Not good to use if # of groups is large. Sidak – More conservative than Bonferroni. ***Bad that SPSS gives so little in the table; you know nothing about the alpha adjustment, adjusted F, or the computation of the ratio. 

Effect Size = f

Cohen’s Effect Size conventions are a bit different for ANOVA:
f Small .10 Medium .25 Large .40 

Effect Size and Power

Effect size = R2 =Proportion of Variance Explained
This effect size calculation must be used when n is unequal; It can be used for equal n cases, too 

R^2 = SS Btwn / SS T

R2 Effect Sizes
Small = .01 Medium = .06 Large = .14 Too liberal. % of the variance in the DV can be explained by the IV; 