Psyc202 - Exam 3 Study Guide

Front 1

a) Experimental Design

Back 1

i) Strongest with respect to internal validity
(1) Internal validity is the center of all causal inferences
(2) If X, then Y (if the program occurs, then the outcome occurs)
(3) If not X, then not Y (if the program is not given, then the outcome does not occur)
ii) Key is random assignment to groups

Front 2

c) Non-Experimental Design

Back 2

i) Cannot infer causality, purely descriptive
ii) ~ Correlation designs, comparative research

Front 3

2) Posttest Randomization

Back 3

c) One of the best designs for assessing causal relationships (internal validity)
d) No pretest needed because randomization infers probabilistic equivalence
e) Strong Against
i) Selection-testing, history, maturation, instrumentation, regression
ii) Single-group threats are ruled out because it is a group design
f) Not Strong Against
i) Selection-mortality, diffusion/imitation, compensatory equalization/rivalry, resentful demoralization
(1) IOW, all social threats to internal validity

Front 4

a) Random Sampling (related to external validity)

Back 4

i) Process or procedure that assures that different units in your population are selected into a sample by chance
(1) ~ 100 clients are drawn from a population list of 1,000 of your clients
ii) Better for generalization because samples better represent a larger population

Front 5

b) Random Assignment (related to design)

Back 5

i) Process of assigning your sample into two or more subgroups by chance
(1) ~ From the 100 clients, 50 are randomly assigned to the program, 50 are controlled
ii) Better for internal validity because it helps ensure probabilistic equivalence

Front 6

6) Signal and Noise

Back 6

a) Signal: the key variable of interest, the construct you’re trying to measure or the program or treatment that’s being implemented
b) Noise: all of the random factors in the situation that makes it harder to see the signal

Front 7

7) Signal Enhancing Experiments

Back 7

a) Factorial Designs
i) The focus is almost entirely on the setup of the program, its components, and its major dimensions
ii) Several different variations of the treatment is observed

Front 8

8) Noise Reducing Experiments

Back 8

a) Covariance Designs & Blocking Designs
i) Information about the makeup of the sample or about pre-program variables to remove some of the noise in your study

Front 9

9) Factorial Designs

Back 9

a) Designs that focus on the program or treatment, its components, and its major dimensions and enable you to determine whether the program has an effect, whether different subcomponents are effective, and whether there are interactions in the effects caused by subcompenents
b) Efficient because they enable you to examine which features or combinations of features of your program or treatment have an effect
c) A factor is a major IV, A level is a subdivision of a factor
i) The number of numbers indicate the number of factors
(1) ~ a 2 x 2 has 2 factors, a 3 x 4 has 2 factors
ii) Treatment groups are easily identifiable through multiplication
d) Need at least…
i) 2 Independent variables
ii) Each needs at least 2 levels

Front 10

10) Main Effect (definition and concept)

Back 10

a) An outcome that shows consistent differences between all levels of a factor
b) Differences shown between the levels show that a main effect has occurred
i) The effect can show that certain conditions react better/worse to the treatment
ii) Two main effects show that the conditions worked better in both IVs

Front 11

11) Interaction (definition and concept)

Back 11

a) An effect that occurs when differences on one factor depend on which level you are on another factor
b) Interaction occurs between factors, not levels!
c) Determining interactions
i) There’s an interaction when you can’t talk about an effect on one factor without mentioning the other factor
(1) ~ If you can say that one factor makes a difference, it would mean that there would be main effect and no interaction (because you did not mention the setting factor when describing the results)
ii) When an interaction occurs, it is impossible to denote one factor without the other

Front 12

Blocking in Experimental Design reduces noise by:

Back 12

i) One way blocking reduces noise is by reducing variability, and retaining the mean.
(1) Mean difference should be the same for each block as it is for the entire sample
(2) Variability should be much less within each block than it is for the entire sample
ii) If homogeneity is not true, the block’s variability would be the same as the sample
iii) Kind of like making experiments within your experiment

Front 13

20) Switching-Replications Design

Back 13

a) This is just a between subjects (a within is exposed to all levels of the IV)
b) Strongest experimental design
i) Good for external validity

Front 14

1) Nonequivalent Groups Design (NEGD)

Back 14

a) A pre/post two-group quasi-experimental design structured like a pretest-posttest randomized experiment, but lacking random assignment to group
i) One of the most frequently used designs because it is one of the most intuitively sensible designs around
b) Nonequivalent means that participants were assigned to the treatment or control conditions

Front 15

2) Quasi-Experimental = Types of NEGD’s

Back 15

a) Quasi-Experimental Design
i) Research designs that have several of the key features of randomized experimental designs, such as pre/post measurement and treatment-control group comparisons, but lack random assignment to a treatment group
b) Internal Validity
i) The approximate truth of inferences regarding cause-effect/causal relationships
ii) Has less than randomized designs

Front 16

3) Selection Bias

Back 16

a) Occurs when you select a participant if there is something about them you want to measure

Front 17

4) Problem vs. Threat to NEGD

Back 17

a) Biggest threat = selection bias/selection threat
b) Key problem = no randomization
i) NEGD works because of block randomization

Front 18

5) Regression Discontinuity (RD) Advantages

Back 18

i) Regressing the lowest/highest towards the mean; so can attribute towards the outside
ii) They are appropriate when you want to target a program or a treatment to those who need in most or deserve it most
iii) Has great potential for evaluation because inferences drawn are comparable in internal validity to conclusions from randomized experiments (strong competitor for randomized designs in regards to internal validity)

Front 19

5) Regression Discontinuity (RD) Disadvantages

Back 19

i) Can’t really generalize the results; will have very little generalizability
ii) Not used in social research much, with the exception of compensatory education (headstart)
iii) Force assignment to conditions solely based on quantitative indicators, discretion or favoritism can be used
iv) Need to understand regression to comprehend the results

Front 20

b) Regression Line

Back 20

i) A line that describes the relationship between two or more variables
ii) ~ Typically, the solid lines through bivariate distributions

Front 21

a) Logic of RD design

Back 21

i) Those below the cutoff score will be given the treatment
ii) To interpret the results of an RD:
(1) You must know the nature of the assignment variable and the outcome measure
(2) Who received the program
iii) A discontinuity in regression lines indicates a program effect in RD design, but the discontinuity alone is not sufficient to tell you whether the effect is positive or negative (need to know who received the program)

Front 22

b) Comparing Groups of RD Designs

Back 22

i) Can’t assume probability equivalence because we are dividing them across the regression line.
ii) In the absence of the program, the pre/post relationships would be equivalent for the two groups
iii) Strengths:
(1) Assumptions that there is no spurious discontinuity in the pre/post relationship that happens to coincide with the cutoff

Front 23

c) Threats to Internal Validity in RD Design

Back 23

i) Does not entail a selection threat because of cutoff scores
ii) Regression to the mean & maturation threats are the most common to this design

Front 24

d) Accountability of RD Designs

Back 24

i) Not a common lab design
ii) Accountability of a program is largely dependent on the explicitness of the assignment or allocation of the program to recipients
iii) 3 designs most applicable to lawmakers and policy makers
(1) Pre/post randomized experiments – analogous to the use of a lottery
(2) RD Design – programs are assigned based on merit or need
(3) NEGD – enables the use of unverifiable, subjective, or politically motivated assignment

Front 25

e) Power in RD Designs

Back 25

i) Increasing the sample size increases the statistical accuracy (when compared to randomized experiment)
(1) If a randomized experiment needs 100, RD needs 275, just multiply by 2.75

Front 26

f) Ethics in RD Designs

Back 26

i) Administering to subjects that apparently need the treatment the most

Front 27

9) Proxy Pre-test Design

Back 27

a) O1 vs. O2
i) Using two different DV at pre/post
ii) Pretest is collected after the program is given, hence proxy
b) Instrumentation threat is built in

Front 28

c) Recollection Proxy Pre-Test Design (terrible)

Back 28

(1) Pretest measures would be asked at the posttest time
(2) Often times people readjust their behavior

Front 29

d) Archived Proxy Pre-Test Design (better than Recollection Proxy)

Back 29

(1) Basically using old data and applying it to the study
(a) ~ Lazy to pretest, so take data from a test taken before
(2) Maturation/history threats

Front 30

10) Separate Pre-Post Samples (R vs. Non-R)

Back 30

a) Different people in pre/posttests
b) Common for evaluating a system/organization
i) ~ Comparing two agencies with customer satisfaction
c) Threats to Internal Validity
i) Instrumentation threat built into it
d) Randomized vs. Non-Randomized
i) Similar, but random sampling is used to draw from each nonequivalent group

Front 31

11) Double Pre-test Design (very strong)

Back 31

a) Nonequivalent groups with two pretests and one posttest
b) Threats to Internal Validity?
i) Overcomes a lot of threats
ii) ~ Regression to the mean occurs after the first pretest, so it is nullified after the second pretest
c) If done consequently, it is strong against the threats
i) Controls for selection & maturation, but susceptible if time gap between pretests
d) “dry run” because double pretests simulate what would occur in the null case

Front 32

12) Switching Replications Design

Back 32

a) Three tests of DV total
i) Strong against a lot of internal validity
ii) Cancels out equalization threat
b) Ethical Standards
i) Most ethically feasible quasi-experiment because all groups get treatment eventually
c) Good for generalization because you can see the difference whenever the treatment is administered

Front 33

13) Nonequivalent Dependent Variable Design

Back 33

b) Only has a single group of participants!
c) The two lines are separate variables, not groups
d) Two DVs
e) Hopeful for construct validity
i) ~ Test to measure if the video does make you want to buy chipotle.
ii) The real allure of this design is the possibility that we don’t need a control group – we can give the program to all of our sample
(1) The divergent validity qualifier acts as a control group (taco bell)
f) Full of internal Threats

Front 34

14) Construct Validity and NEDV

Back 34

a) Pattern Matching good for:
i) Convergent Validity (showing that measures are related, intercorrelation)
ii) Divergent Validity (showing that measures are not related)
b) Good for construct validity, not good for internal validity

Front 35

15) Regression Point Displacement

Back 35

b) Sample to population comparisons
c) Remember “C”?
d) N(n=1) here
e) Regression line
f) Only running one test
g) No pretest because have to have a cutoff, meaning that scores were retrieved from somewhere
h) Selecting a cutoff point defeats randomization and therefore is not good for external validity

Front 36

a) Internal Validity

Back 36

i) Causal, cause & effect
ii) If x then y; if no x then no y

Front 37

b) External Validity

Back 37

i) Generalization

Front 38

c) Construct Validity

Back 38

i) Measuring what you want to measure (taco bell vs. chipotle)
(Discriminant & convergent validity)

Front 39

2) What decides your design??

Back 39

a) Context decides design, most of the time we are just justifying one type of validity

Front 40

3) Different ways to counter threats? Best way?
a) Logic & Reason

Back 40

i) Rule out potential threats that actually threaten the experiment
ii) ~ An instrumentation threat is not a potential threat when the same pre/posttests are used

Front 41

3) Different ways to counter threats? Best way?
b) Measurement/Observation

Back 41

i) Reliability & validity

Front 42

3) Different ways to counter threats? Best way?
c) Type of Design

Back 42

i) most pivotal way to establish validity

Front 43

3) Different ways to counter threats? Best way?
d) Type of Analysis

Back 43

i) Statistical analysis (ANOVA, ANCOVA)

Front 44

3) Different ways to counter threats? Best way?
e) Proactive Thinking

Back 44

i) Control, control, control

Front 45

4) Necessary Elements for Research Design
c) Observation/measure (DV)

Back 45

i) O; if different measures are used at different times, use subscripts
ii) Types of measures
iii) Convergent and Discriminant validity
iv) NEDV

Front 46

4) Necessary Elements for Research Design
a) Time

Back 46

i) The temporal movements in research designs (O > X > O)
(1) Pretest vs. posttest, pretest as baseline
ii) ABAB designs
(1) A – no treatment
(2) B – treatment
(3) A – take away treatment
(4) B – treatment again

Front 47

4) Necessary Elements for Research Design
b) Program/treatment (IV)

Back 47

i) Levels of treatment
ii) No specific assignment to conditions
iii) Polar opposites?
(1) Create increases, opposite results; unethical (~ making a bad behavior worse)

Front 48

4) Necessary Elements for Research Design
d) Groups and participants

Back 48

i) R or N
e) X, O, R, and N

Front 49

a) Theory grounded

Back 49

i) Reflecting the theories investigated
ii) ~ When theory predicts a specific treatment effect on one measure but not on another
(1) The inclusion of both in the design improves discriminant validity and demonstrates the predictive power of the theory

Front 50

b) Situational context

Back 50

i) Reflecting the settings of investigation

Front 51

c) Feasible

Back 51

i) Good designs can be implemented
ii) Carefully planning the sequence and timing of events

Front 52

d) Replicable

Back 52

i) Good research designs have some flexibility built into them
ii) Duplication of essential design features

Front 53

e) Cost effective

Back 53

i) Good designs strike a balance between redundancy and the tendency to overdesign

Front 54

Passive RP

Back 54

i) situation when you ask people to IMAGINE a scenario or read a hypothetical situation and respond
ii) they're not actually in it
iii) ~ Reading a paragraph of on event.

Front 55

ACtive RP

Back 55

i) putting you in the actual situation and seeing how you react
ii) ~ Stanford Prison Experiment

Front 56

Analogue Designs

Back 56

a) Based off real events
b) Modeling a design after an event exactly how it happened

Front 57

14) Multiple Perspectives of Design
a) Systematic Variation

Back 57

i) The notion that no single realization will ever be sufficient for understanding a phenomenon with validity

Front 58

14) Multiple Perspectives of Design
b) Multiple realizations

Back 58

are essential for convergence on the truth of a matter
i) Important to realize which methods to apply in a study

Front 59

15) Mundane Realism

Back 59

a) Focuses on how much the experiment affects real life
b) ~ Milgram Experiment (little mundane)

Front 60

16) Experimental Realism

Back 60

a) Focuses on how much the situation would be applicable as real
b) ~ Milgram Experiment (high experimental)

Front 61

17) Mundane vs. Experimental

Back 61

a) Experimental more important because subjects treat experiment as real. If subjects are not engaged, the results are not reflected well