Nerm Midterm 1

Front 1

Why is random assignment considered to be the gold standard for researchers to infer causality?

Back 1

because of its ability to rule out alternative explanations

Front 2

What is considered to be the gold standard for researchers to infer causality?

Back 2

random assignment

Front 3

What is an experimental design?

Back 3

participants are randomly assigned to either treatment or control conditions

Front 4

What are some other terms for an experimental design?

Back 4

Randomized experiments, randomized control trials (RCT), or randomized clinical trials (RCT)

Front 5

What questions do the counterfactual model ask?

Back 5

1) If the individuals who received the treatment had in
fact not received it, what would we observe on Y for
those individuals?
or
2) If the individuals who did not receive the treatment
had in fact received it, what would we have observed
on Y?

Front 6

Describe the counterfacutal argument.

Back 6

-in random assignment control and treatment are equivalent at start of experiment (theoretically interchangeable)
-they represent each other’s counterfactual
-RA: effect of manipulation is difference between two groups on DV (no other cause = causality)

Front 7

What are quasi-experimental designs:?

Back 7

participants are assigned to groups by a method other than random assignment

Front 8

What are the two types of quasi-experimental designs?

Back 8

–Non-equivalent-group design
–Regression-discontinuity designs

Front 9

What is a non-equivalent-group design?

Back 9

-treatment and control groups are already formed
-existing groups
-experimenter has no control over group assignment

Front 10

What are examples of existing groups? (non-equiv group design)

Back 10

-presence/absence of illness/conditions
-behaviours (smoking, drug use, overweight)
-male, female
-age

Front 11

What is the key difference between non-equivalent group design and regression-discontinuity?

Back 11

assignment variable

Front 12

What are some ideals with non-equivalent group design?

Back 12

-groups should be as similar as possible
-be able to randomly choose what group receives the treatment

Front 13

What is a regression-discontinuity design?

Back 13

participants are assigned to conditions based on a cut-off score on an assignment variable

Front 14

What is the meaning behind the name of the regression-discontinuity design?

Back 14

looking for a discontinuity (break) in the regression line

Front 15

In regression-discontinuity designs, does the assignment
variable have to predict the outcome?

Back 15

no!

Front 16

How are regression-discontinuity designs SIMILAR to non-equivalent group designs? but...

Back 16

in RDD, the method implies that groups are non-equivalent BUT researcher knows how participants were assigned to conditions

Front 17

Draw a regression-discontinuity designs (with letters).

Back 17

Front 18

Draw the results from a RDD when there is a relation between assignment variable and outcome (Y).

Back 18

Front 19

Draw the results from a RDD when there is no relation between assignment variable and outcome (Y).

Back 19

Front 20

How are regression-discontinuity designs analyzed?

Back 20

-using multiple regression

Front 21

Draw a "one time observation" design.

Back 21

Front 22

Draw a "one group pretest-posttest design".

Back 22

Front 23

How could a "one group pretest-posttest design" be strengthened? Draw.

Back 23

-add more pretests and posttests

Front 24

What is a "repeated-treatment design"? Draw.

Back 24

introduce and remove treatment within the same participants over time

Front 25

Draw a posttest-only design.

Back 25

Front 26

What is the weakness of posttest-only designs?

Back 26

because there is no pretest, it is susceptible to all forms of selection threats

Front 27

Draw a "pretest-posttest design".

Back 27

Front 28

Why is a "pretest-posttest design" an improvement on posttest only?

Back 28

can test for group equivalence before treatment

Front 29

Draw a double-pretest design.

Back 29

Front 30

What threats does a double-pretest design help overcome (compared to one pretest only)?

Back 30

-tests for the stability of equivalence
-might identify a change that was going to happen anyway
---selection-regression: one/both groups might regress to average
---selection maturation: underlying change that was going to happen anyway

Front 31

Draw how a double-pretest might detect a selection maturation/regression threat.

Back 31

Front 32

Draw a "switching of treatments" design.

Back 32

Front 33

What are some advantages to a "switching of treatments" design?

Back 33

-eliminates many threats to internal validity
-allows researchers to test if effects of treatment are maintained
-Ethical advantage of giving everyone the treatment

-more observations, more information!

Front 34

What three conditions must exist to measure causality?

Back 34

1. X must precede Y temporally (cause --> effect)
2. X must be reliably correlated with Y (beyond chance)
3. The relation between X and Y must not be explained by other causes

Front 35

What is internal validity?

Back 35

the degree to which we can draw conclusions about causal relationship in our data

Front 36

What do we mean by "degrees of causality"?

Back 36

meeting the three criteria to various degrees

Front 37

How can we answer threats to internal validity?

Back 37

-manipulation
-random assignment
-eliminate/include extraneous variables
-statistical control
-rational argument
-analyze reliable scores

Front 38

How is internal validity established?

Back 38

by controlling for extraneous variables

Front 39

What do all threats to internal validity have in common?

Back 39

-they are ALL different types of third variables (3rd requirement for causality)

Front 40

List the general threats to internal validity.

Back 40

-history
-maturation
-testing
-instrumentation
-attrition
-regression
-ambiguous temporal precedence

Front 41

List threats to internal validity with multiple groups.

Back 41

-selection
-resentful demoralization
-compensatory rivalry
-treatment diffusion or imitation
-compensatory equalization of treatment
-novelty and disruption effects

Front 42

How does "direct manipulation" establish internal validity?

Back 42

-controls when and how “the cause” occurs
-also controls magnitude of effect

Front 43

How does "random assignment" establish internal validity?

Back 43

-minimizes differences between groups by equally distributing characteristics across groups.

-equates groups on all variables (those measured and not measured) before study

Front 44

How does "Elimination or inclusion of extraneous variables" establish internal validity?

Back 44

elimination = making sure other factors are constant

inclusion = direct measurement of a factor and its inclusion as a variable

Front 45

How does "statistical control" establish internal validity?

Back 45

-measure extraneous variables (but don't explicitly represent as variables)

-remove their influence on outcome statistically (covariates)

Front 46

How does "rational argument" establish internal validity?

Back 46

offer arguments for why a potentially extraneous variable is not substantial

Front 47

How does "analyzing reliable scores" establish internal validity?

Back 47

makes sure effects are due to content of measures not random or systematic error

Front 48

What are the two most important methods of establishing internal validity?

Back 48

manipulation and random assignment!

Front 49

What are threats to internal validity (in general)?

Back 49

-something that jeopardizes our ability to conclude that X causes Y

-“Third variables”

Front 50

How is "history" a threat to internal validity?

Back 50

Specific event that take place concurrently with treatment. Could be anything (e.g., weather change, big event in news, local initiative, etc.).

Front 51

How is "Maturation" a threat to internal validity?

Back 51

Naturally occurring changes are confounded in the treatment. People change over time. Get older, more mature, hungrier, make more money, etc.

Front 52

How is "Testing" a threat to internal validity?

Back 52

Practice or reactivity. Simple measuring something may change outcome (i.e., information provided, introspection occurs)

*Relevant for longitudinal or lagged designs where participants are tested more than once

Front 53

How is "Instrumentation" a threat to internal validity?

Back 53

The meaning and interpretation of observations change over time.

*Particularly pertinent for observers or coders (e.g., get tired, change their standards, etc.)

Front 54

How is "Attrition" a threat to internal validity?

Back 54

Loss of cases from study. Problem when it is systematic (e.g., certain people are more or less likely to drop out than others).

*mortality!*

Front 55

How is "Regression" a threat to internal validity?

Back 55

-Statistical regression to the mean.
-Tendency for extreme cases to be less extreme on subsequent measures.

*Problematic when selecting participants based on extreme scores.

Front 56

How is "ambiguous temporal precedence" a threat to internal validity?

Back 56

Lack of understanding about what variable occurred first (which is the cause and which is the effect). Measuring variables at same time.

e.g. job satisfaction and performance

Front 57

How is "selection" a threat to internal validity?

Back 57

-multiple groups

-respondents in conditions differ at the outset.

Front 58

What is resentful demoralization?

Back 58

: Control cases learn about treatment become resentful and stop trying or withdraw from study.

Front 59

What is compensatory rivalry?

Back 59

Control cases learn about treatment and become competitive with treated cases.

Front 60

What is "treatment diffusion or imitation"?

Back 60

Control cases learn about treatment or try to imitate experiences of treated cases.

Front 61

What is compensatory equalization of treatment?

Back 61

Cases in one condition demand to be assigned to the other condition or be compensated.

Front 62

What are novelty and disruption effects?

Back 62

Cases respond well to a novel treatment or poorly to one that interrupts their routines.

e.g. Hawthorne effect

Front 63

What is the "Hawthorne Effect"?

Back 63

participants alter their behavior as a result of being part of an experiment or study

-attention/being observed

Front 64

How can threats combine to produce new threats to internal validity?

Back 64

selection-attrition
selection-regression
selection-maturation

*selection + ARM

Front 65

What is selection-maturation?

Back 65

Different rates of change (or growth) between groups.

Front 66

What is selection-attrition?

Back 66

Rate of missing data is higher in one group than another.

Front 67

What is selection-regression?

Back 67

There are different rates of regression to the mean across groups.

-likely to happen when only one group is selected based on extreme scores

Front 68

What threatens external validity (general)?

Back 68

Any characteristic of a sample, treatment, setting, or measure that leads the results to be specific to a particular study and not generalizable.

Front 69

What is external validity?

Back 69

Inferences about whether the results of a study will hold over variations in participants, treatments, settings, and measures.

Front 70

What are two subcategories to external validity?

Back 70

population validity

ecological validity

Front 71

What is population validity concerned with?

Back 71

Can you generalize from your sample to the population or to another defined population?

Front 72

What is ecological validity concerned with?

Back 72

Concerns whether the combination of manipulation, settings, or outcomes approximate those in real life situation under investigation.

Front 73

How can we establish external validity before and after our experiment?

Back 73

before: probability sampling techniques

after: establish external validity through replication (empirical approach)

Front 74

What are probability sampling techniques?

Back 74

-simple random sampling
-stratified sampling
-cluster sampling

Front 75

What is simple random sampling?

Back 75

Researchers try and select participants at random from the population.

*All observations have equal opportunity to be selected.

Front 76

What is cluster sampling?

Back 76

Population is divided into clusters (groups) and entire clusters are randomly selected.

randomly select clusters

Front 77

What is stratified sampling

Back 77

Population is divided into homogeneous, mutually exclusive groups (strata; e.g., neighbourhoods).

observations are randomly selected from within each stratum.

Front 78

What are two forms of non-probability sampling?

Back 78

-accidental/ad hoc/convenience/locally available
-purposive sampling

Front 79

What is accidental sampling?

Back 79

AKA convenience sampling

-Cases are selected because they are available.
-Very practical approach.
-Samples are not representative (i.e., there may be systematic differences between those in the study and those not in the study).

Front 80

What is purposive sampling?

Back 80

Researcher intentionally selects cases from defined groups (e.g., sick/not sick, managers, employed people, single parents).
-Groups are typically linked to hypotheses in a meaningful way.
-Non-equivalent design
-Regression discontinuity design

Front 81

What is a common criticism of non-probability sampling?

Back 81

•The sample used is not “representative.”
•Criticism is about “sample generalizability.”
– Results will not generalize
– No external validity
– Results are sample specific

Front 82

What is the evidence that "samples matter"?

Back 82

There is none!

Front 83

Why do people think sample characteristics matter?

Back 83

-People confuse random sampling with random assignment.
-People confuse statistical and theoretical generalizability.
-People rely on superficial similarities (“like Goes with like”)

Front 84

What questions should you ask when evaluating the appropriateness of a sample?

Back 84

• Did the research question contain a specific and well‐defined population of interest?
• Is there a characteristic of the convenience sample that may interact with the variables of interest in the study? (threat to external validity)
• If you are testing a theory, does the theory apply to the sample being used?

Front 85

What does significance at p < .05 mean?

Back 85

8. The probability of the data given the null hypothesis is < .05.

Front 86

What is the general sequence of events in significance testing?

Back 86

• Compute a sample statistic (e.g., mean).
• Estimate sampling error (standard error).
• Compare our sample statistic with null (e.g., mean = 0) accounting for sampling error: ratio
(e.g., t-value, z-value).
• Converte ratio to an exact probability (p-value) using tables, computers.
• compare p value against alpha (established critical value ahead of time).

Front 87

On a t-distribution, what is the t-value of a 5% cutoff?

Back 87

1.67

Front 88

What does significance testing actually tell us?

Back 88

Given that the null is true, what is the probability of these (or more extreme) data occurring.

*Important: p always refers to the probability of the data. The null is always assumed to be true.

Front 89

What is the biggest problem with significance testing?

Back 89

misinterpretation!

Front 90

p values are a reflection of what?

Back 90

sample size and effect size

Front 91

What is a Type I Error?

Back 91

rejecting the null hypothesis when the null is true

Front 92

What is a Type II Error?

Back 92

failing to reject null hypothesis when the null is false.

Front 93

What is the most common thing that leads to Type I and II errors?

Back 93

sample size

Front 94

What are the implications of p values being a reflection of both sample size and effect size?

Back 94

(1) a small effect can be statistically significant in a large sample and
(2) a large effect can fail to be statistically significant in a small sample.

Front 95

What is power?

Back 95

the probability of getting a statistically significant result when there is a real effect in the population

Front 96

What are factors that affect power?

Back 96

– study design (stronger manipulations have more of an effect)
– level of statistical significance (this can be set to levels other than .05 ahead of time)
– type of statistical analysis used
– measurement error
– sample size
– effect size

Front 97

What is the "crud factor"?

Back 97

Almost all of the variables that we measure are correlated to some extent.

Front 98

With regard to "crud factor," what is the implication for Type I errors under the nil hypothesis?

Back 98

they would not exist!

things are always going to be non-zero to start with

Front 99

What is the "Inverse probability error"?

Back 99

thinking that p is the probability that the null hypothesis is true

Front 100

What is the "Odds against chance fallacy"?

Back 100

thinking that p is the probability that the results happened due to chance

Front 101

What is the "Local Type I error fallacy"?

Back 101

If p < .05 then the probability that you are wrong in rejecting the null is less than 5%.

Front 102

What is the "Replication fallacy"?

Back 102

p is the probability of finding the same result in another study.

Front 103

What is the "Magnitude Fallacy"?

Back 103

p = effect size

Front 104

What is the "Meaningfulness (causality fallacy) fallacy"?

Back 104

Rejection of null hypothesis confirms the research hypothesis behind it

Front 105

What is the "Success/Failures fallacy"?

Back 105

rejection = success

failure to reject = failure

Front 106

What is the "Zero (equivalence) fallacy"?

Back 106

Failure to reject the null means that the effect is zero.

Front 107

What is the "Sanctification fallacy"?

Back 107

Thinking that there is a big difference between .049 .06, .056, .05 (etc)

Front 108

What is the "Reification fallacy"?

Back 108

Replication success is determined by examining p values.

Front 109

p refers to

Back 109

PROBABILITY OF THE DATA!

Front 110

Name the three requirements to measure causality.

Back 110

-temporal precedence
-no other variables to explain the effect
-actual relationship

Front 111

Describe external validity with one word.

Back 111

Generalizability!

Front 112

What does "meta-analytic thinking" include?

Back 112

-report results so that they can be included in a future meta-analysis (stats to calculate effect size)
-view your study as making at BEST a modest contribution to lit
-appreciate previous study results (especially effect size)
-compare new results to previous effect sizes

Front 113

What is "impact factor"?

Back 113

descriptive quantitative measure of overall journal quality
-number of times a typical article in the journal has been cited in other work

Front 114

What is "information fatigue/burnout"?

Back 114

problem of managing an exponentially growing amount of information

Front 115

what is short citation half-life?

Back 115

few works are cite more than a couple of years after they are published

Front 116

What is the "Trinity of Research"?

Back 116

-design
-measurement
-analysis

Front 117

What are the five structural elements of an empirical study?

Back 117

1. Samples (groups)
2. Conditions (eg. Treatment/control)
3. Method od assignment to groups or conditions
4. Observations
5. Time, or the schedule for measurement or when treatment begins or ends.

Front 118

What are the two kinds of extraneous variables?

Back 118

-nuisance (noise) variables that introduce irrelevant or error variance that reduces measurement precision

-cofounding/lurking variables cannot be distinguished from the dependent variable

Front 119

What are the three essential purposes of measurement?

Back 119

-identify and define variable of interest
-operational definition
-scores

Front 120

What are the three main goals of analysis?

Back 120

-estimate covariances between variables of interest
-estimate the degree of sampling error associated with this covariance
-test the hypotheses in light of the results

Front 121

What is a confidence interval?

Back 121

a range of values that may include that of the population covariance within a specified level of uncertainty

Front 122

What is a point estimate?

Back 122

covariance determined between the IV and DV

Front 123

What is interval estimation?

Back 123

Estimation of the degree of sampling error associate with the covariance.

Front 124

How are confidence intervals often represented?

Back 124

error bars

Front 125

What is conclusion validity?

Back 125

correct implementation of the design

(1) whether the correct method of analysis was used
(2) whether the value of the estimated (sample) covariance approximates that of the corresponding population value.

Front 126

What is "rater drift"?

Back 126

tendency for raters to unintentionally redefine standards across a series of observations

Front 127

What are threats to construct validity?

Back 127

-unreliable scores (unprecise or inconsistant)
-poor construct definition
-monomethod bias (different variables rely on same method or informant)
-evaluation apprehension (anxiety)
-reactive self report changes (want to be in treatment condition)
-researcher expectancies

Front 128

What are some threats to conclusion validity?

Back 128

•Unreliable scores
•Unreliability of treatment implementation
•Random irrelevancies in study setting
•Random within-group heterogeneity
•Range restriction
•Inaccurate effect size estimation
•Overreliance on stat tests (e.g. ignore effect size)
•Violated assumptions of stat tests
•Low power
•Fishing and inflation of Type I error (e.g. don't correct for familywise error etc)

Front 129

What is the principle of proximal similarity ?

Back 129

the evaluation of generalizability of results across samples, settings, situations, treatments, or measures that are more or less similar to those in the original study

-both pop and eco validity fall under this

Front 130

What does quantitative research emphasize?

Back 130

-Classification and counting behaviour
-Analysis of numerical scores with formal statistical methods
-Role of the researcher as an impassive, objective observer

Front 131

What are comparative studies?

Back 131

at least two different groups or conditions are compared on an outcome (dependent) variable

Front 132

What are the three basic types of research questions?

Back 132

-descriptive
-relational
-causal

Front 133

What is a time series?

Back 133

a large number of observations made on a single variable over time

Front 134

What is an Interrupted time series design?

Back 134

goal is to determine whether some discrete event – an interruption – affected subsequent observations in the series