Threats To Validity

Front 1

Ambiguous temporal precedence

Back 1

We don’t know which variable came first.

< internal >

Front 2

Selection

Back 2

A difference in respondents could cause the effect.

< internal >

Front 3

History

Back 3

Events occurring at the same time as the treatment could cause the effect.

< internal >

Front 4

Maturation

Back 4

Naturally occurring change could cause the effect.

< internal >

Front 5

Regression to the mean

Back 5

Respondents selected for extreme scores will score lower on other/later measures.

< internal >

Front 6

Attrition

Back 6

Losing participants could affect the causal relationship.

< internal >

Front 7

Testing

Back 7

Exposure to a test can affect other measures (e.g., practice).

< internal >

Front 8

Instrumentation

Back 8

The nature of a measure may change over time based on definitions or new research.

< internal >

Front 9

Additive and interactive threats

Back 9

selection x history/maturation/attrition

< internal >

Front 10

Inadequate explication of constructs

Back 10

Lack of clarification/definition.

< construct >

Front 11

Construct confounding

Back 11

Lack of distinction from a similar construct.

< construct >

Front 12

Mono-operation bias

Back 12

Only one operationalization used.

< construct >

Front 13

Mono-method bias

Back 13

All operationalizations use the same method.

< construct >

Front 14

Confounding constructs with levels of constructs

Back 14

When the construct being measured is one specific dimension of a more general construct (i.e., physical violence vs aggression)

< construct >

Front 15

Treatment sensitive factorial structure

Back 15

When people exposed to a treatment see a measure differently than those who aren’t.

< construct >

Front 16

Reactive self-report changes

Back 16

Participant motivation to be in a treatment condition can affect self-report.

< construct >

Front 17

Reactivity to the experimental situation

Back 17

Simply being in the experimental situation could influence the relationship and should then be considered a part of the “treatment” definition.

< construct >

Front 18

Experimenter expectancies

Back 18

The experimenter’s conveying of expectations could affect behavior and should then be considered a part of the “treatment” definition.

< construct >

Front 19

Novelty and Disruption effects

Back 19

New innovations >> positive response.

Disruptions to routines >>negative response.

Either should then be considered a part of the “treatment” definition.

< construct >

Front 20

Compensatory equalization

Back 20

When one condition receives a “treatment” and gets some benefit, and then the other condition receives a different benefit to equal out the playing field, this should be considered a part of the “treatment” definition.

< construct >

Front 21

Compensatory rivalry

Back 21

Participants not receiving treatment may be motivated to show they can do as well as the treatment condition, which should be a part of the “treatment” definition.

< construct >

Front 22

Resentful demoralization

Back 22

Not receiving treatment could lead to more negative self-report which should be a part of the “treatment” definition.

< construct >

Front 23

Treatment diffusion

Back 23

Participants may receive treatments in a condition that they weren’t assigned to.

< construct >

Front 24

Interaction of the causal relationship with units

Back 24

An effect found with certain kinds of units might not hold if other kinds of units had been studied.

< external >

Front 25

Interaction of the causal relationship over treatment variations

Back 25

An effect found with one treatment might not hold with other treatments.

< external >

Front 26

Interaction of the causal relationship with outcomes

Back 26

An effect found with one kind of outcomes measure might not be found if another were used.

< external >

Front 27

Interaction of the causal relationship with settings

Back 27

An effect found in one setting might not be found in another.

< external >

Front 28

Context-dependent mediation

Back 28

An explanatory mediator of a causal relationship in one context may not mediate in another context.

< external >

Front 29

Low statistical power

Back 29

Is a threat when sample sizes are too small or when alpha is set low. This is because low statistical power increases the likelihood of making a Type II error (accepting null when it is false).

< SCV >

Front 30

Violated assumptions of statistical tests

Back 30

Is a threat when the assumptions underlying statistical tests (e.g., normality) are not met, biasing statistical tests.

< SCV >

Front 31

Fishing and error rate problem

Back 31

Numerous multiple comparisons (when conducting a large number of statistical tests) increases chance of randomly finding significance -- Type I error.

< SCV >

Front 32

Unreliability of measures

Back 32

Unreliable scales cannot be relied on for detecting true differences or changes (less than .7-.8).

< SCV >

Front 33

Range restriction

Back 33

Floor/ceiling effects, or clustering around the mean. If you ain't got variance, you ain't got covariance.

< SCV >

Front 34

Unreliability of treatment implementation

Back 34

Variations in treatment introduce error, decreasing the chance that a true difference will be detected.

< SCV >

Front 35

Random irrelevancies/extraneous variance in the experimental setting

Back 35

Irrelevant nontreatment features of the experimental setting may influence scores on the dependent variable (e.g., excessive noise for one group but not another) by inflating error variance.

< SCV >

Front 36

Random heterogeneity of respondents

Back 36

If respondents in any treatment group differ on factors that are also correlated with the DV, may cause certain kinds of respondents will be more affected by a treatment than others (a matter of external validity). Error variance may also be inflated.

< SCV >

Front 37

Inaccurate effect size estimation

Back 37

Causes unreliable estimations of covariance < SCV >