Research Methods

Front 1

Statistical Conclusion Validity

Back 1

The validity of inferences about the correlation (covariation) between treatment and outcome.

Front 2

Treats to Statistical Conclusion Validity

Back 2

-Low Statistical Power
-Violated Assumptions of Statistical Test
-Fishing and Error Rate Problem
-Unreliability of Measures
-Restriction of Range
-Unreliability of Treatment Implementation
-Extraneous Variance in the Experimental Setting

Front 3

Low Statistical Power

Back 3

An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant.

Front 4

Violated Assumptions of Statistical Test

Back 4

Violations of statistical test assumptions can lead to either overestimating or underestimating the size and significance.

Front 5

Fishing and Error Rate Problem

Back 5

Repeated tests for significant relationships, if uncorrected for the number of tests, can artificially inflate statistical significance.

Front 6

Unreliability of Measures

Back 6

Measurement error weakens the relationship between two variables and strengthens or weakens the relationship between three or more variables.

Front 7

Restriction of Range

Back 7

Reduced range on a variable usually weakens the relationship between it and another variable.

Front 8

Unreliability of Treatment Implementation

Back 8

The effects of partial implementation may be underestimated compared to full implementation.

Front 9

Extraneous Variance in the Experimental Setting

Back 9

Some features of an experimental setting may inflate error, making decision, of an effect more difficult.

Front 11

Internal Validity

Back 11

The validity of inferences about whether the observed co-variation between the treatment and outcome reflects a causal relationship.

Front 12

Threats to Internal Validity

Back 12

-Ambiguous Temporal Precedence
-Selection
-History
-Maturation
-Regression
-Attrition
-Testing
-Instrumentation

Front 13

Ambiguous Temporal Precedence

Back 13

Lack of clarity about which variable occurred first, may yield confusion about which variable is the cause and which is the effect.

Front 14

Selection

Back 14

Systematic differences over conditions in respondent characteristics that could also cause the observed effect.

Front 15

History

Back 15

Events occurring concurrently with treatment could cause the observed effect.

Front 16

Maturation

Back 16

Naturally occurring changes over time could be confused with a treatment.

Front 17

Regression

Back 17

When units are selected for their extreme scores, they will often have less extreme scores on the other variables, which in turn can be confused with a treatment.

Front 18

Ambiguous Temporal Precedence

Back 18

Lack of clarity about which variable occurred first, may yield confusion about which variable is the cause and which is the effect.

Front 19

Selection

Back 19

Systematic differences over conditions in respondent characteristics that could also cause the observed effect.

Front 20

History

Back 20

Events occurring concurrently with treatment could cause the observed effect.

Front 21

Maturation

Back 21

Naturally occurring changes over time could be confused with a treatment.

Front 22

Regression

Back 22

When units are selected for their extreme scores, they will often have less extreme scores on the other variables, which in turn can be confused with a treatment.

Front 23

Attrition

Back 23

Loss of respondents to treatment or to measurement can produce artificially effects if that loss is systematically correlated with conditions.

Front 24

Testing

Back 24

Exposure to a test can affect scores on subsequent exposures to that test, an occurrence that can be confused with a treatment effect.

Front 25

Instrumentation

Back 25

The nature of a measure may change over time or conditions in a way that could be confused with a treatment effect.

Front 26

Construct Validity

Back 26

The validity of inferences about the higher order constructs that represent sampling participation.

Front 27

Threats to Construct Validity

Back 27

-Inadequate explication of constructs
-Construct confounding
-Mono-operation bias
-Mono-method bias
-Reactivity to the experiment situation
-Experimenter expectancies
-Novelty and disruption effects
-Compensatory equalization
-Compensatory rivalry
-Resentful demoralization
-Treatment diffusion

Front 28

Inadequate Explication of Constructs

Back 28

Failure to adequately explicate a construct may lead to incorrect inferences about the relationship between operation and construct.

Front 29

Construct Confounding

Back 29

Operations usually involve more than one construct, and failure to describe all the constructs may result in incomplete construct inferences.

Front 30

Mono-Operation Bias

Back 30

Any one operationalization of a construct both under-represents and construct of interest and measures irrelevant constructs, complicating inference.

Front 31

Mono-Method Bias

Back 31

When all operationalizations use the same method, that method is part of the construct actually studied.

Front 32

Reactivity to the Experimental Situation

Back 32

Participant responses reflect not just treatments and measures but also participants; perceptions of the experimental situation, and those perceptions are part of the treatment construct actually tested.

Front 33

Experimenter Expectancies

Back 33

The experimenter can influence participant responses by conveying expectations about desirable responses, and those expectations are part of the treatment construct as actually tested.

Front 34

Novelty and Disruption Effect

Back 34

Participants may respond unusually well to a novel innovation or unusually poorly to one that disrupts their routine, a response that must then be included as a part of the treatment construct description.

Front 35

Compensatory Equalization

Back 35

When treatment provide desirable goods or services, administrators, staff, or constituents may provide compensatory goods or services to those not receiving treatment, and this action must then be included as a part of the treatment construct description.

Front 36

Compensatory Rivalry

Back 36

Participants not receiving treatment may be motivated to show they can do as well as those receiving treatment, and this compensatory rivalry must then be included as part of the treatment construct description.

Front 37

Resentful Demoralization

Back 37

Participants not receiving treatment may be so resentful or demoralized that they may respond more negatively than otherwise and this resentful demoralization must then be included as a part of the treatment construct description.

Front 38

Treatment Diffusion

Back 38

Participants may receive services from a condition to which they were not assigned, making construct descriptions of both conditions more difficult.

Front 39

External Validity

Back 39

The validity of inferences about whether the cause-and-effect relationship holds over variation in persons, settings, treatments and measurements.

Front 40

Threats to External Validity

Back 40

-Interaction of the Causal Relationship with Units
-Interactions of the Causal Relationship over Treatment variations
-Interaction of the Causal Relationship with Outcomes
-Interaction of the Causal Relationship with Settings

Front 41

Interaction of the Causal Relationship with Units

Back 41

An effect found with certain kinds of units might not hold if other kinds of units had been studied.

Front 42

Interaction of the Causal Relationship over Treatment Variations

Back 42

An effect found with one treatment is combined with other treatments, or when only partial treatment is used.

Front 43

Interaction of the Causal Relationship with Outcomes

Back 43

An effect found one kind of outcome observation may not hold if other outcome observations were to be used.

Front 44

Interaction of the Causal Relationship with Settings

Back 44

An effect found in one kind of setting may not hold if other kinds of settings were to be used.