Threats To External Validity

Front 1

External Validity

Back 1

This is when the results found are not generalizable to the population of interest.

Front 2

Selection (Interaction of Causal Units)

Back 2

This is a threat when the participants used in a study are of a specific type such that the results found with them might not apply (be generalizable) to other participants, but researchers imply this.

Hint 2

Remedy: Either try to have participants that are very representative (don’t miss out on certain people by only doing college students or white people or high SES etc), or don’t try to generalize to other people.

Risk: EXTREMELY HIGH. This happens ALL THE TIME in studies.

Look in: PARTICIPANTS to see who was used for the study. Then look at the title, abstract, and/or discussion to make sure you know who they are trying to generalize the results to and talk about.

Front 3

Interaction of the causal relationship over treatment variations

Back 3

This is a problem when the research study has a TREATMENT, but a weird variation of the treatment is used, and results found may not be generalizable to other variations of that treatment.

Hint 3

Remedy: Use the whole treatment as it was meant to be given.
Risk: Low. Researches would likely admit that they administered a treatment differently than it was supposed to be administered.
Look in: Methods, measures, and Limitations

Front 4

Interaction of the causal relationship with outcomes

Back 4

This is similar to the idea of mono operation bias, but just for the DV. This is a threat when the outcome (usually just one Dependent Variable) was measured a certain way (usually with only one measure). It is a problem when measuring the outcome in a different way could have given different results.

Hint 4

Remedy: Use more than one way to measure the DV.
Risk: Medium. Look for just one DV with just one measure that could have been measured differently. (If the one outcome was height and they used a ruler, do NOT apply this threat, because there is no other way to measure height and get different results).
Look in: Methods

Front 5

Interaction of causal relationship with settings

Back 5

This is a threat when the research was done in a particular setting (environment), and results may not be generalizable to a different setting. By setting we are talking about specific research setting such as laboratory, school, home, internet, ect. NOT geographic location like Texas or New York. Geographic locations are more of a selection threat because we are talking about different types of people with results that may not generalize to other types of people. Here we are talking about a setting where results may not generalize to other settings.

Hint 5

Remedy: Try to do studies in the appropriate types of settings you are trying to generalize to or in more than one setting.
Risk: HIGH. Only 1 setting is used in most studies.
Look in: Methods in Procedure

Front 6

Context-dependent Mediation

Back 6

This is a threat when a mediating variable may not mediate the same way in a different situation (different setting, or participants, or treatment, or task, ect). This is only when the study has a mediating variable, and this relationship (the variable mediating the relationship between two other variables) may not be the same in a different situation.

Hint 6

Remedy: Try to use generalizable situations.
Risk: RARE. The study must have a mediating variable in it.
Look in: Methods, Results, and Conclusions

Front 7

Interaction of History and Treatment

Back 7

This is different than the Internal Validity threat History. That is when an event affects participants and causes the results found. THIS one is when results found from a study are not generalizable to other time periods. This usually has to do with whole eras. It mostly applies to old studies, when times were different, and results found may not be the same today.

Hint 7

Remedy: Replicate studies over time to see if they still apply.
Risk: Rare.
Look in: Date of publication, Methods