Threats To Construct Validity

Front 1

Threat to Construct Validity

Back 1

This is when researchers did not actually measure the construct they intended to measure.

Front 2

inadequate explication of constructs

Back 2

When researchers do not get at a construct completely. This could be when researchers only measure part of a construct. Or when they measure the wrong construct. Or when they use a measure with low reliability or validity. Or when they use a measure that is too broad to get at the measure they want. Or when they only use part of a measure but don’t check the reliability and validity of this portion of the measure.

Hint 2

Remedy: Use reliable and valid measures that get at your construct well.
Risk: Common. Look for low reliability and validity, or not getting at constructs well.
Look in: Intro to see what they are intending to measure. Methods and Measures to see what they use and see if reliability etc are mentioned. Discussion to see if this is mentioned in limitations.

Front 3

Construct confounding

Back 3

There is a construct that the researchers have not accounted for that is responsible for causing or affecting the results and relationship found. There is a construct here that is unaccounted for (missing). The must some evidence for this, good reason to believe this is a problem.

Hint 3

Remedy: Account for any possible constructs that could affect your findings.
Risk: Medium-High, but be careful not to abuse this threat (make sure you have good reason to suspect it’s a problem before using it.
Look in: Methods

Front 4

Mono-operation bias

Back 4

This is a problem when only one instrument (measure) is used to measure ONE construct. For any particular construct in the study, if there was only one thing measuring it, this is a problem. It may occur multiple times in a study, but only talk about one of those times when asked to describe a threat.

Hint 4

Remedy: Use more than one measurement to measure each of your constructs. (to measure the construct of depression, don’t just use the BDI- use another depression measure as well).
Risk: SUPER COMMON. In some studies, researchers will use multiple measures for most of their constructs, but there is usually at least one construct that only has one measure used for it.
Look in: Methods in MEASURES

Front 5

Mono-method bias

Back 5

This is the same thing except replace “instrument” or “measure” with “method”. Researchers could use more than one measure for any ONE construct, but if that ONE construct only has ONE method used to measure it, you may not be getting at the construct as well as you could.

Hint 5

Remedy: Use more than one method for each of your constructs.
Risk: EXTREMELY COMMON. However, if researchers did a good job with MOST of their constructs, it may not be appropriate to apply this threat for only using one method for one of their constructs.
Look in: Methods under MEASURES

Front 6

Confounding constructs with levels

Back 6

This is when a study draws conclusions about a construct while they looked at limited levels of the construct. When researchers leave out certain levels of a construct. This makes your ability to draw conclusions about things weaker.

Hint 6

Remedy: Include all the different possibilities for each construct. Do not leave out any options for a variable.
Risk: Medium. Though it is not extremely common, this does seem to occur in a fair amount of articles.
Look in: Methods, under measures.

Front 7

Treatment-sensitive factorial structure

Back 7

This is when participants who are exposed to treatment see a measure in a different way than those who were not exposed to the treatment. For people who have received the treatment, the answers may be broken down into different “factor structures”. People who did not receive the treatment have answers that result in a single factor. The structure of the measure is different between the people who received treatment and the people who didn’t.

Hint 7

Remedy: Make sure your treatment doesn’t affect the way people will respond to your measure in some hidden way that is different from the non-treatment group. OR make sure you see how it effects them and take this into account by creating a special measure for them (although this creates a whole other problem).
Risk: EXTREMELY LOW.
Look in: Methods? Limitations? (they may have to explicitly admit to this happening)

Front 8

Reactive Self-report changes

Back 8

This is when people change the way they are answering in a self-report because they want to be accepted to the study of receive treatment, or because they want to appear socially acceptable.

Hint 8

Remedy: don’t use self-report
Risk: Medium, although it could be difficult to identify when this is a problem.
Look in: Methods

Front 9

Reactivity to the experimental situation

Back 9

This is when something in the experimental situation is affecting the way the participants respond. This could include the Hawthorne effect- a reaction to something in the environment. It could also include hypothesis guessing done by the participants, or the placebo effect.

Hint 9

Medium-Low

Front 10

Experimental Expectancies

Back 10

This is also called the Rosenthal effect. It is when the experimenter conveys the message of what is wanted from the participant and that leads PT to act in a certain way. Instead of actually measuring the construct naturally in the participant, you are just picking up on how the participant is trying to meet expectations.

Hint 10

Medium-Low risk

Front 11

Novelty & Disruption Effects

Back 11

This is when participants may respond better to things that are new. So instead of just measuring the construct as it is meant to be, the fact that the treatment or measure is new to them may skew things. OR they may respond particularly poorly or poorly to something that disrupts their routine.

Hint 11

Medium/Low risk

Front 12

Compensatory Equalization

Back 12

This is when treatment provides something very desirable (researchers are testing a Free Ice-Cream treatment), and researchers attempt to compensate by giving things to the non-treatment group (vouchers they can use for ice cream after the study). Doing this must then be included as part of the treatment construct description.

Hint 12

Medium/Low risk

Front 13

Compensatory Rivalry

Back 13

This is when (competitive) participants who are not receiving treatment may be motivated to show they can do as well as those receiving treatment. This rivalry motivation could change what you are measuring, and must then be included as part of the treatment construct description.

Hint 13

Rare/Specific

Front 14

Resentful Demoralization

Back 14

This is when there is a very desirable treatment (Free Ice Cream Therapy). Participants not receiving this desirable treatment may be so resentful or demoralized that they may respond more negatively than otherwise, and this resentful demoralization must then be included as part of the treatment construct. The people not getting the free ice cream may be upset and report extra depression because they don’t care anymore. Then the results would indicate that the ice cream cured depression way better than the control group, when really the people not getting ice cream were not depressed, but answered that way because they were cranky from not getting ice cream.

Hint 14

Rare/Specific

Front 15

Treatment diffusion

Back 15

This is when participants may are getting extra services outside of the study. For example, a CBT was tested for use with blind people to help them see. Meanwhile, some people from the CBT group, and some from the control group were getting vision surgery. And others were getting homeopathic or religious treatments. And others were getting some other vision treatments. Now it is difficult to define what services each group got, because they got extra services outside of the study.

Hint 15

Rare/Specific