Reading...
Play button
Play button
Use LEFT and RIGHT arrow keys to navigate between flashcards;
Use UP and DOWN arrow keys to flip the card;
H to show hint;
A reads text to speech;
45 Cards in this Set
- Front
- Back
- 3rd side (hint)
|
What is Validity?
|
•The best available Approximation to the truth or falsity of a given inference, proposition, or conclusion
•A set of standards by which research can be judged |
|
|
|
Internal Validity (and 3 criteria)
|
must be a causal relationship
•3 criteria for internal validity –IV and DV are meaningfully related; not a chance occurrence –Variation in IV is contemporaneous with or precedes, variation in the DV –There is reasonable conceptual explanation AND THERE ARE NO PLAUSIBLE ALTERNATIVE EXPLANATIONS |
|
|
|
Conclusion Validity
|
Is there a relationship between...
•What you did and what you saw? •Your program and your observations? |
|
|
|
Internal Validity
|
Is there a causal relationship between...
•What you did and what you saw? •Your program and your observations? |
|
|
|
The Causal Context
|
|
|
|
|
Model of Empirical research
|
•3 Conventions Underlying Model
–independent and dependent variables are identified by x and y, respectively –the prime symbol is used to designate that a variable is specified at the conceptual level –arrows represent direction of influence or cause |
|
|
|
Model of Empirical Research Conceptual Relationships
|
Conceptual Relationships
–Represented by Line (a), and usually reflects researchers expectations •such expectations stated before (a priori) not after (post hoc), and are referred to as hypotheses •to degree relationship holds it adds validity to the researchers conceptualization •Never, ever 100% valid; all conclusions or inferences are tentative = verisimilitude |
|
|
|
Model of Empirical Research •Operational Relationships
|
Operational Relationships
–an empirical relationship is line (d) •solid because relationship actually observed usually through some form of statistical procedure –a causal relationship is line (c) •when causality is an issue, research must do more than establish an empirical association •must establish internal validity; confidence we have that IV caused variable in DV |
|
|
|
Conceptual Relationships in Empirical Research Model
|
–Represented by Line (a), and usually reflects researchers expectations
•such expectations stated before (a priori) not after (post hoc), and are referred to as hypotheses •to degree relationship holds it adds validity to the researchers conceptualization •Never, ever 100% valid; all conclusions or inferences are tentative = verisimilitude |
|
|
|
Model of Empirical Research - Conceptual to Operational Relationships
|
Conceptual to Operational Relationships
–Represented by lines (b1) and (b2) –construct validity is present when there is a high correspondence between the observed scores and the mental definition |
|
|
|
Construct Validity
|
|
|
|
|
External Validity
|
|
|
|
|
Validity Questions
|
They are cumulative. Can't skip steps
|
|
|
|
–Statistical Conclusion Validity
|
Is there a relationship between the two constructs?
|
|
|
|
–Internal Validity
|
Given that there is a relationship, is it plausibly causal from one operational variable to the other or would the same relationship have been obtained in the absence of any treatment of any kind?
|
|
|
|
Construct Validity
|
Given that the relationship is plausibly causal and is reasonably known to be from one variable to another, what are the particular cause and effect constructs
involved in the relationship? |
|
|
|
External Validity
|
Given that there is probably a causal relationship from construct A to construct B, how generalizableis the relationship across persons, settings, and times?
|
|
|
|
Threats to Validity
|
You Want to Make an Inference...
•There IS a relationship between the cause and effect. •The relationship is causal. •You can generalize to the constructs. •You can generalize to other persons, places, and times. |
|
|
|
Threats to Conclusion Validity
|
* There is a relationship, but you don’t see it. (Type II - False Negative)
* There is no relationship, but you do see one (Type I - False Positive) |
|
|
|
Type I Error (α)
|
Type I (α): reject the null ypothesis when the null hypothesis is true (false positive)
|
|
|
|
Type II Error
|
Type II (β): fail to reject the null Hypothesis when the null Hypothesis is false (False Negative)
|
|
|
|
Threats to Internal Validity
|
* There is a causal relationship, but you don’t see it. (Type II) OR
* There is no causal relationship, but you do see one. |
|
|
|
Threats to Construct Validity
|
•You can generalize to constructs, but you conclude you can’t.
•You can’t generalize to constructs, but you conclude you can. |
|
|
|
Threats to External Validity
|
•You can generalize to other contexts,
but you conclude you can’t •You can’t generalize to contexts, but you conclude you can |
|
|
|
Statistical Conclusion Validity
|
•Are the presumed cause and effect related based on the validity of our statistical analysis?
1.Is the study sensitive enough to permit reasonable statements about covariation? –Statistical Power (especially if relationship non-significant) 2.If it is sensitive enough, is there any reasonable evidence from which to infer that the presumed cause and effect covary? –Type I and Type II errors 3.If there is such evidence, how strongly do the two variables covary? –Statistical vs. practical significance |
|
|
|
7 Major Threats to Statistical Validity
|
•Low Statistical Power
•Violated Assumptions of statistical Tests •Fishing and the Error Rate Problem •The Reliability of Measures •The Reliability of Treatment Implementation •Random Irrelevancies in the Experimental Setting •Random Heterogeneity of Respondents |
|
|
|
How to reduce error terms?
|
•Individuals act as their own controls
•Select homogeneous samples •Matching •Ex-post blocking •Covariance analysis •Reliable measures |
|
|
|
Type I & II Errror Diagram
|
|
|
|
|
Internal Validity
|
Is the relationship between the two
variables causal and in what direction? –Cause precedes effect in time –There is a correlation between cause and effect – All rival alternative explanations are ruled out (third variable alternative interpretations) |
|
|
|
Threats to Internal Validity
|
Selection Threats
•History •Maturation •Testing •Instrumentation •Statistical Regression •Selection •Mortality •Interactions with Selection •Ambiguity about Direction of causal Influence Social Threats •Diffusion or Imitation of Treatments •Compensatory Equalization of treatments •Compensatory Rivalry by Respondents Receiving Less Desirable Treatments •Resentful Demoralization of Respondents Receiving Less Desirable Treatments |
|
|
|
Randomization
|
Addresses Selection threats but not social threats to Internal Validity
|
|
|
|
Assessing Plausibility of Threats to Internal Validity after Randomization
|
•Assess proportion of respondents who actually
provide post-test data to see whether differential attrition (mortality) across treatments •Examine pretest scores for those who provided post- test data. Indicates whether drop-outs differed across groups on key background characteristics that could influence post-test. •Assess imitation, compensatory equalization, compensatory rivalry by focusing on the process via which IV was meant to influence DV. |
|
|
|
Construct Validity
|
Construct Validity
The degree to which measures of construct actually measure the construct they are intended to measure •Can I generalize from the specific measures of the construct to the referent construct? •Do the measures truly capture the intended theoretical construct? |
|
|
|
Bagozzi's 6 Aspects of Construct Validity
|
–Theoretical meaningfulness of construct
–Operational meaningfulness –Reliability –Convergent validity –Discriminant validity –Nomological validity |
|
|
|
Theoretical Meaningfulness
|
* Construct defined in theoretical terms
* Domain defined * Dimensionality defined: unidimensional versus multidimensional |
|
|
|
Operational Meaningfulness
|
•Do items correspond to the theoretical definition of the construct?
•Do they cover the domain of the construct? That is: Content validity –The degree to which elements of an assessment instrument are relevant and representative to the targeted construct |
|
|
|
Reliability
|
•Test-retest reliability
–Correlation of the same person’s score on the same items at two different points in time •Internal consistency –The interrelatedness among items or sets of items in the scale |
|
|
|
Convergent Validity
|
•The degree to which two maximally different measures designed to measure the same construct are related.
•Convergence is found if two different measures of the same construct are highly correlated |
|
|
|
Discriminant Validity
|
•The degree to which two measures designed to measure similar or conceptually related constructs are distinct.
•Convergent and discriminant validity are assessed through a number of statistical techniques |
|
|
|
Nomological Validity
|
The degree to which predictions from a formal theoretical network containing the construct are confirmed
|
•Criterion-related validity: typically whether the construct correlates well with other measures known to tap at construct
•Known group validity: Does measure distinguish between groups that are known to have different levels of the trait (e.g., high/low) |
|
|
Threats to Construct Validity
|
•Inadequate Pre-operational Explication of Constructs
•Mono-operation bias •Mono-method bias •Mono-method bias •Hypothesis-guessing within Experimental Conditions •Evaluation Apprehension •Experimenter Expectancies •Confounding Constructs and Levels of Constructs •Interaction of Different Treatments •Interaction of Treatment and Testing •Restricted Generalizability Across Constructs |
|
|
|
Reducing Monomethod Bias Threats
|
–Use multiple methods
–Include reversed scaled items –Vary whether positive or negative end of response scale appears on left or right of page –(2 and 3 above also have critics) |
|
|
|
External Validity
|
•Does the causal relationship generalize to and across different types of persons, settings, and times?
|
|
|
|
Threats to External Validity
|
•Interaction of Selection and Treatment (people)
•Interaction of Setting and Treatment (place) •Interaction of History and treatment (time) |
|
|
|
Increaseing Internal Validity
|
Specify Target Population
•Random Sampling for Representativeness(most powerful) •Deliberate Sampling for Heterogeneity •Impressionistic Modal Instance Model (least powerful of 3 but can be used when no targets are specified and can be used with convenience samples) •External Validity is a matter of REPLICATION |
|
