Reading...
Play button
Play button
Use LEFT and RIGHT arrow keys to navigate between flashcards;
Use UP and DOWN arrow keys to flip the card;
H to show hint;
A reads text to speech;
75 Cards in this Set
- Front
- Back
|
What are the Four Types of Validity?
|
1. Statistical.
2. Internal. 3. Construct. 4. External. |
|
|
Statistical Validity
|
The extent to which covariance is assumed to exist, given the type 1 error rate
If there is something there – how confident – have you documented what your found appropriately |
|
|
7 Threats to Statistical Validity
|
1. Low Power - your study does not have the ability to detect an effect.
2. Violated assumptions - when your data does not take on the form you are arguing for. 3. Familywise Error - occurs when you conduct too many statistical tests. 4. Unreliable measures - is the test a good measure of what you intended to measure. 5. Unreliable implementation of intervention occurs when you freelance your interventions. 6. Heterogenous Subjects - you do not evaluate subjects with similar qualities. 7. Random Factors - bad random sample. |
|
|
Statistical Validity a problem when?
|
Field work.
|
|
|
How to control statistical validity
|
Matching - subjects in treatment and control conditions match on relevant variables. Exact or conceptual.
Blocking. Only use subjects with certain level of variable. |
|
|
Internal Validity
|
ability to detect a true causal relationship, given multiple possible relationships
. Have you detected the correct causal relationship? |
|
|
Four possible States of Nature:
|
• A -> B
• B → a • A→C→B • A, B unrelated |
|
|
Internal Validity: 8 Threats due to lack of random assignment
|
History - something in external environment impacts study
Maturation - do some kind of repeat assessment Testing - over assess (give tests within one week) Instrumentation - respondents react to a feature of the question, rather than the ? itself. Regression - first test is unusually bad or good, second test is back to normal Selection - unusual sample (best way to fix is build sample) Mortality - people drop out of the study over time Selection Interactions - If a selection problem and another threat. |
|
|
Inability to Randomize (field work) can be due to:
|
- lack of control over subject assignment (ex- 5th graders are already assigned.)
- administrator dislike of treatment (ex: 5th grader principal does not like that one class is getting treatment and one is not) Admin dislike of inequity - administrator may do things to compensate those in the control group (ie hire a tutor) |
|
|
Internal Validity: 4 Information Flow Threats
|
1. Diffusion of Treatment - people talk a lot about what they did. Subjects know what's coming.
2. Resentment - person expects to receive one treatment but gets another, ie, he is in the control group. Could get hostile and sabatoge study 3. Compensatory equalization - takes steps to compensate themselves for being in disadventageous group. ie, brings in candy, or gets tutor. 4. Compensatory rivalry - members of disadvantaged group work harder to demonstrate they are OK. "change behaviors" |
|
|
Internal Validity: 1 Theoretical Threat
|
Ambigious Causality - can't tell which variable comes first. Happens when you study two things that are closely related, ie, intelligence and motivation
"unclear if A causes B, or if B causes A" |
|
|
Construct Validity
|
Did you measure what you intended to measure, or did you measure something else?
You said A causes B, but maybe A causes C, or C causes B. Right direction, but wrong pieces. |
|
|
Confound Variable
|
The something else you might have measured.
|
|
|
Example of Construct Validity
|
Placebo Effect. Is it the pill, or the taking of the pill that you are measuring?
|
|
|
10 Threats to Construct Validity
Six are under experimenters control: |
1. Inadequate operational definition - what do you mean by "effective"
2. Mono-operation bias - (one measure) - is there a construct always measured by a single measure? 3. Mono-method bias (one method) - only get the effect in a lab setting, or in a field setting. 4. Experimenter expectancy - experimenters own biases influence the results. 5. Novel Treatment - radical treatment, people react to the task, rather than the treatment. 6. Treatment of continuous variables as discrete. . If you ask a continuous question, keep the answer and interpretation continuous. Ex - IQ of 110 and above, 109 - 80, etc. |
|
|
10 Threats to Construct Validity
Four are NOT under experimenters control: |
1. Hypothesis Guesses - "good student effect"
2. Evaluation Apprehension - dealing with sensitive subjects, who worry about how they will look if they respond, ie, group sex 3. Pretest Sensitization - Pretest clues people in to what the study is about. 4. Construct Generalizability - trying to study a construct that is not situated for studying at this time. Are you measuring A -> B or X -> B, because A and X are so closely related. |
|
|
External Validity
|
The extent to which the results can be generalized. ie, focus on school kids in Seattle, not the entire world.
|
|
|
Sample is supposed to mirror what?
|
Population
|
|
|
Why can't samples capture all of the variation in a population?
|
Samples are biases
|
|
|
the mean is a _____ statistic?
|
unbiased
|
|
|
variance is a _____ statistic?
|
biased.
it always gives you very rough approximation of pop. variance even unbiased variance estimate is a bad estimate of population. |
|
|
Putting together a sample
|
need to assemble a sample that best reflects a certain aspect of the population and lets other aspects suffer
|
|
|
Sampling methods focus on?
|
Aspects
|
|
|
Sample Methods: Random Sample
|
Maximizes likelihood that sample will be broadly reflective of population.
- even with sufficiently large sample- can still get unusual sample - maybe what you need to be present is just not in your sample (or not sufficiently sample). I.e., pull random sample and it is 95% male and sex may have an influence – then this sample is no good. - most "random" samples are not truly random need to look at characteristics of pulled sample and make sure it is a good (not an unusual sample) sample. Do this by getting descriptive stats on the variables you will be using to make sure the sample is relatively normal |
|
|
Sample Methods: Stratified Sample
|
- "should replace random sample"
- break the sample into subgroups (strata), ie, ethnicity - randomly select cases from each subgroup - ensures representatives along key dimensions - Can provide excellent precision with smaller sample size - Main drawback: Can be hard to assemble |
|
|
Sample Methods: Cluster Sample
|
- Break entire sample into smaller units (clusters) not stratifying along a dimension. You are organizing along natural sub-groups. For example, in school district – there are 60 1st grade classrooms – each classroom is a cluster.
- Randomly select subset of clusters. I.e., pick 15 classrooms - Can work with all members of each selected cluster (one-stage sampling) - can randomly select members from selected clusters (two-stage sampling) o Often resource-efficient, but may be systematicity within subgroups. The mere fact that they are grouped together might introduce a possible confound – may introduce an unseen variable (i.e., some of the kids have a good teacher, others have bad) – teacher quality is a confound – it is an unseen variable. This is systematicity within the group. |
|
|
Sample Methods: Quota Sample
|
- trying to build a sample that mimics the population on some factor.
Ex: build sample of 55% women, 45% men if sex is critical - Similar logic to stratified sampling, but may not involve random selection |
|
|
Sample Methods: Fortuitous Sample
|
- "American Idol method"
- take whoever will respond - can't conclude much, but can be used for exploratory purposes |
|
|
Sample Methods: Modal Distance Sample
|
When working with intact group
Use group(s) that is most typical on key indicators |
|
|
No shows for study -
|
Why didn't they come?
Hope: for random reasons, If systematic, could be major limitation on study conclusions Good idea to track people down, but hard to do |
|
|
College student Samples: Good or Bad?
Cons: |
Cons:
- Very homogenous - Many features still ill-formed (attitudes, beliefs, understanding of the world) - inexperienced with real world |
|
|
College student Samples: Good or Bad?
Pros: |
Very homogeneous
Broad sample of society often hugely variable, members not comparable Amenable and helpful Willing to follow instructions, engage in novel tasks Convenient and cheap Are citizens engaged in society |
|
|
Pre WWII: No formal ethics code
|
True
|
|
|
1947: What to do with Nazi medical data?
|
How long can someone survive in the ocean in winter
How to revive hypothermia victims What happens to the brain at high altitude How long can one survive drinking sea water How does blood coagulate Wound treatment Can gangrene be treated |
|
|
Response to What to do with Nazi medical data?
|
Nuremberg Code
|
|
|
Nuremberg Code
|
Voluntary Consent to Participate
Data will benefit society Test on animals first Do not inflict suffering or injury Do not study if injury is expected (unless you study urself) Minimal risk to individuals Proper Protections (safe environment) Properly trained researchers Study subject can withdrawl at any time Experiment must end immediately of problems arise |
|
|
1932: The Tuskegge Study of Untreated Syphilis in the Negro Male
|
600 Black men
300 with syphilis 300 not infected Would be treated for “bad blood” Goal: Document natural history of syphilis To justify developing treatment program for Black men In exchange for participating, men offered Free checkups Free food Free funeral Never made clear to men that they could quit Supposed to last 6 months 1936: Decision made to track men until death Local doctors told not to treat study subjects 1940: Army told not to treat drafted study subjects 1945: Penicillin developed Effective against syphilis Not given to men 1968: Peter Buxtun CDC, AMA 1969 1972: Buxtun to NY Times 1974: Subjects and their families awarded free medical care until death 2004: Last subject dies |
|
|
1966: Henry Beecher, NEJM. Review of unethical studies
|
Found studies withholding treatment for rheumatic fever, typhiod, strep
Administration of liver-damaging drug to the incarcerated (including children) Injection of carbon dioxide into the lungs until cardiovascular system disrupted Administration of nitrogen to alcoholics to simulate coma Insertion of a needle into the heart to see how it reacts Continual bombardment of infant bladders with x-rays while filling and voiding |
|
|
Troubling Psychology Experiments
|
1963: Milgram
1969: Modeling of help 1971: Mincturation in presence of others |
|
|
1973: Am Psych Assn ethical principles for research
|
First discipline to publish ethical research principles
|
|
|
National Research Act (1974)
|
Created the Institutional Review Board (IRB)
Created National commission for the protection of Human Subjects |
|
|
1979: Belmont Report
Three Principles: (Only recommendations, NOT LAW) |
Respect for persons
- People can make their own decisions - Protect those who cannot Implies: informed consent, voluntary involvement Beneficence - Only do things that help Implies: risk-benefit analysis Justice - Be fair and equitable to subjects - Do not withhold a benefit without good reason Implies: Treating people fairly, Involve people from all walks of life |
|
|
Congress has never endorsed Belmont Report
|
True.
|
|
|
Special Ethical Considerations:
Children |
Can they understand what's going to happen?
Current answer: No. you need to solicit parental consent as if the parent themselves is going to be in the study. Need to be detailed with the parent about what exactly is going to happen. If treatment is theraputic, only consent from parent is required. If treatment not theraputic, get child's ASSENT - yes or no |
|
|
What can child understand, and when?
|
< 6, no research concept
< 12, no concept of confidentiality |
|
|
Is it ethical to study children?
|
If therapeutic, little debate. But need to make sure to not instill false hope into child, ie, cancer treatment
If not therapeutic, Big debate: |
|
|
Paul Ramsey
|
Studying children is unethical b/c children can NEVER fully grasp purpose of research, and parents can never fully know what kid wants
|
|
|
Richard McCormick
|
Studying children is ethical because it gives child a rare chance to help others
|
|
|
Special Ethical Considerations:
Animals |
Before 1963, no standards for animal care (housing, feed)
|
|
|
1963: American Association for Accreditation of Lab Animal Care (AAALAC)
|
came up with recommendations
|
|
|
1966: Animcal Welfare Act
|
charged USDA with inspecting animal research sites, not specific labs. So when come to WSU – just look at school in general (i.e., enough food, housing, etc.) Looked at whether there was sufficient facilities, but not specific – like are they feeding the rats. So – if had 400 rats, then should order xxx lbs of food – if this condition is met then the rats must be being fed
|
|
|
1971: The animal welfare act revised
|
• 1) AAALAC-certified each lab
• Or 2) create an animal care committee – the act did not say who should be on this committee |
|
|
1979: Animal welfare act revised again
|
now each facility must have an animal care committee required. The committee must have at least 5 people - one veterinarian and at least 4 others who have to know something about animal research
|
|
|
1986: revised again – the animal care committee must:
|
• Institutional Animal Care and Use committee (IACUC - == to IRB)
• At least one veterinarian that is trained on every animal that you study • At least one active animal researcher – the propriety of the procedures – is surgery being done correctly, etc. • At least one nonscientist - does this question need to be studied on animals; are there other ways to do it. • At least one community member – no connection to the facility. Often a member of the clergy or the medical community. Represents the larger interests of the general community. I.e, - this is too extreme • Plus one more – could be anyone, but it would make sense to have someone who is competent and is value-add (i.e., don’t want a rubber stamp) • IACUC works with researchers on things like things like how fed, how housed, how disposed of, sleep/wake cycles, etc. • WSU’s code is 125 pages |
|
|
William Summerlin
|
• Interested in how to reduce rejection of organ transplants.
• His idea – place organ in tissue culture before grafting. I.e., soak the organ in the person’s own tissue so it would soak up their own • Experiment – graft human corneas into rabbits. Soak human corneas in the rabbit’s tissue. It worked well in his labs, but no one else could replicate • So – got another idea – graft skin from black mice onto white mice. And it worked well – until he was found to be a fraud – it was just black marker • People went back and found that the rabbits cornea transplants failed 100% • Clear ethical violation – false reporting of data • Penalty – given permanent medical leave |
|
|
Ranjit Chandra
|
• Can vitamins improve brain functioning in the elderly?
• His research said – yes. Specific combination of vitamins and mineral can do it. • British Medical Journal – no way, Impossible • Nutrition – said – yes, we’ll publish. Sparked whole new line of research • Other researchers didn’t believe, so asked for data – he refused. Retired instead, formed private company. Claimed data is not privileged. Note – data from university is public record. • Nutrition – retracted the article. • Ethical Violation? Should he have turned over the data? |
|
|
Harry Harlow
|
• How does depression develop? He worked with monkies – how do you make them depressed? Let young monkey’s be with mother for period of time. Then put infant monkey in the “pit of despair” (sleek steel cage – can’t climb) – so the little monkies in the pit for 10 weeks. After 1-2 weeks, monkeys would huddle on the floor in fetal position.
• When took monkies out – they displayed clear psychotic tendencies – usually did not recover. Harlow bummed – they didn’t develop depression, they developed psychosis • So he concluded that suggests that temperament no defense against development of depression – • Massive amount of critizism against him. Why was this research allowed to go forward? • Ethical? Unsurprising results? Some people said that the findings were worth the emotional pain to the monkeys. |
|
|
How does bias in a sample affect the sample’s mean and variance?
|
A bias has no affect on the mean, but does affect the variance and makes it unrepresentative of the population. For this reason, we work with unbiased variance values.
|
|
|
Identify two benefits and two drawbacks to using simple observation as a data collection technique.
|
Benefits: Unobtrusive, simple, direct
Drawbacks: Can be ethically questionable, can produce ambiguous data |
|
|
Explain the difference between anonymous and confidential data. Under what conditions do you need to inform subjects which type of data you are using? (2)
|
Data are anonymous when no identifying information is collected.
Data are confidential when identifying information is collected, but is known only to the experimenters. You must always inform subjects as to which type of data you are collecting. |
|
|
What is the responsibility of an Institutional Review Board (IRB)? What is the IRB’s relationship to the federal government? Under what conditions do they not need to review a study? (3)
|
An IRB reviews studies to make sure the procedures do not pose undue risk to subjects.
It represents the federal government. There is no condition under which they do not need to be made aware of a study. |
|
|
Explain why dichotomizing continuous data compromises construct validity.
|
You end up treating all members of the dichotomized group as if they are exactly the same on the measured variable, when in fact they are not.
|
|
|
Identify two benefits and two drawbacks to using self reports as a data collection technique. (4)
|
Benefits: Can study phenomena that cannot ethically be observed
Can study cognitive processes Drawbacks: May be biased or distorted Subjects may not be able to clearly explain why something occurred |
|
|
Identify all of the ways in which use of a flawed or inappropriate measuring instrument compromises validity. (3)
|
Unreliability impacts statistical validity
Use of an inappropriate instrument compromises internal validity Continual reliance on the same instrument compromises construct validity |
|
|
. Identify two arguments against using college students as study subjects, and two arguments in support of their use. (4)
|
For: Homogeneous, helpful, convenient, inexpensive, engaged in society
Against: Homogeneous, still developing, lack real-world experience |
|
|
A food researcher is studying the impact of food color on perceived tastiness. She dyes fresh cottage cheese a number of different colors and tests to see if a particular color leads people to say that the cottage cheese tastes especially good or bad. In pilot tests people said that pink-dyed cottage cheese looked spoiled. When she runs the study proper, she cautions those in the dyed-pink condition that the cottage cheese will have an unpleasant appearance. What type of validity is threatened by this design? Explain your reasoning. (2)
|
This is experimenter expectancy, so construct validity is threatened.
|
|
|
. Explain quota sampling
|
We assemble a sample that mimics the population on some key factor. The sample is usually built, rather than assembled at random.
|
|
|
Identify all of the threats to construct validity that are beyond the experimenter’s control.
|
Hypothesis guessing, evaluation apprehension, pretest sensitization, construct generalizability
|
|
|
Briefly explain the informed consent process that is used when involving minors (those under age 18) in a research study.
|
Informed consent must be first solicited from the child's parent or guardian. The researcher must then obtain assent (a yes or no confirmation) from the child, and must also tell the child that his or her parent said yes.
|
|
|
Identify all the threats to statistical validity
|
Low power, violated assumptions, familywise error, unreliable measures, unreliable implementation of intervention, heterogeneous subjects, random factors
|
|
|
Karen Ruggerio
|
Harvard, late 1990’s
How does status relate to perception of failure? High-status people blame failure on discrimination Low-status people blame failure on themselves Graduate student could not replicate, asked to see her data Admitted results were fake Retracted 4 papers But electronic copies are not marked as such! Banned from applying for grants or serving on government advisory committees for 5 years |
|
|
Eric Poehlman
|
University of Vermont/University of Montreal, late 1990’s
How to help women deal with menopause Hormone replacement therapy Falsified up to 17 data sets Did 5 years in prison. Research career over |
|
|
Estimates of published falsified data (Public Library of Science, 2009)
|
Outright fakes—2%
Seriously altered—14% Slightly altered—33% |
|
|
HARKing
|
Hypothesizing After Results are Known 40%
|
|
|
Why?
|
Tenure pressure
Grant-getting pressure Grant-justification pressure Ego |
