Program Eval Unit 1 (Objectives 11-21)

Front 1

Objective 11: Unit 1: Compare and contrast the terms in each of the following sets 1) Conceptual Definitions vs. Operation Definitions.

Back 1

Conceptual definitions: Defined in a qualitative way intended to communicate interpretation and meaning.
o Qualitative Descriptions and concepts
o There are often problems in taking a conceptual definition or a concept and measuring it….such as trying to measure creativity, problem solving skills, imagination etc.

o “If you can’t measure what you want to do, then do what you can measure”


Operation definitions: Defined in terms of how it is measured.

Front 2

Objective 11: Unit 1: Compare and contrast the terms in each of the following sets: 2) Research Grant vs. Research Contract

Back 2

Research Grant- The investigator or researcher develops the idea and proposes it to the funding agency along with why the idea is significant and how the study will be implemented.

Research Contract: The funding agency specifies what they want done, and prospective contractors make the case for how well they could execute the project.
o You apply to win this contract to do this work.
o A lot of times governments are forced to go with the lowest bidder.

Front 3

Objective 11: Unit 1: Compare and contrast the terms in each of the following sets: 3) Research vs. Evaluation.

Back 3

Research:

Intended to develop theory or generate new knowledge.

Evaluation:

More applied and intended to make judgments about the worth of a program which is summative evaluation or you are trying to monitor programs during initial development and use it for the purpose of improving the program; which is formative evaluation.

Front 4

Objective 11: Unit 1: Compare and contrast the terms in each of the following sets: 4) Limitations vs. Delimitations.

Back 4

Limitations:

Factors that threaten the internal validity of the study (i.e. threats to internal validity). These factors are not created by the researcher, but she or he attempts to identify and control for them.

o Basically all of the things that can threaten the confidence in your conclusions.
oPrimarily affects internal validity.


Delimitations:

“narrowing the scope” Ways in which the investigator intentionally narrows the scope of his or her research with respect who and what is being studied as well as how is it being studied.

o You as the researcher has the control….and you would narrow the scope of the study.

o I.e. only trying to study female or males, those living in suburban areas.

o Effects external validity because results may not generalize without collecting data from the groups that you chose not to study by narrowing your scope of the study.

• Sometimes a factor can be both a limitation and delimitation.

Front 5

Design Validity: Epitomized by the following questions:

Back 5

1) Statistical conclusion-Is there validity to the statistical conclusion that A and B related?

2) Internal Validity-If A and B are related, then how plausible is it that A causes B.
•Limitations are the factors that threaten internal validity and may also threaten statistical conclusion validity.

3) External Validity- If A and B are related, if it is plausible that A causes B, to what extent can you generalize that causal relationship to other people, places and times.
•Delimitations are the things that have an effect on external validity.

4) Construct validity- (to be defined) but Construct validity refers to the degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based.

These are in hierarchical order. So, basically if you haven’t established statistical conclusion validity then you can have internal validity.

So, when you “delimit” the scope of your study….can the results still generalize?

Front 6

Objective 12: Unit 1:Describe 5 factors that can create bias in a research study and 3 things that can be done to avoid or minimize bias.

Back 6

1) Conflicting goals between the researcher and the funding agency. (i.e. funding agency may have a particular goal but the researcher may not feel comfortable).

2) Conflict of interest regarding publication, future job security, tenure, and financial gain (e.g. you own the program you’re evaluating).

Pharmaceutical industry…they have a strong financial interest in the outcomes in drug development.

3) Ways data are collected.
4) Ways data are analyzed.
5) Ways data are reported.

Front 7

Objective 12: Unit 1: Describe 5 factors that can create bias in a research study and 3 things that can be done to avoid or minimize bias.

Back 7

3 Things that can be done to avoid or minimize bias:

1. Subject research plan to scrutiny and comments of a broad spectrum of interested audiences.

2. Seek expert opinion or technical aspects of the methods (e.g. sampling, analysis, etc).

3. Pilot test measurement protocols and have them review by qualified personnel.

4. Submit final reports for peer review by qualified personnel with no conflicts of interest.

Front 8

Objective 13: Unit 1: Explain the limitations of evaluative data for informing decisions about programs and policies and give 3 examples of other factors that may influence such decisions.

Back 8

Limitations:

o Measurements are often problematic.

• For example, if you are interested in adolescents sexual behavior or substance use….then how are you going to measure it? Usually self-report but not always accurate and therefore difficult to validate.

• There are also biological and other markers that can be used as through a blood or urine sample but collecting those data are often difficult and costly and you wont necessarily have their consent for that in research.

o Disagreement about what constitutes compelling evidence.
• Often routed in politics, religion, ideology
• How do I know this program worked?
• If you have a program and it doesn’t changing behavior, but the people made an informed decision about not changing their behavior. Did it work?

o Decisions about programs and policies are often based on factors such as:
• 1. Who knows/likes who
• 2. Who owes who a favor, or might want a favor
• 3. Political and practical factors

Front 9

Objective 14: Unit 1: Outline the information that would be included when describing a methods section of an evaluation study and provide 3 reasons why this information is important.

Back 9

Procedures:
o Design
o Setting and Subjects
o Inclusion/Exclusion Criteria
o Sampling
o Intervention methods (if applicable)
o Measurements
o Data Collection Methods
o Data Analysis Methods

Front 10

Objective 14: Unit 1: Outline the information that would be included when describing methods section of an evaluation study and provide 3 reasons why this information is important.

Back 10

3 Reasons why this information is important:

1) To evaluate the quality of the and, therefore, determine the confidence that can be placed in the conclusions.

• Randomized control trials-thought of as the highest type of study design
• The quality of the study is all about the methods.

2) To enable others to replicate the study.
• Without specific details about the methods it will be very difficulty to replicate a study.

3) To contribute to development of new methods.

Front 11

Objective 15. Explain the concept of statistical conclusion validity and list at least 3 threats to this kind of design validity.

Back 11

Statistical conclusion validity:

Concerned with determining whether 2 variables are related. There are several threats to this type of validity that may result in a conclusion that 2 variables are related when in reality they are not (this type is a Type 1 error and you rejected the null in hypothesis testing)

OR

that 2 variables are not related, when in reality they are (this is a Type II error and you retained the null) in hypothesis testing).

Front 12

Objective 15. Explain the concept of statistical conclusion validity and list at least 3 threats to this kind of design validity. (see notes!!)

Back 12

Some examples of threats are:

o Low Statistical Power- “It means that there are some things there and you can’t see them”.

Statistical power is your ability to reject a false null hypothesis. It’s the ability to see differences when they are really there. That’s, how much statistical power do you have? Statistical power is largely affected by the size of the sample, the size of the relationship (magnitude or strength of the relationship), the reliability of the measurements of the data. So, why is that? It gets back to this idea about… It is the same as saying…..Is there a difference between a group that gets a program and a group that does not get a program. If the program works, then there is going to be a big difference between the group that gets the program and the group that does not get the program. The way that you say how likely is that due to chance or not is by comparing the difference between the groups relative to the difference within the groups. So, basically you have the variance between/variance within (formula); and to the extent that the quotient here gets bigger, the likelihood of it not being due to chance increases in other words the likelihood of it being statistically significant increases.

o Violated Assumptions of Statistical Tests-

There are all types of assumptions for tests but if you violate those assumptions there are ways that you can adjust or correct for them, e.g. interclass or inter-cluster correlation adjustment.

o Fishing and the Error Rate Problem

-If you continue randomly generated data then find some that are significantly significant then, how many can you expect to find by chance.

Front 13

Objective 16. Unit 1: Distinguish between internal and external validity of research design.

Back 13

Internal Validity is concerned with the extent to which 2 variables are causally related.

While:


External Validity is concerned with the extent to which causal relationships can be generalized to other people, places and times.

Front 14

Objective 17. Unit 1: Identify and define seven threats to internal validity of research design. (focus on first 7-SEE NOTES FOR THOROUGH DESCRIPTION AND EXAMPLES)

Back 14

1) History-historical events and not your program that really influences the outcome.

2) Selection bias- Is the way that you selected people to participate in your study and were differences in the selection criteria between those who got your program and those did not had an effect on the outcome.

3) Maturation-internal processes within the individuals you are studying. Maturation can only occur on studies that you are collecting data more than once; and sees whether maturational affects really influence the outcome.
• People become stronger or fatigued, older, more skilled

4) Testing-(Reactive effects of testing)-it wasn’t your program and was actually the testing protocol that had an influence on the outcome.

• E.g. Measuring Testing dietary intake and had them keeping a log, and then try to educate them. When in fact what had an effect on the outcome was because they were keeping a log and they self-monitored themselves and the fact that you tried to educate them.

5) Instrumentation-how you measure things and that’s what influenced the outcome and revealed false results.

6) Statistical regression- when you select people on the basis of extreme scores their scores their scores will regress toward the mean on the second measurement.

7) Attrition-Dropout had a big affect on your study outcome. (i.e. started off with 100 people but 70 dropped out). But the 30 that were left were really successfully, so then how successful was the program.

Front 15

Objective: 18: Unit 1: Identify and define four threats to external validity of research design.

Back 15

External validity is epitomized by the question…“If there is a relationship between 2 variables and it is a causal relationship, to what extent can it be generalized to other people, places and times?

Threats include:
1) Interaction of selection and treatment.
• This can occur when selected treatment or comparison groups are more or less sensitive to the treatment prior to initiating the treatment (or intervention)

2) Interaction of history and treatment.

• (...to everything there is a time...)
Not only should researchers be cautious about generalizing to other population, caution should be taken to generalize to a different time period. As time passes, the conditions under which treatments work change.

3) Interaction of setting and treatment.

• This can happen when the population from which the comparison group samples were selected is not the same as the target population.

3) Interaction of testing and treatment.

• The occurs whenever the pretest sensitizes the treatment group to the effects of the treatment.

Front 16

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:

Halo effect:

Back 16

Halo effect:

Positive past experiences result in favorable future evaluations.

Front 17

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:

Hawthorne Effect:

Back 17

Hawthorne effect:

Participation in a study or novelty causes an effect and that has an effect on people’s performance.

Front 18

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:

Experimenter Bias:

Back 18

Experimenter Bias:

Preconceptions or other characteristics of the experimenter cause an effect.

Front 19

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:

John Henry Effect:

Back 19

John Henry Effect:

When people in a control group learn thy are being deprived they sometimes reach with compensatory rivalry.

Front 20

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:


Placebo effect:

Back 20

Placebo effect:

Expectations of participants cause an effect.

Front 21

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:

Misplaced Precision:

Back 21

Misplaced Precision: Exercising an over amount of precision in some aspects of the research design while not paying sufficient attention to other aspects.

Front 22

Objective 19: Unit 1: Recognize the following sources of error when given definitions or example:

The law of the instrument:

Back 22

The law of the instrument:

When an instrument becomes so popular that it is used whether or not it is appropriate.

Front 23

Objective 20: Unit 1: Outline the major steps to be achieved in conducting a research or evaluation study

Back 23

1) Formulate the question-often one of the most challenging parts especially if you don’t have enough creativity or know the literature enough.

o So What? Who cares?

2) Demonstrate the significance of the question: (what do we already know? What are the gaps in what already know and how is it connected to the longer term goal?)

Synthesize current knowledge.
o Describe how the proposed study will fill the gaps.
o Describe the relationship between the specific aims and longer term objectives.

3) Design: and that’s basically just the structure of the design.

• From whom and when the data will be collected?
• Then you have the sampling size and the sampling plan.
• You have the data collection plan and who is going to collect what data from whom, when and how.
• And then there are the intervention methods, if its an intervention study. And the data analysis plan.

Front 24

Objective 21: Unit 1 Identify and describe 5 functions of research and give one example of each.

Back 24

1) Exploration
2) Description
3) Prediction
4) Hypothesis Testing
5) Causal Exploration

Front 25

Objective 21: Unit 1 Identify and describe 5 functions of research and give one example of each.

Back 25

1. Exploration-Exploratory research is used to improve understanding of a topic about which very little is known.

o What is the cholesterol level that most doctors think drug treatment should be initiated?

2) Description-Describe a phenomenon in depth. “Just trying to tell the story or describe the phenomenon. This may be in terms of quantitative methods (e.g. frequencies and percentages, means and standard deviations) or qualitative methods( e.g. ethnography, narrative, case study, photo essay).

o What are the main barriers to accessing health care in a defined population?


3) Prediction-Predict relationships between variables, or what will happen to one or more variables in the future?

• What is the relationship between exposure to a cell phone and incidence of brain cancer?

• Will aggressive behaviors among adolescents in high school increase in the future?

4. Hypothesis testing-To test if two variables are or are not related statistically. Its all about statistical association.

o There is no difference in rates of colon cancer screening between participants who receive tailored telephone counseling vs. those who receive print communications.

o There is no relationship between knowledge about HIV transmission and self reported sexual behaviors.

5. Causal explanation-To determine the causal mechanisms that explain the relationships between two or more variables (i.e. why does A cause B?)

o Why does cigarette smoking cause lung cancer?
o Why does expressive writing reduce anxiety and depression?