Program Planning Evaluation

Front 1

When do you want to think about doing evaluations?

Back 1

Ideally during planning and implementation. Not just at the end.

Front 2

What is meant by the methods of evaluation? And what are some examples?

Back 2

Methods are indicating the way in which the data are collected as a part of the eval. Consists of surveys, interviews, or observations (can do one, all, or combination).

Front 3

What is the dependent variable?

Back 3

The outcome variable. This is the same thing specified in your outcome objectives. The end goal.

Front 4

What is reliability (consistency)?

Back 4

The extent to which the data are free of errors... So is the instrument, tool, measure, or test used to collect the data flawed? Is there a reason respondents may vary day to day? Think about how temporality, where, when, and how can affect reliability.

Front 5

What is validity?

Back 5

The degree to which the tool captures what it is intended to. Is it accurate?

Front 6

Evaluation Design

Back 6

The grand scheme that delineated when and from whom data are collected.

Front 7

Retrospective designs

Back 7

Gather data at the point of intervention backward in time.

Front 8

Prospective designs

Back 8

Gather data forward in time; begins at a point prior to start of the intervention

Front 9

Time Span of the Evaluation

Back 9

The length of time encompassed by the evaluation.. How close is the eval to the Ix?

Front 10

Time Span: Cross Sectional

Back 10

Data collected at one point in time. "Snap shot."

Front 11

Time Span: Longitudinal

Back 11

Data collected over time

Front 12

Time Span: Pre-test/post-test

Back 12

Before and after Ix

Front 13

Attrition

Back 13

The loss of participants over time. Can be due to death, moving, loss of interest, can't find them...can control some of those.

Front 14

Group who participates in Ix

Back 14

Experimental/Intervention/Exposed Group

Front 15

Group who didn't participate in Ix

Back 15

Control/Comparison/Unexposed group

Front 16

What does your eval design heavily depend on?

Back 16

YOUR QUESTIONS THAT YOU WANT ANSWERED

Front 17

What are some common eval questions?

Back 17

Does a particular program cause a particular change in knowledge/attitudes/behavior? What aspects are essential to change? What is most influential? Will this work across communities?

Front 18

Threats to Internal Validity: Instrumentation

Back 18

Could be a scale/BP cuff. Do you see changes in how something works. Also applies to people using the tool. Instruments and observer must be consistent.

Front 19

Threats to Internal Validity: Regression to the Mean

Back 19

Overtime, people on extremes will go backward towards the mean. Is a natural phenomenon.

Front 20

Threats to Internal Validity: Maturation

Back 20

Mental/cognitive/physical changes that naturally occur. So changes could be attributed to this. Also--how much time is lapsing?

Front 21

Threats to Internal Validity: Testing

Back 21

Just by asking the questions, you're opening the door to seeing them understand what you're researching. How and when you ask, as well as what you ask.

Front 22

Threats to Internal Validity: History

Back 22

Your observed program results can be bc of other interal or external events. Something other than your Ix causing your results.

Front 23

Threats to Internal Validity: Selection

Back 23

When you have a non-equivalent comparison group. How people got in Ix is explanation for your results. Be alert for this bias if working with a non-equivalent comparison group.

Front 24

Designs: One group, Post test only

Back 24

Data collected on the outcome from only participants, and only after completion of the program. No comparison group. Pros: Fast, easy, cheap. Cons: History and Maturation issues, less depth.

Front 25

Designs: One group, pre-test/post-test

Back 25

Evaluation collects outcome data only on participants, before & after the program. Pro: Only one group asked, baseline assessment, observations happen more than once so can track changes. Con: No comparison group. Maturation, History, Instrumentation, Regression to Mean, Testing threats. Not sure if changes are results of those, or the Ix.

Front 26

Designs: Comparison Group, Post-test only

Back 26

Collects data from participants as they complete the program, as well as nonparticipants from target audience. Pro: Can rule out history and maturation. Con: Groups are not randomly assigned.

Front 27

Designs: Ecological Design

Back 27

Considers differences between groups as opposed to differences between individuals; comparison happens only at a population level. Ecological Fallacy can exist. Con: Only group data.

Front 28

Designs: One group, Time series.

Back 28

Collecting data at multiple times across Ix time. 1 of 2 designs that can be used if the entire population is the unit of analysis. Pros: More data, can control for maturation, testing & regression to the mean. Helps identify for external factors.

Front 29

Designs: Multiple group, Time series.

Back 29

Data is collected from a targeted population and a comparison population at several time points before/after the program. Pros: Repeated measures, two groups, enhance external validity. Cons: Seasonal variation? Other variation?

Front 30

Designs: Two group, Retrospective (Case Control)

Back 30

Use historical, existing data to compare those with and without the outcome, based on whether they were exposed to the program.

Front 31

Designs: Patched-up Cycle

Back 31

Allows the post-program data from participants in an initial cycle to become a comparison group for the pre-program data for the next cycle of participants. Pros: History and maturation threats avoided.

Front 32

Designs: Two Group, Pre-test/Post-test

Back 32

Collects outcome data from participants and nonparticipants, both before and after the program. Participants and non-participants should be as similar as possible.

Front 33

Designs: Two group, Pre-test/Post-test, with random assignment (Randomized Trial)

Back 33

Highest quality level of evidence. Random assignment is the process of determining at random who receives the health program and who does not (maximizes comparability). Pros: Controls for confounders, avoids ambiguity in time--exposure always happens first.

Front 34

Considerations in choosing a design: Causality

Back 34

Is the Ix the causal factor?

Front 35

Considerations in choosing a design: Bias

Back 35

Are there biases?

Front 36

Considerations in choosing a design: Retrospective vs Prospective

Back 36

Do you want to look back in time or forward in time?

Front 37

Considerations in choosing a design: Time span

Back 37

Time allotted for pre/post tests

Front 38

Considerations in choosing a design: Groups

Back 38

Comparison & participant group should be as similar as possible.

Front 39

What is the importance of focusing your evaluation?

Back 39

Having a focused set of things you want to evaluate will keep costs down, make data manageable, and keep the program and evaluation focused of key outcomes. Realistically you can't evaluate everything.

Front 40

Three Characteristics of the "right" evaluation questions

Back 40

1. Can relevant data be collected (Can you get the info?). 2. Is more than one answer possible? Perceptions, attitudes, and beliefs--is it subjective? 3. Do stakeholders want the information this evaluation will produce?

Front 41

Sampling: Random Sample

Back 41

If each member of the population has a known chance of being chosen to participate in the program, then the sample is random.

Front 42

Sampling: Nonrandom sample

Back 42

Sampling does not rely on chance to determine who is in the sample

Front 43

Sampling, Hard to Reach Pop: Snowball Sampling

Back 43

When respondents refer you to other potential respondents

Front 44

Sampling, Hard to Reach Pop: Venue Based sampling

Back 44

Sampling through locations that cater to a specific type of person. All about place! E.g. hospitals, bars, liquor stores

Front 45

Sampling, Hard to Reach Pop: Respondent Driven Sampling

Back 45

Combines snowball sampling with a mathematical model that weights the sample to compensate for the fact that the sample was collected in a non-random way

Front 46

Sampling: Convenience Sample

Back 46

Invited everyone who is accessible to participate. Ie) asking your class

Front 47

Sampling: Purposive sample

Back 47

Participants are chosen based on a specifiv characteristic. E.g. race/ethnicity, disease status, age, enrollment in school district

Front 48

Sampling: Quota sample

Back 48

Participants are selected on a specific characteristic, but the number of participants is matched to their representation in the pop at large. E.g. Matching race to demographic representation in Buffalo

Front 49

Sample Size: Power

Back 49

The level of probability that a significant result may be found by the evaluation, if the study was in fact successful.

Front 50

Sample Size: Effect Size

Back 50

The degree of difference that is expected.

Front 51

Sample Size: Power Analysis

Back 51

The statistical process of analyzing the relationships among the power of a study, the type of statistical tests to be done, the number of variables, the effect size, and the sample size. Plug in numbers, effect size, and power you want to tell sample size. The bigger the sample size, the greater the ability to detect differences.

Front 52

Response Rate

Back 52

the percentage of individuals who were invited to participate and who actually participate

Front 53

Non-response bias

Back 53

Extent to which there are systematic differences between those who participate and those who don't.

Front 54

Incentives

Back 54

Money, gift certificates, etc. Giving cash, return postage, etc.

Front 55

Response Bias

Back 55

Intentional, unconscious way that an individual selects his/her/their response

Front 56

Social Desirability

Back 56

Telling you what you want to hear. This decreases with open ended questions, assuming its anonymous. Also can decrease it by asking the same question in different ways, or framing socially desirable options last.

Front 57

Primary Data

Back 57

Evaluator collects the data. E.g. Surveys, focus groups, in depth interviews

Front 58

Secondary Data

Back 58

Existing data is used. Vital records, medical records, national surveys.

Front 59

Established measures

Back 59

Take advtg of validated questions that exist. Also, validated measures and scales should be used--don't just create something new.

Front 60

Physical data

Back 60

Biological samples, measurements (height, weight)

Front 61

What do really good qualitative studies do?

Back 61

Provide data that changes how we think about a social process.

Front 62

What is the aim of great qualitative research?

Back 62

To understand an experience from the respondent's POV.

Front 63

Step 1 to Qualitative Research

Back 63

Learn as much as you can about the people, events, locations, stakeholders, and gatekeepers involved in the phenomenon you seek to study.

Front 64

Getting In: Join Relevant Organizations

Back 64

Sign up for on-line fourms, list serves, attend meetings. Eg. support groups, advisory boards

Front 65

Getting In: Volunteer

Back 65

Act as a volunteer at organizations relevant (even marginally) to your research interest. E.g. Homeless shelters

Front 66

Getting In: Shadow

Back 66

Contact individuals at organizations related to your interest and ask to shadow them. E.g. How clinics are currently using PrEP

Front 67

What do Key Informant Interviews do?

Back 67

Gives you and opportunity to ask people for their perspective on constructs relevant to your specific research interests. May need IRB approval if you're doing several, so they can be used as data.

Front 68

Qualitative Sampling: Convenience

Back 68

Participants or cases that are accessible and willing

Front 69

Qualitative Sampling: Critical Cases

Back 69

Exemplar cases or participants, cases important in some unique way.

Front 70

Qualitative Sampling: Deviant Cases

Back 70

Highly unusual participants or cases

Front 71

Qualitative Sampling: Maximum Variation

Back 71

Participants or cases with differing experiences. Don't want everyone to be the same, helps to get breadth

Front 72

Qualitative Sampling: Random Purposeful

Back 72

Participants or cases randomly selected from a large sample pool.

Front 73

Qualitative Sampling: Typical Cases

Back 73

Usual or normal cases.

Front 74

Qualitative Sampling: Theory based

Back 74

Participants or cases with the theoretical construct you're interested in.

Front 75

"Inconvenient Facts"

Back 75

Qualitative researchers should try to identify these.

Front 76

The Ethnographic Trial

Front 77

Inconvenience Sampling

Back 77

Are there people outside of the sample whose existence is likely to have impact for the argument I'm making?

Front 78

In Depth Interviews

Back 78

Can be open ended, structured (don't deviate from script), or semi-structured. Has individual participants. Data collected: Interviewer's field notes (nonverbal too), audio recorded interview, relevant notes

Front 79

Observation: Non-participant Observation

Back 79

Researcher acts as "fly on the wall"

Front 80

Observation: Participant Observation

Back 80

Researcher engages in activity with respondents

Front 81

Observation: To understand...

Back 81

How events unfold, how people act/make decisions in realistic settings, how people interact, how process change over time, who the key players are.

Front 82

Observation: Sample & Data

Back 82

Sample: Individuals, Settings, or Both

Data: Researcher's field notes

Front 83

Ethnography

Back 83

To learn and understand cultural phenomena which reflect the knowledge and system of meanings guiding the life of a cultural group. Combines interviews, observation, participant observation. Sample: individ, settins, or both. Data: Field Notes, Interview transcripts

Front 84

Focus Groups

Back 84

Assess group dynamics. Sampling. Led by a moderator. Data: Interview notes, audio recording, video recording

Front 85

General Strategies for Data Analysis

Back 85

Transcribe data; read and re-read; code (for reliability and consistency)' analysis of themes or patterns; iterative, evolving process; data saturation

Front 86

IRB and Qualitative Research

Back 86

If you're collecting data, need IRB approval. Informed consent may take various forms depending on the nature of the data collection.

Front 87

Nurturing lasting relationships

Back 87

Respondents, informants, and gatekeepers may be people you will repeatedly ask to participate. Think how you will nurture the relationships.... Hand written thank you notes, volunteering your own time, providing research updates