Health Program Evaluation

Front 1

Purpose of PE

Back 1

To provide information on the effectiveness of programs. In order to optimize:
- Outcomes
- Efficiency
- Quality of health care

Front 2

Primary Function of PE

Back 2

Provide data on the extent to which a program's objectives are achieved

Front 3

Process Evaluation

Back 3

Focus on program's activities

Front 4

Outcomes are difficult to measure because:

Back 4

- Lack of consensus on definition
- Not enough time
--> Usually focus on extent to which programs achieve more easily measured goals and objectives

Front 5

Program Impact

Back 5

- Scope of Effect
- Duration of outcomes
- Extent of influence

Front 6

PE consists of following activities

Back 6

- Posing questions
- Setting standards
- Designing the evaluation
- Selecting participants
- Collecting, managing, analyzing data
- Reporting results

Front 7

Setting Standards

Back 7

Deciding what information is necessary to provide convincing evidence of program's effectiveness
(Merits are equated with effectiveness)
- must be measurable and credible

Front 8

Evaluation Design

Back 8

The manner in which the environment is structured so that any documented effects seem to have resulted form the program can be linked to it

Front 9

Selecting Participants

Back 9

- Inclusion and exclusion
- Number of people

Front 10

Reliability

Back 10

Consistency of Data

Front 11

Validity

Back 11

Accuracy of data

Front 12

Evaluation Report

Back 12

Description of a program's characteristics and explanations/judgments of program's merits
(Created after data is collected; report is disseminated)

Front 13

Baseline Data

Back 13

Shows participant's conditions before the program begins
- Describes the characteristic of participants
- Used when interpreting effects of the program

Front 14

Interim Data

Back 14

Collected During course of program to show progress in meeting needs

Front 15

Formative evaluation

Back 15

Interim data collected at start of program, but before the conclusion to describe progress

Front 16

Process Evaluation

Back 16

Extent to which planned activities are implemented
- findings may be reported at any time
- Can show that proper protocol is not being taken for example
- Almost always useful

Front 17

Qualitative Evaluations

Back 17

Collect data through in person interviews, observations and written documents
- Personalized
- add emotion
- compliment existing data
- help define goals

Front 18

Participatory Evaluation

Back 18

Invites representatives of the community/organizations that will be affected by evaluation findings to join the team as partners
- Reduces health disparities!!
--Improves research using local knowledge/history
--Concerns are viewed ecologically
- Assists with resources

Front 19

Ecologically

Back 19

In political and Social context

Front 20

Evaluation Framework

Back 20

Provides guidance for evaluation helping them to be certain that their evaluations overall design and consider the origin of contests of the programs they examine

Front 21

PRECEDE-PROCEED Framework

Back 21

Assessment of:
-Implementation/evaluation of process impact/outcome
- Social
- Administrative/policy
- Behavioral/environmental
- Educational/ecological
- Epidemiological

Front 22

Social Assessment

Back 22

Determine perceptions of people's needs and quality of life

Front 23

Epidemiological Assessment

Back 23

Identify health programs that are most important

Front 24

Behavioral/environmental

Back 24

Uncover factors that might contribute to health problems

Front 25

Educational/Ecological

Back 25

Identify factors that might foster change in health behavior

Front 26

Administrative/policy

Back 26

Review policies/resources that facilitate or hinder implementations

Front 27

Implementation/evaluation of process impacts and outcomes

Back 27

Asses programs activities and outcomes

Front 28

RE-AIM

Back 28

Reach, Efficacy, Adoption, Implementation, Maintenance
== Provides a score that can be applied to program/evaluation

Front 29

Reach

Back 29

Percentage of potential participants exposed

Front 30

Efficacy

Back 30

Intended effects and possible unintended outcomes

Front 31

Adoption

Back 31

Participation rate of eligible subjects and how well the setting/people who deliver the intervention as intended

Front 32

Maintenance

Back 32

Long term effects and extent to which it is continued

Front 33

Purpose of Framework

Back 33

- Provides guidance!!!
- Models predict behaviors/outcomes!!!!!!!!

Front 34

Institutional Review Board

Back 34

Review the design of evaluation to guarantee that structure will protect rights and privacy

Front 35

Belmont Report

Back 35

Ethical Principles for evaluation research. Includes:
- Respect for persons
- Beneficence- treatment in ethical manner
- Justice- those who reap benefits vs. the burden are balanced

Front 36

Quality Improvement evaluations

Back 36

Improve deficiencies in health care quality

Front 37

Informed Consent

Back 37

All subjects are knowledgeable about risks/benefits of participation and activities

Front 38

HIPAA

Back 38

Safeguards health information of individual obtaining health care

Front 39

Evaluation Questions

Back 39

Guide evaluators in gathering/analyzing data on characteristics and merits of program

Front 40

Objective

Back 40

Refers to specific goals of a program

Front 41

Substantive

Back 41

Years of life saved

Front 42

Cost Benefit analysis

Back 42

Relationship between cost and monetary outcomes

Front 43

Cost effectiveness analysis

Back 43

Relationship between cost and substantive outcomes

Front 44

Sensitivity analysis

Back 44

Measuring program by increasing funding incrementally to test sensitivity of effectiveness to changes in funding levels

Front 45

Measure effectiveness in terms of:

Back 45

Program's structure, process or outcomes of health care

Front 46

Structure of care

Back 46

environment, setting and organization of care

Front 47

Process of care

Back 47

What is done to and for the patients (procedures, tests, prevention, treatment, etc)

Front 48

Outcomes of Care

Back 48

The results for the patient
- Morbidity and mortality

Front 49

Questions asked when comparing two groups:

Back 49

- Are the groups comparable?
- Is the magnitude of difference meaningful?

Front 50

Pilot Study

Back 50

Small Scale Study to get estimates of effect sizes
- When you cannot find prior research

Front 51

Expert

Back 51

Professional, consumer or representative likely to use results of an evaluation

Front 52

Guidelines for Expert Panel

Back 52

- Must specify evaluation questions
- Provide data to assist them
- Select experts based on knowledge/influence and how they will use findings

Front 53

Published Data Sets

Back 53

Provide benchmarks to measure effectiveness of new programs
- Must be TRULY applicable

Front 54

Cost Effectiveness Standards

Back 54

A is effective and a lower cost program

Front 55

Cost Benefit

Back 55

A has MERIT if it benefits or equal to or exceeds costs

Front 56

Cost minimization

Back 56

A and B have equal benefits but A is lower cost

Front 57

Cost Utility

Back 57

A produces "x" quality of adjusted life years are lower cost than B

Front 58

Impact Evaluation

Back 58

Examines proximal outcomes (relationship between randomized intervention and effectiveness end points)

Front 59

Explanatory/predictor Variables

Back 59

Indpendent variables that are present before start of program and are used to explain/predict outcomes

Front 60

Outcome Variables

Back 60

Dependent variables that are factors that evaluators expect to measure
- Ex. health status, knowledge, etc.

Front 61

Patient Mix

Back 61

Characteristics of patients that might affect outcomes
- Ex. SES, functional status scores, chronic disorders

Front 62

Steps/Questions of ED

Back 62

1. EQ and ES
2. Independent variables
3. Exclusion/inclusion criteria
4. Control group vs. no control group
5. When will measures be taken
6. How often will measures be taken

Front 63

Minimum Design

Back 63

Uses standards as the guide

Front 64

Eligibility Criteria

Back 64

Foundation for evaluators inference of conclusions about the groups who are most likely to benefit from participation
---> comes from evaluation questions

Front 65

Exclusion Criteria

Back 65

Exclude potential participants whose inclusion is likely to impair functioning of the evaluation or skew its data

Front 66

Group Requirement

Back 66

Control group must start out demonstratively like the experimental in composition but definably and measurably unlike the other group

Front 67

Pre-measures

Back 67

help select groups to participate, check the need for program and ensure comparability of groups

Front 68

Prospective Investigation

Back 68

Data is collected for the specific purpose of the evaluation commencing at the start of the evaluation
- Provide the most control

Front 69

Summative evaluation

Back 69

Retrospective studies, historical analysis
- Leaves too much to chance

Front 70

Randomized controlled trials/true experiments

Back 70

Evaluations with concurrent controls in which participants are randomly ASSIGNED to groups

Front 71

Nonrandomized controlled trials

Back 71

Quasi Experiments
- Evaluations with concurrent controls which participants are NOT randomly assigned to groups

Front 72

Before and after designs

Back 72

Evaluations with self controls
- Require pre and post measures
- Group participation serves as its own comparison
--> biases such as motivation and historical events
--> depends on number and timing od measurements
--> can be ineffective if data is collected before wanted outcome

Front 73

Membership Bias

Back 73

Bias found in preexisting groups because one or more of the same characteristics that cause people to belong to the group are associated with the outcome evaluated

Front 74

Non-response bias

Back 74

participants in evaluation study is voluntary
- Those who accept invitation may be somewhat different from nonaccepters in willingness to try new programs

Front 75

Descriptive/observational study

Back 75

lack interventions
- use surveys to guide them in developing in programs and setting standards for merti

Front 76

Cross Sectional Design

Back 76

Colelct baseline information o experimental and control groups to guide program development and as source of data regarding program/envionrmetn
- Portrait of one group at ONE period of time

Front 77

Cohort

Back 77

Group with something in common that is followed over period of time
- prospective study and longitudinal
- Determine the extent of program's effects that have lasted

Front 78

Factorial Design

Back 78

-Experimental or descriptive
- Descriptive- lays out all combination of several independent variables at one time
- Experimental- three or more factors are independent variables are possible

Front 79

Main Effect

Back 79

Effect of each factor on the dependent variable

Front 80

Interaction Effects

Back 80

Effects on the dependent variable that occur as a result of the interaction between two or more independent variables

Front 81

Internal Validity

Back 81

Program is effective in specific experimental instance
- evaluator can tell if a finding is due to program or some other factor/ bias as a result
- Risks: maturation, instrumentation, history and attrition

Front 82

External Validity

Back 82

Evaluator can demonstrate that the program's results are applicable to participants in other places/at other times

Front 83

Maturation

Back 83

People may mature intellectually/emotionally and this new maturity may be just as important to producing change

Front 84

Instrumentation

Back 84

Unless the measures used to collect data are dependable, the evaluator cannot be sure that the findings are accurate

Front 85

Attrition

Back 85

Participants who remain in the evaluation sample are different from those who drop out

Front 86

Hawthorne Effect

Back 86

In experimental evaluation participants behave in atypical way because they know they are in a special program