Research 2 Sw

Front 1

What is a program eval?
What areas would we reseach –

Back 1

Efficiency, morale, client goals, attrition, client satisfaction, cultural competence etc

Front 2

Parsimony

Back 2

Preference for the simpler of available solutions as an explanation

Front 3

Scientific Skepticism

Back 3

Point of view that all claims should be questioned re: validity until substantiated using credible scientific data

Front 4

Positivism

Back 4

The belief that valid knowledge about the objective world can be arrived at through scientific research

Front 5

Empiricism

Back 5

Preference for evidence collected systematically through observation or experiment that has been verified (through replication) using satisfactory standards for evidence

Front 6

Qualitative evaluation: types

Back 6

Case studies, narrative analysis, grounded theory(?) etc
Qualitative Evaluation “AKA” Qual. Research and flexible method research

Front 7

Qualitative evaluation studies - key components:

Back 7

1) focus on naturalistic inquiry, in situ
2) rely on researcher as the main instrument of data collection
3) reports emphasize narrative rather than numbers

Front 8

**Qualitative and Quantitative are both

Back 8

Empirical (systematic observation) and Systematic (adheres to standards),

Front 9

characteristics 1 only ea
Empirical

Systematic

Back 9

(systematic observation) and
(adheres to standards),

Front 10

QUALITATIVE: associated with what kind of evaluations:

Back 10

PROCESS evaluation, ie : “How and Why” questions about the programs, and
FORMATIVE evaluations, when one is evaluating the program effectiveness PRIOR to its beginning.

Front 11

QUANTITATIVE: associated with what kind of evaluations

Back 11

OUTCOME evaluation – looks at the measurements – quantifiable

Front 12

The precautionary principle:

Back 12

says that if there are urgent (but unquantifiable) threats, one must forego waiting for evidence based backing (and I assume, take action to prevent threat) as such, this is another hurdle that EBP faces.

Front 13

QUALITY CONTROL - measures one can take:

Back 13

Triangulation of data – find research that supports the claims/ results etc
Peer Debriefing – peers help limit bias
Member checking: go back to clients to review/confirm findings (also aids clt engagement)
Negative Case analysis: researcher is obligated to play devils advocate – find results that reject the findings and explore, may help to understand an issue, requires integrity.
Audit Trail – so someone can follow your steps in evaluation and analysis.

Front 14

PROS AND CONS OF QUALITATIVE

Back 14

BENEFITS:
In-depth questions, probe more deeply bc not structured/predefined
IE: you can ask staff or clients complex questions about the program, staff/clients feel more invested in the program bc they are being asked their opinion.
CONS:
Sacrifice Breadth for depth – not effective for wide sample of participants
time consuming and expensive
Not great for reporting precise/ exact outcomes (quantitative designs w/ control for error, stat tests for degree of confidence etc) are more useful in these cases.
-----
Nutshell, Quant rests to see if a programs worked better than the other (tests offer precision) , while Qual provides factors that may explain why it did or didn’t work. One can make comparisons to other programs (though primarily done with quantitative), but they lack control mechanisms

Front 15

single system designs: ??? come from a quantitative research design –

Back 15

refers to a Single Client System (of analysis) ie: individual client, the couple, or the family

Front 16

Steps fr qualitative evaluataion:

Back 16

Determine that Qual. is the appropriate method
Decide on units of analysis (individual, staff, or program, unit etc)
Sampling strategy
Type of data collection and analysis
Strategies for rigor (sim to post test)

Front 17

Units of Analysis

Back 17

Can be an individual client, or couple, or can be hospital unit, group home, program, etc – these are ethnographic nturaliastic settings thus good to study

Front 18

Gatekeepers

Back 18

You will need the OK/permission of the gatekeepers, people who have access to make the study happen

Front 19

Key Informants

Back 19

Are very knowledgeable about a number of areas in the agency & willing to share with you.

Front 20

Reciprocity, payback, feedback

Back 20

Ways in which you can compensate the agency for their participation (cash, catered meal, therapy at the end of research

Front 21

Types of sampling:

Back 21

Deviant VS Typical–
Maximum variation
Snowball
Convenience sampling –

Front 22

Deviant VS Typical–

Back 22

interested in knowing about the outliers – not the sample that most typically matches the population (which is what typical sampling is interested in)

Front 23

Maximum variation

Back 23

– when you need a sample that offers the wides representation of the population (ir all races and ages etc)

Front 24

Snowball

Back 24

(esp good for immigrant – you tell key persona, they pass along the question etc). Since it is organically growing it reaches the segment of the pop you want to target on the most organic way.

Front 25

Convenience sampling –

Back 25

when you just take sample from whoever is at hand

Front 26

Data collection for Qualitative:how/where do you get data

Back 26

filed notes transcripts ther docs

Front 27

DOCUMENTS ae good bc:

Back 27

bc they are already existing and were lielly written outside of interview context. are the least reactive to interviewer bias or distortion Examples of documents: memos, financial records, agency mission statement, guidelines etc.

Front 28

On site observation –

Back 28

ethnography, fieldwork style of data collection – unobtrusive observation

Front 29

Participant observers

Back 29

actively involved with those you are studying

Front 30

Focus groups – what, pros and cons

Back 30

group of peers (no diff in hierarchy) PRO: can be very informative as group discussion may bring up new material/issues, CON: but you don’t get the in depth confidential info that one-on-one interview provide.

Front 31

Reflexivity

Back 31

being aware of one’s own biases – checking “in” to self-monitor for biases (biases are one of the “cons” of qualitative eval.

Front 32

Interview guide-

Back 32

a compilation of key questions and areas that will be explored, and is made in advance of data collection and may be giving to the program in advance as well.

Front 33

Kinds of interview questions

Back 33

attitude / their opinion, feelins, thoughts ( what they know, behavioral - what would they do in X situation

Front 34

Kinds of interview questions

Back 34

attitude (opinion, feeling, thoughts, behavoir

Front 35

When do you have enough data?
group, indiv, and capacity

Back 35

Groups: at least 2 or more groups
Single Subject Design: at least 9 studies
at Saturation: when data patterns start to repeat or become redundant

Front 36

Mixed Methods
benitfit of doing so

Back 36

Qualitative and Quantitative methods used in combination. Benefit is the triangulation

Front 37

Temporal sequencing

Back 37

ex - focus groups (qual formative, followed by outcome srvey,

Front 38

Sequencing designs:

Back 38

most common mixed method: Qualitative used First , (or after) then quantitative survey or experiment.
Combo allows BREADTH and DEPTH!

Front 39

Empowerment evaluation & Participatory evaluation

Back 39

– researcher gets actively involved n advocating or t least supporting the group/community etc it is studying. As such, the research itself acts as a kind of grass roots empowerment “tool” also example of reciprocity

Front 40

Rapid evaluation and assessment methods: QUALI??) ASK ON TUES!*****

Back 40

use verbatim tanscript and id'd domains

Front 41

Methods of client satisfaction studies: SURVEY Pros and Cons...

Back 41

EX: ask about agency environment, goals identified, goals met, etc
--PRO’s : inexpensive and easy to administer
--CON’s: it is only the client’s (or unit of analysis’s) opinion –
--Survey doesn’t provide accurate / reliable measurement of improvement etc (quantitative research would)

Front 42

characteristics of qual survey for client satisfaction
low and high ratings due to

Back 42

mandated not satisfied, drop outs noyt included, unrealistically high - due social desireability, longterm txmnt, etc like other thigs
cant test practicl effect /benefit with qual.

Front 43

Recommended methods of client satisfaction studies:

Back 43

1-- the instrument should be reliable measurement scale, proven through previous research
2-- make sure survey captures Psychometric data (the measurement of mental traits, abilities, and processes.)
3-- establish a baseline response and test several more times over the course of the txmnt. Useful to see to see if there is a trend in increase or decreasing satisfaction (or no change)
4-- Add a couple open ended questions (“what did you like most/ least? What would you change/keep?about the program?”) to get more insight into issues that you may not be aware of.
5--Don’t / can’t generalize survey results to a broad population.
6--Check behavioral cues that might influence results or point to inaccuracies: low attendance rate, but high satisfaction is incongruous. – very low attendance of
7-- IDEAL GOAL IS TO ACHEILVE 90% SATISFACTION.

Front 44

What is an evaluation proposal?

Back 44

A written outline of well thought out, plan for research. Includes: author’s knowledge about subject, the intent of the study, areas of interest for this study/exploration, budget, timelines. Helps to guide the actual proposal

Front 45

Elements of an Evaluation proposal and report

Back 45

Executive Summary
Longer than an abstract, briefly describes the report and findings (two-four pgs)
Introduction (PPP)
Identify the problem
The program that will be evaluated, and questions
The purpose – establish need for the study
Lit Review
Theoretical context / history
Survey of recent/relevant literature
** funnel approach, identify lit examples of broad applications of txmnt, bring to a narrow focus - to specific issue
Methodology
Evaluation design and data collection method
Sampling design
Subject description
Instrument description
Procedure for analyzing the data
Results
Factual figures and tables
Statistical significance
Discussion
Explain findings
How it is applicable to agency or program or client
Limitations of this study
References
Appendices

Front 46

name the parts of eval study

Back 46

Executive Summary
Introduction (PPP)
Lit Review
Methodology
Evaluation design and data collection method
Sampling design
Subject description
Instrument description
Procedure for analyzing the data
Results
Discussion

References
Appendices

Front 47

Types of lit reviews – 3 types describe and what kind of data do they provide (Qual/Quan?)

Back 47

Narrative Reviews:
Systematic Reviews:
Meta Analysis:

Front 48

Types of lit reviews – 3 types describe and what kind of data do they provide (Qual/Quan?)

Back 48

Narrative Reviews: like qualitative – descriptions
Systematic Reviews: Quantitative – uses a replicable procedure to select r and review the texts, this reduces bias and error EXAMPLES CAMPBELL COLLABERATION & COCHERANE COLLECTIVE (?)

Meta Analysis: what kind of data does it report Qual/Quan? - describe
Quantitative?? Uses stats to offer comprehensive and report on a Converts results to a common metric then compares to effect sizes

Front 49

FREQUENCY POLYGON

Back 49

reflects the distribution of the data – connect the dots to form a continuous line- helps to see a trend – ie increase or decrease etc – ** x axis from 0-“X”
Shows the shape of a distribution

Front 50

UNIVARIATE BAR CHARTS

Back 50

usually for categories of nominal data (x axis: type of flower nominal , y-axis: they show one variable – ie # petals, *** since it is nominal category data on X-axis – the boxes don’t touch - space in between each item.

Front 51

HISTOGRAM

Back 51

Like univariate bar chart BUT used for continuous (rational/integer data), like a bar graph but the bars touch on x-axis to represent continuous data.

Front 52

Measures of Center name 3, describe and symbol

Back 52

mean (xbar) median M mode (?)

Front 53

Measures of Variability (Dispersion) – name 5, describe

Back 53

*a group of stats that precisely describe the amount of variability
min-max - 1-10
range - 9 (10-1)
- interquartile range – a measure of variability, 50 % scores usually in the middle two IQR boxes
- standard deviation (relationship between the score mean of the sample)
- variance – the means squared

Front 54

f, N, n SD, %, Proportion – describe each

Back 54

f= frequency (number of cases)
N = total population
n = sample
SD = standard deviation
r2 = Pearson’s r,
% = # per 100
Proportion
a part of 1.00 (one) , calculate by by multiplying by 100

Front 55

Descriptive Statistics

Back 55

Summarizes data – the range, the mean, etc *will not have a significance level

Front 56

Inferential Statistics

Back 56

How much confidence we can have in making prediction from sample group to population
** look for significance level, p<.05

Front 57

Correlation statistics

Back 57

Show the relationship in a group of data between two or more variable – or one variable
over time.

Front 58

3 Factors that decrease likelihood that difference is due to sampling error (chance) t tests

Back 58

3 Factors that decrease likelihood that difference is due to sampling error (chance)
i.e. – increase chance of ejecting null hypothesis.
• Large sample group
• Small variance within groups
• The larger the observed difference between 2 means

Front 59

t tests

Back 59

Tests for significant differences between two sample group means.

Front 60

Factors for causal relationships:

Back 60

There must be a statistical association between two vaiables
Cause must precede effect – TEMPORAL PRECEDENCE A—LEADS TO B
Rule out moderating variables, ADD THE THIRD ONE!