Research Design In Cd (Kim)

Front 1

*
Purpose of Research Design

Back 1

a. To answer the research question, while controlling factors that may hinder meaningful interpretation
b. Concept of research design typically applies to Quantitative Research

Front 2

**
Classification of Research Design

Back 2

a. Experimental - Studies that seek to identify causal relations thru manipulation of IVs
b. Non-Experimental - Studies that seek to describe events thru passive observation (no manipulation)
c. Examples: Descriptive (Survey, Developmental) & Correlational (“Predictive”)
NB: Most basic distinctions

Front 3

***
Experimental Designs
(3)

Back 3

a. True ED
b. Quasi ED
c. Pre ED

Front 4

****
True Experimental Design

Back 4

a. Three requirements:
-Deliberate, active manipulation of IV
-Use of control group/condition
-RANDOM ASSIGNMENT of subjects to at least two groups/conditions (random assignment does NOT = random selection wh/ is sampling)

Front 5

*****
Quasi-Experimental Designs

Back 5

a. Quasi b/c lacking one of three requirements (the random assignment)
b. Examples:
-A group experiment which lacks random assignment
-A small-N study --> Single-subject experimental Designs belong to this category (different from a single case study) (N = subject)

Front 6

a
Pre-Experimental Designs

Back 6

a. Studies that fail to meet last two criteria of a true experiment (e.g., random assignment, control group)
b. Limited value - evidence not as strong
c. Examples:
-One-shot case study
-One-group pre/posttest study

Front 7

aa
*Internal Validity*

Back 7

a. An experiment is internally valid if there are no confounding variables…that is, the only reason why the groups are different (with respect to the DV) is “actually and only” because of the differences in the IV.
b. Establishing internal validity of a study means must address possible threats to validity
B: Include this info in chart

Front 8

aaa
Facts that can Affect Internal Validity (6)

Back 8

a. History
b. Maturation
c. Reactive Pretest
d. Instrumentation
e. Differential Subject Selection
f. Attrition/Morality

Front 9

aaaa
History

Back 9

a. An event occurs b/t two measurements that is external to the participant/study (e.g., stat quiz on tues vs. thurs b/c of rampage)

Front 10

aaaaa
Maturation

Back 10

a. Systematic, time related changes in the participants that occur between measurements (e.g., babies, aphasics)

Front 11

b
Reactive Pretest

Back 11

a. Changes in the DV occur simply because the DV was measured (not due to particular level of the IV) (e.g., I know you are watching me…)

Front 12

bb
Instrumentation

Back 12

a. Changes in the DV which result from changes/errors in the recording device (e.g., Oops; I’m bored --> TA "forgets" to calibrate audiometer; person makes errors in their observation results)

Front 13

bbb
Differential Subject Selection

Back 13

a. Differences b/w subjects in the experimental and control groups can influence the results (e.g., singing in the shower and being an American Idol fan, or find out one group had much higher IQ, after experiment)

Front 14

bbbb
Attrition/Mortality

Back 14

a. A high rate of mortality is a problem whether occurring in one group only or similarly across the groups
-Yes, I guess it has always been a problem philosophically.

Front 15

bbbbb
Quantitative Experimental Group Designs

Back 15

a. Between-Subjects Design
b. Within-Subjects Design

Front 16

c
Between-Subject Design

Back 16

a. It compares separate, independent groups of individuals for each condition/treatment
b. It allows only one score per participant - thus, also called an INDEPENDENT MEASURE DESIGN

Front 17

cc
Between-Subject Design
(Continued)

Back 17

a. Individuals are assigned to groups using a procedure that attempts to create equivalent groups.
b. Goal of the experiment: to determine whether differences exist b/t two or more (Tx) groups

Front 18

ccc
Between-Subject Design Experiment Example

Back 18

a. Interested in understanding the effect of energy drink consumption on learning
b. A group of students takes a memory test (i.e., a list of words to memorize) after drinking 3 cans of the drink vs. another group takes it after drinking three cans of water. (Random assignment into 2 groups)
c. The groups and conditions were generally comparable.

Front 19

cccc
Between-Subject Design Treatment Example

Back 19

a. Interested in the effectiveness of two Korean teaching methods (on-line game vs. audio)
b. Two groups of military personnel learn 100 words via on-line game method and another group via audio/written word list method
c. Vocabulary test before and after tx
d. Groups and instruction length generally comparable

Front 20

ccccc
Advantages of Between-Subjects Design

Back 20

a. Each score is independent of the other scores
b. Participant’s score is not influenced by practice or fatigue:
-Practice or experience gained in other treatments
-Fatigue or boredom from participating in a series of treatments

Front 21

d
Disadvantages of Between-Suject Design

Back 21

a. Large number of participants
b. Individual differences from one participant to another (eg: varying IQ scores)
-Can produce high variability in the scores
-Can become a confounding variable – participant characteristics differ from one group to another

Front 22

dd
Equivalent Groups

Back 22

a. In a between-subjects experimental design, the researcher has control over the assignment of individuals to groups
b. The separate groups must be:
-Created equally
-Treated equally
-Composed of equivalent individuals

Front 23

ddd
Group Equivalence Techniques
(2)

Back 23

a. Subject Randomization
b. Subject Matching

Front 24

dddd
Subject Randomization

Back 24

a. Assigns subjects to experimental and control groups on a random basis, ensuring that each subject has an equal probability of being assigned to either of the group
b. Extraneous factors that could affect the subject performance on the DV are more likely to be balanced among the groups
c. A powerful technique for reducing systematic bias in assignment to groups

Front 25

ddddd
Subject Matching

Back 25

a. Purposely attempt to match the members of the two groups on all extraneous variables considered relevant to the experiment
b. Two common techniques:
-Matching the overall distribution of the extraneous variables in the groups
-Matching pairs of subjects for assignment to the two groups
-Matching pairs + randomization --> a powerful technique
NB: Need small groups to be able to use this technique

Front 26

e
Subject Matching
(Continued)

Back 26

a. Randomization is preferred when a large number of subjects are available
b. Matching pairs + subsequent randomization of pair members is useful when the size of N is small

Front 27

ee
Within-Subject Design

Back 27

a. The performance of the same subjects is compared in two or more different (Tx) conditions. (Same subjects, twice!)
b. Often called A REPEATED MEASURES DESIGN b/c the study repeats measurements of the same individuals under different conditions
c. Used to compare different conditions or to investigate changes occurring over time

Front 28

eee
Within-Subject Design Study Example

Back 28

a. Interested in the effect of different types of exercise on memory
b. All participants take a memory test after 10 min. of aerobic exercise and after 10 min. of anaerobic exercise.
c. Compare the memory test scores to answer the Q as to what type of exercise aids memory the most.

Front 29

eeee
Advantages of Within-Subject Design

Back 29

a. Requires relatively few participants/1 group
b. Eiminates all of the problems based on individual differences:
-No difference b/w groups
-Each individual serves as his own control/baseline
c. The concern is not group equivalence but assuring the equivalence of all conditions.

Front 30

eeeee
Disadvantages of Within-Subject Design

Back 30

a. Time-related problems --> participant attrition, history, maturation
b. Order Effects
- Improvement may occur between the beginning and end of an experiment due to practice, familiarity, or fatigue.
-CARRYOVER EFFECTS: May occur whenever exposure to one Tx condition permanently or temporarily affects performance in subsequent condition (e.g., new skill from Tx 1 can influence results in Tx 2).

Front 31

f
Dealing w/ Disadvantages

Back 31

a. When carryover is likely to be permanent, a within-subjects design is not a good idea.
b. Counterbalancing:
- Involves changing the order in which tx conditions are administered from one participant to another
-The goal is to use every possible order of treatment with an equal number of individuals participating in each sequence

Front 32

z
Revisiting Internal Validity

Back 32

a. Threats to internal validity are confounds that serve as plausible alternative explanations for a research finding. (So, how do you deal with them?)
b. Extraneous variables
c. Confounding variables

Front 33

zz
Extraneous Variables

Back 33

a. Variables that MAY compete with the IV in explaining the outcome of a study/DV

Front 34

zzz
Confounding Variables

Back 34

a. An extraneous variable that DOES indeed influence the DV
b. Systematically varies or influences the IV and also influences the DV
c. Researchers must always worry about extraneous variables when they make conclusions about cause and effect (Think about everything that could possibly go wrong, make a list, and then use experts to assess)

Front 35

zzzz
Research Design & Dealing w/ Threats to Internal Validity

Back 35

a. Many threats are controlled to large extent by using true experimental design
b. Some of these are not a threat to an experimental, two group design because it equally affects the two groups: history, maturation, reactive pretest, differential subject selection, & instrumentation (to a large extent)
c. Attrition - a problem for both internal and external validity.

Front 36

zzzzz
Research Design & Dealing w/ Threats to Internal Validity
(Continued)

Back 36

a. Instrumentation (inconsistent machine or human performance not a systematic error) could be a problem
b. Effort to fix: short time frame for testing (so less option for people to drop out, protocol development, reliability... )
c. There are always other issues/potential problems which are unique to a given study, & may not necessarily fall under the categories of the threats to internal validity but you have to address, anyway.

Front 37

ff
Limitations of Group Comparison Approach

Back 37

a. Ethical objections (“Rob needs it, too!”)
b. Practical problems (“What was I thinking when I said 500?”)
c. Averaging of results--> Don't look at individual, look at avg results of group as a whole
d. Generality of findings (“Would size S-M-ML fit Cinderella? Can’t tell until we try….”) --> Related to Averaging of Results problem
NB: Clinically, group therapy does't easily translate to clinical treatment)

Front 38

fff
Quantitative, Quasi-Experimental, Single-Subject Research Design

Back 38

a. Withdrawal and Reversal Designs
b. Multiple-Baseline Designs
c. Alternating Treatment Design

Front 39

ffff
Introduction to Single-Subject RD

Back 39

a. Research designs that use the results from single participant to establish the existence of cause-effect relationship
b. The design can be applied to a small number of subjects who are evaluated as separate individuals (not avg of whole group)
c. Also known as "small-N designs" (N=Sample)

Front 40

fffff
Introduction to Single-Subject RD
(Continued)

Back 40

d. Often called TIME-SERIES DESIGN because involves the systematic collection of a series of measurements of the DV over a period of time
e. Quasi-experimental design b/c subjects are not randomly assigned (However, DOES permit cause-effect conclusions)

Front 41

g
Introduction to Single-Subject RD
(Continued 2)

Back 41

f. Useful in gathering preliminary data
g. The unit of analysis is usually a single participant
h. Typical sequence of events: (O=Observation, I=Intervention)
-O1 O2 O3 I1 O4 I1 O5 I1 O6
(NB: second number in each set is supposed to be subset)

Front 42

gg
Two Basic Phases of Single-Subject RD

Back 42

a. Baseline Phase
b. Treatment Phase
NB: Phase = series of observations of the same individual under the same conditions

Front 43

ggg
Baseline Phase A

Back 43

a. When no tx is being administered, the observations are called "baseline observations"
b. A series of baseline observations is called a baseline phase (identified by the letter A)
c. Purposes: To describe the extent of the individual’s problem or status of the target behavior AND/OR To predict future behavior if intervention is not provided
d. If the observed behavior is stable, with a predictable trend, an intervention phase is initiated.

Front 44

gggg
Treatment Phase B

Back 44

a. When a tx is being administered, we have treatment observations.
b. A series of these observations is called a treatment phase (identified by the letter B)

Front 45

ggggg
Withdrawal Designs

Back 45

a. Compares a subject’s behavior at a time when the IV (experimental intervention) is present with the behavior observed and when the IV is absent or withdrawn

Front 46

h
A-B-A Withdrawal Design

Back 46

a. Baseline-Intervention-Baseline (withdrawal phase)
b. A prototype for single subject design
c. Comparison between the baseline and treatment phase performance allows one to infer the effect of the treatment
d. Return to baseline of the DV during the withdrawal phase also suggests the causal relationship b/t the IV and the DV

Front 47

hh
A-B-A-B Withdrawal Design

Back 47

a. An extension of the A-B-A design in which intervention is reintroduced in the B2 phase (treatment reinstatement segment/phase)
b. If a regression to baseline effect is observed in A2, the intervention is reintroduced in B2.
e. A similar change in DV observed during B1 and B2 allows one to conclude that the intervention caused the change in behavior.

Front 48

hhh
Advantages of A-B-A-B Withdrawal Design

Back 48

a. It addresses the ethical concern that the experiment might conclude without the benefit of intervention for the participant by ending in withdrawal phase
b. It replicates the A-B comparison, providing additional support for a cause-effect relationship between IV (intervention) and DVs (observed behavior).
c. A2 is important because it allows one to rule out history and maturation effects (b/c they were able to go back to baseline phase)

Front 49

hhhh
A-B-A-B WD Example

Back 49

a. Dr. Kim is working to improve a new CMD 160 student’s ability to remain in seat during class using an ABAB Design.
b. Phase A: How many times did he get out of his seat in class today? (Getting baseline data)
c. Phase B: Bob gets M&Ms every 5 min. when he remains in seat. This goes on for one week. Count the frequency of his out of seat behavior.
d. Phase A: Teacher runs out of money. No M&M’s. (Baseline phase, again)
e. Phase B: Teacher gets a salary increase. The M&M program reinstated.
NB: Treatments are the same in both treatment phases and Dr. Kim proved effectiveness of treatment program twice in one study

Front 50

hhhhh
A-B-A-C Design

Back 50

a. Useful in testing the effects of two treatment variables (two treatments - B & C are each their own treatments)
b. Potential problem: multiple treatment interference (a threat to internal validity) - A problem inherent in this design because the effects of treatment B are likely to carry over to the C treatment
c. Controlled by counterbalancing the order of treatment across participants (A-B-A-C, A-C-A-B,…)

Front 51

i
Reversal Designs

Back 51

a. Use when expecting difficulty to revert the DV to baseline levels by simply withdrawing the intervention (i.e., extinction)
b. A phase following the Tx involves an active reversal (by reinforcing alternative behavior that is incompatible with the target behavior trained previously)
NB: Clinically, you hope reverting back to Baseline Phase (A) doesn't take away what the client learned during Treatment Phase (B)
NB: Can raise ethical question of withdrawing and reversing a successful treatment

Front 52

ii
Multiple-Baseline Design

Back 52

a. Initiates intervention following different baselines sustained for different lengths of time, to establish the relationship b/t the DV & IV, while minimizing the threats to the internal validity (maturation, timing of training, etc.) --> start at varying baselines
b. Useful for studying the effects of a Tx in which two or more behaviors, people, or settings can be tracked on a single experiment

Front 53

iii
Multiple-Baseline Design
(Continued)

Back 53

a. Target variables must be independent of one another: A change in treated variable should not be accompanied by a change in an untreated variable (b/c if change occurs before you initiate treatment, there's a problem!)
b. Tx effect is demonstrated by showing that the response changes whenever the Tx is introduced at the different points in time.

Front 54

iiii
Multiple-Baseline Design Example

Back 54

a. Pre-intervention reading fluency baseline
b. A staggered entry into the Tx program
NB: Change only occurs when treatment is presented, so it must be effective
NB: Needs at least three people for this type of test
NB: Clinically, you can see which your pt resembles and decide on treatment - so more clinical design!
NB: Reading Fluency Treatment for 3 Future CMD Grad Program Applicants (See Graph)

Front 55

iiiii
Advantages/Features of Multiple-Baseline Design

Back 55

a. A flexible design that can simultaneously analyze the tx effect on multiple people, behavior, or setting
b. It is not necessary to return a participant to baseline condition.
c. It avoids the ethical concern regarding withdrawal of treatment (i.e., a return to baseline is not desirable)
d. A return to baseline is not possible - a DV does not respond to withdrawal of the treatment due to carryover or long-lasting effects of the Tx.
NB: This Design used more often in our field than in others b/c we don't want our clients to return to baseline after treatment

Front 56

j
Disadvantages of Multiple-Baseline Design

Back 56

a. The design is weaker than the A-B-A design because the controlling effects of the treatment variable on the DV are not directly demonstrated.
b. It requires a multitude of data collection over an extended period of time, requiring resources for planning, implementing, and ensuring the cooperation of participants.

Front 57

jj
Alternative Tx Design

Back 57

a. Used to compare the effects of two treatments on one subject across the same span of time (A-B-A-B-A-B-A-B) --> sometimes you see pt's twice a day - in the morning you're in tx a and in the afternoon you're in tx b (naming treatment with two diff methods - diff word selection but comparable)
b. Two Txs are introduced and alternated at each of the training sessions, usually at a rapid succession
c. Example: comparison of two reading fluency program for a student

Front 58

jjj
Advantages of Alternative Tx Design

Back 58

a. If one treatment is shown to be superior to the other, the superior treatment may be continued in both sessions during phase C.

Front 59

jjjj
Disadvantages of Alternative Tx Design

Back 59

a. Possible multiple treatment interference
b. The design depends on treatments that have little or no carryover effect from treatment to treatment. (If you expect carryover effect, don't even try to use this design!)

Front 60

jjjjj
Visual Inspection vs. Statistical Tests in Assessing Tx Effectiveness

Back 60

a. Repeated Measurement
b. One Variable Rule vs Exceptions
c. Other Variables
d. Visual inspection (eyeballing) is the initial choice, however, it is subjective with problematic interrater agreement.
d. Typical parametric tests (e.g. t-test, ANOVA) are inappropriate due to correlated nature of the data.
f. Alternative procedures examining the overall change, change in slope, etc., are more appropriate (e.g., Interrupted Time Series Analyses (ITSA))

Front 61

k
Repeated Measurement

Back 61

a. The operations must be clearly specified, observable, and replicable
b. Measurements taken repeatedly must be done under totally standardized conditions of:
-Measurement devices
-Personnel involved
-Time or times of day measurements are recorded
-Instructions
-Environmental conditions

Front 62

kk
One Variable Rule vs. Exceptions

Back 62

a. Rule: change one variable at a time when proceeding from one phase to the next

Front 63

kkk
Other Variables to Consider

Back 63

a. Length of Phases: baseline vs. intervention phase? (want them to be comparable in length, but hard to do so, make baseline longer and tx phase shorter)
b. Carryover Effects: Multiple observations can affect participant’s responses (Next variable may change much more rapidly than first tx)