Research Methods - Research Designs

Front 1

Sampling Error

Back 1

The difference between sample mean and population mean

Front 2

Sampling error in One-sample t-test

Back 2

standard error of the mean

Front 3

Sampling error in Independent t-test:

Back 3

standard error of difference

Front 4

Sampling error in Analysis of Variance F-test

Back 4

mean square error

Front 5

Sampling error in Correlation coefficient-r

Back 5

standard error of estimate

Front 6

Sampling error in Multiple regression

Back 6

standard error of regression coefficient

Front 7

Quasi-experimental research

Back 7

No random sampling assignment

Front 8

Best way to reduce sampling error

Back 8

random sampling (experimental research)

Front 9

Error of Measurement

Back 9

The difference between an obsered and a true score

Front 10

Random error of measurement

Back 10

fluctuations in the characteristics of individuals, fluctuations in examiner assessment procedures

Front 11

Systematic error of meausrement

Back 11

decrease/increase in scores in predictable way

Front 12

Does measurement error always exist?

Back 12

measurement error exists in all studies! We cannot eliminate errors in measurement, but we can minimize their effects.

Front 13

Internal Validity

Back 13

When a researcher controls for all extraneous variables and the only variable influencing the results of a study is the one being manipulated by the researcher. (the variable intended to study is indeed the one effecting the results)

Front 14

T.H.I.S. M.E.S.S. D.R.E.A.D. - what to do with

Back 14

Threats to Internal Validity. (THIS MESS) can be assessed and ruled out through the use of experimental designs and the last five threats (DREAD) are corrected using experimental procedures

Front 15

THIS MESS DREAD - what are they

Back 15

Testing Design

History

Instrumentation

Selection (differential)

Maturation

Experimental Mortality (attrition)

Statistical Regression

Selection-Maturation Interaction

Diffusion of Experimental Effect

Rivalry (Compensatory)

Equalization of Treatments (Compensatory)

Ambiguous Temporal Precedence

Demoralization (resentfulness)

Front 16

Testing Design - threat to internal validity + example

Back 16

The treatment effect may be confounded when changes in post-test scores of participants are influenced by their experience from taking a pretest. Hence the test becomes part of the intervention.

Example: A study with repeated measures or a pre-/post-test design.

Front 17

History - threat to internal validity + example

Back 17

An event that occurs during a study that can effect the responses of the participants; it could be something in the news, or national publicity, (i.e. an earthquake)

Example: Longitudinal study, a study with repeated measures.

Front 18

Instrumentation - threat to internal validity + example

Back 18

Pretest and posttest scores may change because of a faulty measurement instrument, irrespective of the treatment.

Example: Physiological instruments, or those in which researchers are collecting data in person.

Front 19

Selection (differential) - threat to internal validity + example

Back 19

Differences in the characteristics of participants assigned (usually not random) to treatment conditions may confound attributing the changes in the dependent variable to the treatment.

Example: Quasi-Experimental studies and convenience samples.

* Doing statistical comparisons between groups can help address this threat.

Front 20

Maturation - threat to internal validity + example

Back 20

Psychological and physical changes within participants may occur in an experiment, especially over time. These participant maturational changes may have an extraneous effect on the dependent variable.

Example: Longitudinal studies, where the participants are more likely to change such as adolescents, infants or people who are severely ill.

Front 21

Experimental Mortality (attrition) - threat to internal validity + example

Back 21

No, this doesn’t mean the sick people die!

The loss of participants to treatment or measurement of the dependent variable can produce unbalanced attribution of the effects of the treatment on the dependent variable.

Example: Longitudinal studies.

NB: Its important to analyze the existing data to determine if there are differences between those who dropped out v.s. those who continued. If more than 10% of the participants are lost to follow-up this seriously effects the ability to generalize findings.

Front 22

Statistical Regression - threat to internal validity + example

Back 22

This is the phenomenon that extremely high or low group scores on a variable tend to regress to the mean on the second measurement of the variable, confounding the treatment effect.

Example: Pretest-Posttest designs, small sample size studies, fewer people means that each individual has a greater effect on the average score.

Front 23

Selection-Maturation Interaction - threat to internal validity + example

Back 23

This is a combination of two threats to internal validity. For example, some participants in one assigned treatment condition group may have matured in math self-efficacy more than participants in another treatment condition group and purpose of the study is to increase math achievements. Additionally, other combinations of threats could interact to confound the treatment effect.

Example: Three most likely interactions are history, maturation and instrumentation.

Front 24

Diffusion of Experimental Effect (Diffusion of Treatment) - threat to internal validity + example

Back 24

The treatment may diffuse to the control group over time because the control group may seek access to the more desirable treatment. “Contamination” of the control group.

Example: Studies with an intervention when the two groups are being studied at the same time in the same location (clinic, small town). This is essentially no control group. Fatal flaw!

Front 25

Rivalry (Compensatory) - threat to internal validity + example

Back 25

The control group participants may perform beyond their usual levels because they perceive that they are in competition with the experimental group.

Example: Intervention studies where participants know which group they’re in.

Front 26

Equalization of Treatments (Compensatory) - threat to internal validity + example

Back 26

A treatment group may receive experimental rewards that appear more desirable than those received by the control group. Efforts are made by individuals outside of the experiment to compensate the control group participants with similar desirable goods. This would obscure the results of treatment.

Example: health care provider with a heart of gold (and little common sense). Intervention Studies.

Front 27

Ambiguous Temporal Precedence - threat to internal validity + example

Back 27

A lack of clarity is provided by the researchers as to which variable occurred first, leading to a question of which variable is the cause and which is the effect.

Example: lung cancer, smoking, which came first? Does smoking really cause lung cancer? (No! :)

Front 28

Demoralization (resentfulness) - threat to internal validity + example

Back 28

Lower performance of control group participants on the dependent measures may result from their belief that the treatment group is receiving a desirable treatment.

Example: Just like rivalry (compensatory), but the opposite.

Front 29

Experimental Designs - key symbols

Back 29

 R: random assignment

 O: observation

 X: treatment condition

 C: control condition

Front 30

Randomized Treatment and Control (2 groups) with Posttest-Only Design --> what test do you use?

Back 30

Independent sample t-test

Front 31

What does a t-test assume?

Back 31

normal distribution, homogeneity of variance

Front 32

Randomized Multiple Treatments and Control with Posttest-Only Design --> what test do you use?

Back 32

One-way ANOVA

Front 33

What does ANOVA assume?

Back 33

normal distribution, homogeneity of variance +

either independence of scores related to the DV if it is between groups (one-way ANOVA) or covariance related to the DV (RM-ANOVA)

Front 34

Randomized Multiple Treatments and Control with Pretest and Posttest Design

Back 34

3x2 repeated-measures ANOVA

Front 35

What does t-test examine?

Back 35

compare means between experimental and control group conditions for significant differences

Front 36

What does one-way ANOVA examine?

Back 36

compare means across different experimental conditions for significant differences (post-hoc analysis to compare three pairs of means)

Front 37

Randomized Multiple Treatments and Control with Pretest and Posttest Design -- what test do you use?

Back 37

3x2 repeated-measures ANOVA

Front 38

What does 3*2 repeated-measures ANOVA examine?

Back 38

compare means across different experimental conditions for significant differences (post-hoc analysis to compare three pairs of means)

also: diffs pre and post, interaction treatment x time

Front 39

Recommended experimental research procedures to reduce the threats of DREAD

Back 39

1 Use different persons implement treatments

2 Minimize contact experimental/control groups

3 Use blinded procedures

4 Assess expectations of ppl in treatment cndtn

5 Consider not using experimental rewards

6 Clearly define treatment conditions

7 Use past research and theory to guide the evidence of an A (IV) to B (DV) causal relationship

Front 40

Quasi-Experimental Designs

Back 40

no random assignment, but has a control group or multiple measures

Are considered compromised designs

Front 41

Methods to improve Quasi-experimental designs

Back 41

1. Adding control or comparison groups 2. Adding pretest and posttests 3. Removing and reinstituting treatments 4. Adding replications 5. Reversing treatment 6. Case matching (e.g. propensity scores)

Front 42

Simple ex post facto design (Quasi-Experimental) --> what kind of test to use?

Back 42

X is not a 'treatment' but a *prior* event

can still use Independent sample t-test

Front 43

Correlational Research Designs - purpose?

Back 43

investigation of bivariate or multivariate relationships, predictions

Front 44

Independent variable vs . Dependent variable

Back 44

IV - what is before the measurement, how to separate the groups;

DV - what is measured, the test scale

Front 45

"Relationship of participants' scores across groups being compared - Not-related (Independent) vs. Related (Dependent)"

Back 45

Between-group design = independent (not-related) ; Within-group design = dependent (related) {scores on the same, or matched, participants are obtained two or more times};

Correlation/Multiple Regression = dependent

Front 46

Scale of measurement of DV in one-way, RM, multifactor ANOVA

Back 46

continuously scaled (interval or ratio) DV

Front 47

Simple RM Anova - how many IVs?

Back 47

Only one

Front 48

Factor

Back 48

= independent variable

Front 49

Purpose of simple RM-ANOVA

Back 49

compare mean differences across groups, conditions, testing times (i.e pretest-post-test)

Front 50

Purpose of factorial (mutli-factor) ANOVA

Back 50

assess mean differences across main effects, interaction effects and simple effects

Front 51

One-way ANOVA = between or within groups?

Back 51

Between group design

Front 52

RM ANOVA = between or within groups?

Back 52

Within group design

Front 53

Factorial ANOVA - between or within groups?

Back 53

Either; can include both scores that are not related (between groups) and scores that are related (within groups)

Front 54

Purpose of one-way ANCOVA

Back 54

compare mean differences among groups when the DV has been adjusted for by one more covariates

Front 55

Number of IV in a one-way ANCOVA

Back 55

One!

Front 56

Scale of measurement in one-way ANCOVA

Back 56

continuously scaled (interval/ratio) DV

Front 57

one-way ANCOVA - between or within groups?

Back 57

Depends.. there are different participants in the groups, DV scores are not related (independent) --> so between groups.

But, if a covariate used is a pretest / posttest score, then there are elements of *both* between-group and within-group designs

Front 58

Mann-Whitney U Test - purpose and use

Back 58

compare mean rank differences between two groups

the non-parametric alternative to the independent t-test

Front 59

Number of IVs in a Mann-Whitney U Test

Back 59

One!

Front 60

Scale of measurement of DV in a MWU test

Back 60

Discrete-ordinal DV is used in the MWU analysis

Continuous DVs can be used and the observed scores are converted to ranks in the analysis

Front 61

Pearson’s Product Moment Coefficient of Correlation (r) - the purpose

Back 61

Analyze the relationship between two variables

Front 62

Number of IVs in Pearson’s Coefficient of Correlation

Back 62

Two variables are used in the analysis.

It is *not* necessary, but one may be called an IV (X) or predictor variable and the other may be called a DV (Y) or criterion variable

Front 63

Scale of measurement of variables in Pearson's correlation coefficient analysis

Back 63

Continuously scaled (interval/ratio)

Front 64

Relationship of participants scores in Pearson's correlation analysis

Back 64

Scored on the two variables paired on same participants - dependent on each other ("within group")

Front 65

Underlying assumptions of the MWU test

Back 65

Non-parametric - "distribution-free" test

Therefore assumptions of normality and homogeneity of variance are not necessary!

However it is important that observations are independent from each other and there is some degree of continuity of the variable used

Front 66

Underlying assumptions of the Pearson's Correlation Coefficient test

Back 66

*Requires* meeting normality and homogeneity in arrays that are the residual variance of a Y conditional to a specific X

Front 67

Multiple Regression Analysis - purpose ?

Back 67

To analyze the extent that two or more independent variables relate to a dependent variable

Front 68

Number of IV in a Multiple Regression Analysis

Back 68

Two or more continuously scaled independent (predictor) variables used in the analysis

Front 69

Scale of measurement of DV in Multiple Regression Analysis

Back 69

Continuously scaled (interval/ratio)

Front 70

Relationship of participants scores across groups compared in a Multiple Regression Analysis

Back 70

Participants have scores on *all* of the variables used in the MRA, therefore the scores are related to (dependent on) each other

Front 71

Underlying assumptions of Multiple Regression Analysis

Back 71

MRA assumes or asses various issues, including normality, homoscedasticity, linearity, independence of errors, multicollinearity

Front 72

homoscedasticity

Back 72

a sequence or a vector of random variables is homoscedastic if all random variables in the sequence or vector have the same finite variance. This is also known as homogeneity of variance

Front 73

NOIR

Back 73

level of measurement of the DV

Front 74

Multi-Group Design - what test do you use

Back 74

One-way ANOVA

Front 75

Distributing survey comparing two things that people do anyway -

Back 75

Correlation (Bivariate)

Front 76

Two-group, Pre-Post Design - what test do you use?

Back 76

t test for dependent means (Paired Samples)

Front 77

GLM-Univariate

Back 77

same thing as Factorial / Two-way ANOVA

Front 78

1+ categorical variables; looking for even distribution of cases per category

(e.g Determine if number of males/females differs by year in school)

- what is the test used?

Back 78

Chi-square

Front 79

2 continuous variables

e.g - Determine association between intelligence and GRE scores.

- what is the test used?

Back 79

Correlation (Bivariate)

Front 80

3+ continuous variables (1 of which you are trying to predict that serves as the DV) --

e.g what the best predictor is of counseling skill (college grades; emotional intelligence; hours of experience)

- what is the test used?

Back 80

Regression (Linear)

Front 81

2 x 2 Mixed Design - what is the test used?

Back 81

GLM-Univariate (Factorial/Two-Way ANOVA)

Front 82

Testing Males vs Females on verbal ability

Back 82

Two-Group/Simple Experiment: 1 IV; 2 levels (Between-Subjects); 1 DV

T-test for independent means

Front 83

Testing Mood before and after an exercise program

Back 83

Two-Group/Pre-Post Design: 1 IV; 2 levels (Within-Subjects); 1 DV

T-test for dependent means (Paired Samples T-test)

Front 84

Effectiveness of psychoanalysis vs. cognitive-behavioral vs. no treatment on depression

Back 84

Multi-Group Design: 1 IV; 3+ levels (Between-Subjects); 1 DV

One-way Analysis of Variance (ANOVA)

Front 85

Stress level measured each week following either a week of problemfocused coping; emotional-focused coping; or nothing.

Back 85

Multi-Group Design/Repeated Measures: 1 IV; 3+ levels (Within-Subjects); 1 DV

GLM-Repeated Measures Analysis of Variance (ANOVA)

Front 86

Want to test the effect of product advertising (magazine vs. tv) and product cost (low vs. high) on sales

Back 86

Factorial Design: 2+ IV; 2+ levels each (Between-Subjects); 1 DV

GLM-Univariate (Factorial/Two-way ANOVA)

Front 87

Want to test if there is a change in willingness to help before/after a course on public service and if it varies by gender.

Back 87

Mixed Design: 1 IV; 2+ levels (Between-Subjects) 1 IV; 2+ levels (Within-Subjects) 1 DV

GLM-Repeated Measures Analysis of Variance (ANOVA)