04 Methodological Problems

Front 1

Participant observation

Back 1

Front 2

Naturalistic observation

Back 2

Field study

Front 3

Direct observation

Back 3

With direct, systematic, observation of behavior:

– People are put in a particular situation

– Their behavior in that situation is observed

– Observation typically involves a protocol

– The protocol allows scoring of how often or to what extend a particular behavior is displayed

Hint 3

• Research has shown that babies often don’t recognize changes in their sight

• This is called ‘change blindness’

• Hypothesis: adults also suffer from change blindness

Front 4

structured observation

Back 4

• In this example, observations are relatively straightforward: does the person notice the change?

• Typically, the observation follows a clear protocol that describes how particular behaviors are to be scored.

Hint 4

• Conditions under which behavior is observed are controlled by the researcher

• We can check inter-rater reliability

• And the procedure can be replicated: used in different or in the same context repeatedly to study generalizability

• By varying conditions we can also study causal factors

Front 5

Observer bias

Back 5

• We see what we want to see!

• Observation is an active process that can be heavily influenced by expectations

Observer bias occurs when:

• The observer assesses the situation incorrectly

Front 6

Solutions observer bias

Back 6

• Objective and unambiguous scoring protocols

• Interrater reliability or interobserver reliability:

Do different observers observe the same things independently from each other?

Front 7

Blind study

Back 7

Experimenter does not know

who is assigned to which condition

Front 8

Double-blind study

Back 8

– Experimenter does not know who is assigned to which condition

– Participant does not know to which condition (s)he is assigned to

Front 9

Placebo-controlled Double-blind study

Back 9

– Experimenter does not know who is assigned to which condition

– Participant does not know to which condition (s)he is assigned to

– Those in control group receive placebo treatment

Front 10

Observer effects

Back 10

• When the observer influences

the behavior in such a way

that it becomes in line with the hypothesis

Front 11

Reactivity

Back 11

When the observer influences behavior

Front 12

Rosenthal effect / Pygmalion effect

Back 12

higher expectations lead to

an increase in performance

Front 13

Solutions observer effects & reactivity

Back 13

1. Hide: ensure that participants are unaware of being observed

2. Wait it out: wait until participants are accustomed to being observed

3. Deceive: lead participants to believe that they are not being observed (or that you’re interested in something else)

4. Use unobtrusive measures: look not at behavior but rather at effects or outcomes of that behavior

5. Use blind observers and experimenters: if observers or experimenters do not know what is to be expected, their expectations cannot affect participant’s behaviors

Front 14

Sampling bias

Back 14

occurs when the sample differs systematically

from the target population on relevant variables

Hint 14

– What are the relevant variables?

– What is the target population?

Front 15

SAMPLING

Back 15

Front 16

Convenience sampling

Back 16

just select participants that happen to be available

Front 17

Purposive sampling

Back 17

including a selected group that has desirable characteristics

Hint 17

choose a state that voted similarly

to the overall population on the last election

Front 18

Cluster sampling / multi-stage sampling

Back 18

1) select a certain cluster of units (e.g., 5 municipalities in a country) randomly

2) randomly select participants within the selected units (e.g., schools).

Front 19

Quota sampling %%%

Back 19

type of purposive sampling in which one ensures

that each “stratum” is sufficiently represented in sample

Front 20

Stratified random sampling

Back 20

create strata, and within each stratum

randomly select individuals

(so each stratum is well-represented within the sample)

Front 21

cons of case studies

Back 21

Hard to determine causality with only one case (bc of: confounding, present-present bias)

• Sensitive to bias (‘observer bias’ en ‘observer effects’)

• Poor generalizability: Only with very homogeneous population would one case suffice to understand other cases