Chapter 4: Reliability And Validity

Front 1

Reliability

Back 1

- consistency

- reliability coefficient normally assumes values of 0.00-1.00

Front 2

What types of things are tested for reliability?

Back 2

1. observers or raters

2. tests over time

3. different versions of the same test

4. a test at one point in time.

Front 3

internal consistency

Back 3

- inter-correlations among individual items of a knowledge test or trials on the same day.

Front 4

what is a sufficient number for reliability?

Back 4

.70 or higher

Front 5

Types of Internal consistency statistics?

Back 5

1. Split Halves - compare odd to even

2. KR-20 - better than split halves. It randomnizes the order of items (uses to measure correct vs incorrect response).

3. Cronbach's alpha - same as KR-20 but can be used with scaled items like Likert scale of strongly agree to strongly disagree.

Front 6

Stability (test-retest)

Back 6

- results would be the same if administered to the same person again at a later time.

- coefficient generally decreases as the time between testing increases

-

Front 7

Parallel-forms

Back 7

- administer two forms of the same instrument to a single group within a short time period between testing

- correlate the results

Front 8

Parallel-forms reliability measures what?

Back 8

- stability across forms

Front 9

Inter-Rater or Inter-Observer Reliability

Back 9

-measure if different observers are consistent

-establishes percent of agreement

-can use correlation (with continuous ratings)

Front 10

Inter-rater reliability statistics

Back 10

1. Kendall coefficient of concordance - each ranks the thing being evaluated. The higher the agreement between raters the closer to +1

2. Cohen's Kappa - used for nominal level data (can be used with more than 2 raters, but no in SPSS)

3. Intraclass correlation coefficients (ICC) - for interval or ratio level data.

Front 11

ICC

Back 11

- based on repeated measure ANOVA

- 6 different formulas for different situations

- can do more than 2 repeated measures (correlation only two)

- determines absolute and relative reliability

Front 12

how do you assess observer reliability?

Back 12

1. unit by unit agreement requires two or more observers to agree on individual instances of response being measured

2. # of units on which both agree is counted A

3. # of units on which both agree is counted D

4. index = (A/A+D) x 100

Front 13

list observer issues

Back 13

1. bias - may see what they want to see or their own preconceived idea

2. observer should report data not inferences

3. use of trained observers

Front 14

how do you measure reliability with Pearson r

Back 14

- add paired t-test to determine the magnitude of the differences

- regression - absolute reliability established if slope of the line is close to 1 and intercept is close to 0.

- calculate SEM for paired data

Front 15

Standard Error of the Measurement (SEM)

Back 15

- can be used to estimate the range a score would fall if a given measured object was re-measured (confidence bands)

- if reliability is high, SEM will be small.

Front 16

can an instrument be reliable without being valid?

Back 16

yes

if the instrument is valid it will be reliable

Front 17

what is validity

Back 17

a test or survey is valid when it measures what it is supposed to measure

Front 18

how does one gain content validity?

Back 18

- literature or expert panel

- subjective measurement

Front 19

2 types of criterion based validity

Back 19

1. establish criteria called gold standard to compare with new measure

2. compares between two measures

Front 20

concurrent validity

Back 20

- two measure at the same time (exam scores and overall GPA)

- measured by a correlation coefficient (r)

Front 21

Predictive validity

Back 21

- measurement correlated with a future measurement (GRE scores used to predict graduate school success)

Front 22

construct validity

Back 22

-psychological constructs measured such as self-esteem

-phsical measures such as the broad construct of strenght

Front 23

steps to a factoral analysis

Back 23

1. group of variables analyzed for inter-relationships (correlation matrix)

2. underlying factors determined by reviewing variability of values in data set that can be accounted for by the factor).

3. factors are extracted-removal of factors that don't measure up

4. rotate factors-maximize the appearance of differences between factors to create factor loadings

5. examine rotated factors to determine simple structure. Retain variables in each factor that loaded above a cutoff point (commonly .30)

6. name and interpret factors based upon the remaining variables in each factor

Front 24

reliability and validity are what?

Back 24

just estimates. not absolute values