Test Construction

Front 1

Classical Test Theory

Back 1

Any given score is combination of truth and error

Front 2

Classical Test Theory: Variability

Back 2

Total Variability in a group is combination of true score variability (differences in test taker, 'reliability') and error variability.

Front 3

Sources of Error in Variability

Back 3

Content Sampling (items do or don't tap into domain by chance)

Time Sampling

Test Heterogeneity (more domains tapped into = more error due to chance)

Front 4

Reliability Coefficient (rxx or rtt)

Vs. Pearson R

Back 4

rxx/rtt
Range: 0.00-1.00
acceptable reliability: .80
True score variability: 80%

Pearson R: -1 - +1
Square for true variability

Front 5

Factors Affecting Reliability

Back 5

Number of Items

Range of Scores (full range + reliability)

Homogeneity of Items

Ability to Guess

Front 6

Estimates of Reliability:
Test-Retest reliability (AKA coefficient of stability)

Back 6

test steady over time
Source of Error: Time Sampling

Front 7

Estimates of Reliability:
Parallel Forms Reliability (AKA Coefficient of Equivalence)

Back 7

2 different forms given, 1 group.
Source of Error: Time sampling, content sampling

Front 8

Estimates of Reliability:
Internal Consistency Reliability
-split half reliability

-Kuder Richardson/Cronbachs Coefficient

Back 8

Consistency across items. Given 1 time, 1 group.
1. Split half reliability
Spearman Brown Prophecy Formula: shows how much more reliability would be w/all items (vs. 1/2, which has lower reliability)
-inappropriate for speeded (vs. 'power' tests).

Source of error: Item/content sampling

Front 9

Estimates of Reliability:
Internal Consistency Reliability
-Kuder-Richardson (kr20 & 21) & cronbach's coefficient alpha

Back 9

Compares all possible halfs
KR20/21 used for dichotimous data (20 if varying difficulty, 21 if not)

Cronbachs coefficient alpha: for nondichotimous (likert type q's)

Front 10

Inter-rater Reliability

Back 10

Used when ratings are subjectively scored.
Calculated w/Pearson R, Kappa, Yules Y.

Improved by: Group discussion, practice exercises, feedback.

Front 11

Standard Error of Measurement

Back 11

Avg amount of measurement error in any given test

Smea = SDx X sq rt 1-rxx

Front 12

Standard Error of Measurement Range

Back 12

0.0 (perfect) - SD of test (whatever that is). Given formula (sd x sqrt 1-rxx)

Front 13

*Calculating Confidence Bands:
-need to know (2 things):

Back 13

Need persons score on test ex. 120
Need persons standard error of measurement ex. 6
Add/subtract standard error of measurment.
68%: 114-126
95%: 108-132,
99%: 102-138

Front 14

Validity - what are the three subtypes?

Back 14

Content: is tool measuring skills it should be (expert validated)

Criterion: Is test accurate predictor? -2 subtypes (Concurrent/predictive)

Construct: Is tool measuring trait it should be

Front 15

Criterion Related Validity:
Calculated by?

Subtypes definitions?

Back 15

Pearson r - score on x/y (range -1 - +1) valid=.20 or higher

variance calculated by squaring: validity-.5 = 25% of varability in scores of y accounted for by x.

Front 16

Criterion Related Validity:

2 Subtypes

Back 16

Concurrent: Tests are given/measured about same time

Predictive: Predictor is done long after criterion is given (ex. SAT > College GPA.

Front 17

Review:

Standard Error of Mean

Standard Error of the Measurement

Standard Error of Estimate

Back 17

Error in group mean vs. population mean

error in scores based on given test

Error in predictive power of a criterion

Front 18

Standard Error of Estimate formula

Back 18

Sest = SDy x sq root of 1-rxy squared

Front 19

Standard Error of Estimate range

Back 19

0 - SD of criterion

Front 20

Mnemonic estimate goes w/Y (criterion) - repairs, cry about estimate. Y so High?

Measurement goes w/X

Back 20

that's it

Front 21

Criterion Related Validity Coefficient application:

Expectancy Table

Back 21

Table that shows probably a given score w/fall in a range of a predictor based on the criterion outcome.

Front 22

*Taylor Russell Tables
Base Rate, Selection Ratio, Incremental Validity

Back 22

table that outlines how much better hiring decisions will be when using a test vs. no test

Base Rate: Optimal: .5 (moderate, 50% ee's successful), Selection Ratio: Optimal: .1 (large pool, 10 applicants per job)

Incremental validity: Degree to which a new predictor will impact the critereon.

Front 23

*Taylor Russell Tables - Base Rate

Back 23

Rate of successful employees w/out using any test.

ex. 80% are turn out good ee's (base rate=.80)

Front 24

*Taylor Russell Tables - Selection Ration

Back 24

# of openings / # of applicants

ex. 1 opening, 10 applicants. 1/10, .10 selection ration. (low)

Front 25

Incremental Validity

Back 25

Amount of improvement when using predictor test vs. no test

Front 26

Taylor Russel Table
Incremental Validity Calculation

Back 26

Base rate (ex. .40) - 40% are good. W/Test, 65%.
Incremental Validity: .25 (amount of improvement)

Front 27

Taylor Russel Tables:
Factors affecting INcremental Validity

Back 27

1. Criterion Related Validity of instrument (rxy)
2. base rate
3. selection ratio

Front 28

How to optimize incremental validity

Back 28

*moderate base rate (.5)

*Low selection ration (.1)

Front 29

Decision Making Theory:
4 options in predictions

Back 29

true pos (predicted good, did good)
False pos (predicted good, did bad)
true neg (predicted bad, did bad)
false neg (predicted bad, did well)

Front 30

Decision Making Theory:
How do you de
crease false positives?

Back 30

increase predictor (1st choice, sometimes can't change criterion)

decrease criterion

Front 31

Developing a Predictor Test

Back 31

1. Conceptualization (objective, administrtion, etc)
2. Test construction. Item format, write items.

Front 32

Item difficulty

Back 32

want it between .3-.8 (30%-80% get it right)

Front 33

Item Characteristic Curve

Back 33

Shows degree to which item indicates testee has the trait being measured

Front 34

Item Response Theory

Back 34

Used to develop individually tailored, adaptive test. one answer determines following items.

Front 35

Test Revision

Back 35

Items are retained after validation, then cross validated. Results in SHRINKAGE (validity coefficient is smaller on new sample)

Front 36

Factors Affecting Validity Coefficient

Back 36

Range of Scores (want broad, unrestricted range)

Reliability of Predictor (caps validity)

Reliability of Predictor and Criterion (correction for attenuation)

Criterion Contamination: Y (criterion) is subjective scored, rater has knowledge of predictor. Inflates validity

Front 37

*Correction for Attenuation

Back 37

tells you how much more valid instrument would be if X or Y was perfect

Front 38

Construct Validity (trait)
2 subtypes

Back 38

Convergent (degree to which test aligns w/similar instruments - MonoTrait) - moderate/high correlation

Divergent (alignment with tests measuring different traits) - Heterotrait want low correlation

Front 39

Construct Validity - what table shows different validity types?

Back 39

Multi-trait/Multi method matrix