Aatbs Review - Ctw Test Construction-

Front 1

Kappa Statistic - Think of...

Back 1

Inter-Rater Reliability

Front 2

Item Difficulty:

Back 2

# got answer correct/# who answered question

.50 is ideal, .75 ideal for t/f

Front 3

"D" Item Discrimination:

Back 3

From -1.0 to 1.0, ability to discriminate between high and low.
High group gets them approach 1.0, low group gets them: approach -1.0

Front 4

True +
False +

True -
False -

Back 4

T+ = Hi Predict, Hi Crit
F+ = Hi Predict, Low Crit

T- = Low Predict, Low Crit
F- = low predict, low critertion

Criterion: True/False
Predictor: Positive/negative

Front 5

Examples of Norm Referenced:

Back 5

Percentile

Z-Score (=0 @1)

T-Score (=50 @ 10)

Front 6

Relevance:

Back 6

Extent that test items contribute to goals of testing.

Front 7

Reliability:

Ways to get it:

Back 7

AKA Consistency:

Test-Retest
Alternate Forms (ie, A & B form WJ3)
Split-Half (Spearman/Brown)
Coefficient Alpha (KR-20)
Inter-Rater (Kappa Stat)

Front 8

Std Error Measurement:

Back 8

For measurement error:
CI around the test score.

Front 9

Std Error Estimate:

Back 9

Error for predicting criterion from predictor.

CI around criterion.

Front 10

2 Types of Construct Validity:

Back 10

CV: Extent test measure the hypothetical construct (ie, intelligence, etc etc)

Convergent: Correlates with other measures of same construct.

Discriminant: Correlates with other measures that dont correlate.

Front 11

Criterion Referenced Interpretation:
Examples:

Back 11

Interpret scores by pre-specified std.

% correct
Regression Equation
Expectancy Table

Front 12

Content Validity:

Back 12

Extent test samples domain of info/knowledge/skill it claims to.

Determined by expert judgment.
Important for achievement/ job sample.

CF: Construct validity.

Front 13

Classical Testing Theory:

Back 13

Variability reflects combo of TRUE SCORE DIFFERENCES and EFFECTS of ERROR (measurement, etc)

Thus: Reliablity = Measurement of true score. (80% true score, 20% error)

Front 14

For test item with discrimination index (D) of +1:

a. high achievers more correct than low
b. low more correct than high
c. low and high to be equally correct
d. low and high to be equally incorrect.

Back 14

High get more (all) correct.

Front 15

Internal Consistency on Dichotomous:
a Spearman-Brown
b. Kappa Stat
c. KR-20
d. coefficient of concordance

Back 15

KR-20: Best for dichotomous (t/f, right/wrong, etc)

Front 16

to make a coefficient alpha stronger:
a. add more similar items
b. add more heterogenuous items
c. use true/false
d. all of the above

Back 16

Add more similar to increase alpha coefficient: IE: ask the same question more, ask more q's about US history and you'll get a more accurate test of knowledge, than a ten question test.

Front 17

Multimodal Multimethod Matrix: For validity you want:
a. MM low and HH high
b MH high, HM low
c. MM high, HM low
d. MM High, HH low

Back 17

MH High, HM Low

Front 18

Preventing Criterion Contamination:
a. keep raters independent from each other
b. make sure they dont have predictor's scores
c. make them aware of possible biases through training
d.

Back 18

Make sure they dont have predictors scores

Front 19

Oblique Rotation for FA if:
a. assessing construct validity of single trait test
b. if he believe constructs in the analysis are correlated
c. determining factorial validity
d. determining reliability

Back 19

Oblique: looking for correlations between constructs measured by the test.

Front 20

Orthogonal
vs
Oblique

Back 20

Orthogonal: uncorrelated, independent

Oblique: Correlated

Front 21

Shared variability

Back 21

calculated by squaring factor load