Chapter 7: Item Analysis

Front 1

(2) are dependent on the quality of the items

Back 1

reliability and validity

Front 2

(2) approaches to item analysis

Back 2

qualitative
quantitative

Front 3

item difficulty index(p) =
only applicable to _______tests
ranges from ____to ___
easier items have a ____decimal number
harder items have a ____decimal number

Back 3

# of examinees correctly answering the item/#examinees
max performance
0-1
larger
smaller

Front 4

T or F
Items with p values of either 0.0 or 1.0 provide no information about individual differences and are of no use from a measurement perspective

Back 4

T

Front 5

Item difficulty is dependent on the ___

Back 5

sample

Front 6

For maximizing variability and reliability, the optimal item difficulty is ____

Back 6

0.50
*Not necessary for ALL items to have a difficulty level of 0.50, Often desirable to select some items with a difficulty level below and above 0.50, but with a mean of 0.50

Front 7

-T or F Different levels are desirable in different testing situations
-constructed response items, ___is typically the optimal level? selected response items?

Back 7

T
0.50 ; optimal level varies due to influence of guessing

Front 8

-in criterion reference tests the expectation is that most test takers will..
-common for items to have p values as high as ___

Back 8

eventually be successful
0.90

Front 9

Percent Endorsement Statistic=
for ____tests
dependent on the ____

Back 9

-percentage of examinees that responded to an item in given manner
-typical response tests
-sample

Front 10

Item Discrimination=

Back 10

how well an item differentiates among test takers who differ on the construct being measured.

Front 11

more than than ___ different indexes of item discrimination have been developed

Back 11

50

Front 12

-Discrimination Index (D)
-two groups are typically defined in terms of___performance
-common approach used?

Back 12

-difference in performance between two groups
-total test
-select the top and bottom 27% of test takers in terms of
their overall performance on the test, and exclude the middle 46%.

Front 13

calculation of D

Back 13

-difficulty of the item is computed for each group separately, and these are labeled pT and pB (T for top, B for bottom)
-D = pT − pB

Front 14

example:
pT = 0.80 means that
pB = 0.30 means that

Back 14

-80% of the examinees in the top group answered the item correctly
-30% of the examinees in the bottom group answered the item correctly

Front 15

Items with D values over ___ are acceptable
Items with D values below should be reviewed
**these are general rules there are exceptions

Back 15

-0.30
*larger is better
-0.30

Front 16

Most indexes of item discrimination are biased in favor of items with ______difficulty levels

Back 16

intermediate

Front 17

Items that all examinees either pass or fail (i.e., p values of either 0.00 or 1.0) do not provide any information and their D values will always be _____

Back 17

zero

Front 18

If half of the examinees correctly answered an item and half failed (i.e., p value of 0.50) then it is possible for the
item’s D value to be ___

Back 18

1.0
*This does not mean that all items with p values of 0.50 will have D values of 1.0; just that the item can conceivably have a D value of 1.0

Front 19

As a result the relationship between p
and D, items that have excellent discrimination power (i.e., D values of 0.40 and above) will necessarily have p
values between___and___

Back 19

0.20 and 0.80

Front 20

In testing situations where it is desirable to have either very easy or very difficult items, D values can be expected to be ____than those normally desired.

Back 20

lower

Front 21

___D values often indicate problems, but these guidelines should be applied in a flexible manner.

Back 21

Low

Front 22

-The interpretation of discrimination indexes is also complicated on ___tests
-It is normal for traditional item discrimination indexes to __-estimate an item’s true measurement characteristics

Back 22

mastery
under

Front 23

approach for item discrimination on mastery tests
limitation of this approach

Back 23

-administer the test to two groups: one group that has
received instruction and one that has not.
D = p instruction − p no instruction
-limited due to difficulty locating group that has not received instruction on relevant material

Front 24

-another approach for item discrimination on mastery tests
-limitations

Back 24

-administering the test to the same sample twice, once before instruction and once after instruction.
D = p posttest − p pretest
-Requires that the test be used as both a pretest and posttest, may involve carryover effects

Front 25

-another approach for item discrimination on mastery tests

Back 25

-Use item difficulty values based on the test takers who reached the mastery cut-off score and those who did not reach mastery
D = pmastery − pnonmastery

Front 26

Item-Total Correlation Coefficients
Large item-total correlation suggests

Back 26

-Item discrimination can also be examined by correlating performance on the items (scored as either 0 or 1) with the total test score. calculated using the point-biserial correlation
-that an item is measuring the same construct as the overall test measures

Front 27

Item Difficulty and Discrimination on Speed Tests:
-Item performance depends largely on the
-Measures of item difficulty and discrimination will reflect _______________, rather than the item’s actual difficulty level or ability to discriminate.

Back 27

speed of performance
-the location of the item in the test

Front 28

Allows you to examine how many examinees in the top and bottom groups selected each option on a multiple choice item.

Back 28

distractor analysis

Front 29

effective distractor attracts more examinees in the ___group and demonstrates ____discrimination

Back 29

bottom
negative

Front 30

qualitative item analysis (4 tips)

Back 30

-Set the test aside and review a few days later
-Have a colleague review the test
-Have examinees provide feedback after taking the test
-use both quantitative and qualitative approaches

Front 31

Theory of mental measurement that holds that the responses to items are accounted for by latent traits

Back 31

Item Response Theory

Front 32

_____is an ability or characteristic that is inferred based on theories of behavior, as well as empirical evidence, but cannot be assessed directly.

Back 32

latent trait

Front 33

Central to IRT is a complex mathematical model that describes

Back 33

how examinees at different levels of ability will respond to individual test items.

Front 34

Item Characteristic Curves (ICC):
graph with _____reflected on the horizontal axis and the ________ reflected on the vertical axis

Back 34

ability
probability of a correct response

Front 35

T or F
Each item has its own specific ICC

Back 35

T

Front 36

ICCs incorporate information about(2)

Back 36

item’s difficulty and discrimination ability

Front 37

ICC:
point halfway between the lower and upper asymptotes is referred to as the ____

Back 37

inflection point

Front 38

inflection point represents what?

Back 38

difficulty of the item (b parameter)

Front 39

ICC:
Discrimination (i.e., the a parameter) is reflected by

Back 39

the slope of the ICC at the inflection point

Front 40

ICCs with ____slopes demonstrate better
discrimination than those with ___slopes

Back 40

steeper
gentler

Front 41

the "simplest model" also referred to as a one-parameter IRT model

Back 41

Rasch IRT Model

Front 42

Rasch IRT Model assumes that items differ in only one parameter, which parameter?

Back 42

difficulty (b parameter)

Front 43

Two-Parameter IRT Model assumes that items differ in both (2)

Back 43

difficulty and discrimination

Front 44

which one, one parameter IRT or two-parameter IRT model, better reflects real life test development applications

Back 44

Two-Parameter IRT Model

Front 45

Three-parameter model assumes that

Back 45

even if the respondent essentially has no “ability,” there is still a chance he or she may answer the item correctly simply by chance

Front 46

In both the one- and two-parameter IRT models, the ICCs asymptote toward a probability of ___
this assumes that essentially a ____percent possibility of answering the items by chance

Back 46

zero
zero

Front 47

Item difficulty (i.e., p) and item discrimination (i.e., D) are based on _____theory

Back 47

classical test (meaning both are sample dependent)

Front 48

In IRT, the parameters of items (e.g., difficulty and discrimination) are sample-_____

Back 48

free/independent

Front 49

(4) Special Applications of IRT

Back 49

• Computer Adaptive Testing (CAT)
• Scores based on IRT (see Chapter 3)
• Reliability (see Chapter 4)
• Detecting biased items