Group Wiki 2- Reliability Part 2

Front 1

What is interesting about methods used to calculate reliability?

Back 1

Each of the methods used to calculate reliability uses a slightly different definition of true score and error

Front 2

What is something to keep in mind about the reliability coefficient?

Back 2

Always keep in mind that the reliability coefficient is an estimate that can change depending on the method used to calculate it

Front 3

How do you choose a method to estimate the reliability?

Back 3

The method you choose to estimate the reliability should fit the way in which the test will be used.

Front 4

What is test/retest reliability used for?

Back 4

Utilized to measure the stability of scores over a period of time.

Front 5

How is test/retest reliability used? How does it work?

Back 5

-A test developer gives the same test to the same group of test takers at two different times.
-Scores on the 1st administration are compared to scores on the 2nd administration using correlational techniques.
-If individuals respond consistently from one administration to the next, the correlation between test scores will be high.

Front 6

What is one thing to remember about the interval between the administration of the two tests during test/retest reliability?

Back 6

The interval between the administration of the two tests can be several hours or several years.

Front 7

What are some common problems with test/retest reliabilty?

Back 7

-You assume that the test takers have not changed over time- but they have.

-The magnitude of the reliability of scores evidenced changes as the time between administrations changes

-Practice (carryover) effects could influence observations.
Due to attribution of error – error becomes more systematic

Front 8

What are Parallel forms of Reliability used for?

Back 8

To eliminate errors.

Front 9

Why do test makers practice parallel forms of reliability?

Back 9

The reason for this type of reliability estimation is to develop an alternate test form that is equivalent in: Content, Response processes, Statistical characteristics, in order to account for some error

Front 10

How do parallel forms of reliability work?

Back 10

You are looking to achieve a “true score” for both tests. This comes from the fact that the tests are parallel in content and processes. If this is done correctly, this will result in identical distributions of scores for the two tests.

Front 11

How do you eliminate practice effects and other problems with test-retest reliability?

Back 11

Test developers often give two highly similar forms of the test to the same people at almost the same time.
-To eliminate practice or transfer effects, half the participants take one form followed by the other. The other half take the forms in reverse order- this is known as counterbalancing

Front 12

What are some common problems with equivalence reliability?

Back 12

Common Problems:
•Order effects-What order are you going to give the tests?
•Questions Order- can cue test takers to answer questions on the alternate form in specific ways.
•Creation: How do you make parallel tests? How do you make sure the tests are parallel?

Front 13

What is something interesting about parallel testing forms?

Back 13

The parallel test is only used for reliability purposes, and then discarded.

Front 14

Internal consistancy?

Back 14

examines the degree to which the items on the test are all assessing the same thing.

Front 15

In parallel testing, how can you tell if a construct is consistent?

Back 15

Consistent constructs are assessed throughout the scale.

Front 16

What is an example of internal consistency?

Back 16

•Think of the tape measure:
-The first foot is the same length as the second foot.
-The fourth foot is the same length as the eleventh foot.
*The tape measure has internal consistency because measures each foot as the same distance therefore the same scale is established

Front 17

Internal Consistency is beneficial because:?

Back 17

-It eliminates the need to administer a test twice
-There is no need to go through the difficulty of creating a parallel test
-Eliminates carryover/practice effects.
-Eliminates changes that may occur in people over time

Front 18

How does split half reliability work?

Back 18

-Administer a test to a group of individuals.
-Split the test into two halves.
-Each half is an alternate form.
Correlate the scores on one half with those on the other half.
-This correlation can be used to estimate the reliability of the test.
-Longer tests have higher reliability

Front 19

How do you explain split half reliability?

Back 19

We must make an adjustment to the correlation to determine the reliability

Front 20

What do the variables stand for in the spearman-brown prophecy formula?

Back 20

k = ratio by which your test length changes after splitting
(full # items you want / # items on which correlation was determined)
r = correlation between parts of the test

Front 21

What does the Spearman-Brown prophency formula do?

Back 21

-This formula adjusts the correlation based upon the number of items in the “new” split tests and the number of items in the original test.
-You can also use this formula to predict what the reliability would be if you wanted to change the number of items on the test.

Front 22

What are some common problems with split half- reliability?

Back 22

-We are not assessing reliability of the test scores, but reliability scores on half the test, and then estimating.
-It is assumed reliability will always increase given more items.-This is only true theoretically.
-How do you split the test?Which items?
-Are the items in the splits homogeneous or heterogeneous?
-Some “splits” give a much higher correlation than other “splits.”

Front 23

What is Cronbach alpha (sometimes called coefficient alpha) used for?

Back 23

Used for polytomously scored items

Front 24

What is Kuder-Richardson 20 (KR-20) used for?

Back 24

Used for dichotomously scored items

Front 25

What is Kuder-Richardson 21 (KR-21) used for?

Back 25

Lower bound of KR-20
Used when item statistics are unknown.

Front 26

What were Cronbach Alpha Family of Internal Consistency Reliability Coefficients created for?

Back 26

In order to get around problems with split-half, a new family of reliability coefficients were developed.

Instead of splitting a test in half, these methods compare every item to every other item.

Front 27

What is good reliability according to (Nunnaly & Bernstein, 1994)?

Back 27

.70 is acceptable for early stages of research.
Basic research generally requires .80
When important decisions are to be made, you should have a reliability coefficient of at least .90.

Front 28

What is the most important thing to remember when interpreting reliability?

Back 28

These are general guidelines. There are no gold-standards for interpreting reliability.

Front 29

When is lower reliability okay?

Back 29

-You are making preliminary decisions
-You are sorting people into groups
-You are conducting some research
-You are making a classroom test
--Classroom tests generally test diverse content.

Front 30

What are some factors that influence reliability?

Back 30

-Length of test
-Pressure of time limit
-Heterogeneity of the group taking the test
-Level of difficulty of the tasks required by the test

Front 31

Why do we use standard error of measurement?

Back 31

To better understand what reliability is we often use another statistic in conjunction with reliability.

Front 32

What is the standard error of measurement?

Back 32

-SEM is an index of the amount of inconsistency (or amount of expected error) in an individual’s score.
-Acts as another measure of variability in scores.

Front 33

What is a confidence interval?

Back 33

Range of scores that likely includes the true score

Front 34

How can you affect a confidence interval?

Back 34

-SEM can be used to construct a confidence interval (CI).
-CI can vary according to how confident you want to be.