Group Wiki 2-Validity Part 2

Front 1

What are some important questions to ask when thinking of validity?

Back 1

-What meaning can be drawn?
-What uses of the test are appropriate?
-What is validated are the inferences derived from the test scores and the actions that follow from those inferences.

Front 2

Why do we test for validity?

Back 2

-Because we test for a purpose, it is that purpose for which we must validate, not just overall for a test.
-Scores are always in context, and so must validation be in context.

Front 3

How do we validate a test?

Back 3

To validate, we make the most reasonable case we can to guide current use of the test and research into its score meaning.

Front 4

According to Messick...

Back 4

-Validity is a unified concept about which we provide a number of different pieces of evidence.
-Additionally, all validation of the interpretation and use of test scores consists of various aspects of the construct the test is supposed to measure.
-The construct is KEY!

Front 5

According to Messick... the value implications are?

Back 5

implicit in testing and of much importance.

Front 6

In value implications... what are some sources of values?

Back 6

-Construct labels
-Broad theories
-Ideologies about humans and society

Front 7

What is consequential validity?

Back 7

The values inherent in testing result in consequences, both intended and unintended.

Front 8

What is consequential validity evidence?

Back 8

Consequential validity evidence: Refers to the social consequences of using a test for a specific purpose

Front 9

What is something that is important to remember about negative consequences?

Back 9

Negative consequences do not necessarily signify a problem with the validity of the interpretation or use of the test.

Front 10

What are some different aspects of construct validity?

Back 10

-Content
-Substantive
-Structural
-Generalizable
-External
-Consequential

Front 11

What is content domain?

Back 11

The extent to which the test content represents an appropriate sample of the skills and knowledge that are the goals of instruction.

Front 12

What is substantive?

Back 12

Refers to the cognitive processes that underlie student performance and correlations across items. Models of the cognitive processes that underlie the tasks are needed.

Front 13

What are some important things to know about substantive?

Back 13

-Tests need to assess processes in addition to content.
-In other words, does a test item actually measure what it is supposed to measure in the way that the content exists in the domain, using the processes that occur in the domain.

Front 14

What does structure refer to?

Back 14

-The degree to which the score scales are consistent with the structure of the domain being measured.
-The construct domain should guide the creation of tasks
-The construct should also guide the scoring criteria.

Front 15

What does generalizability question?

Back 15

-Are score interpretations consistent across assessment tasks, groups of people, different times, settings, etc.?
-Does the assessment provide representative coverage of the content and processes such that the score represents the construct of interest?
-The assessment should provide representative coverage of the domain so that the score can be interpreted as assessing participants’ knowledge of the domain, not their knowledge of the domain under a bunch of extraneous conditions.
-Transfer – consistency of performance across tasks that are representative of the broader construct. Range of tasks that performance predicts.

Front 16

What are external factors?

Back 16

-The extent to which performance on the test relates to other variables.The relations should be predicted by theory.
-Convergent evidence
-Discriminant evidence

Front 17

What are some consquential aspects of validty?

Back 17

-Evidence and rationales for evaluating the intended and unintended consequences of use of the test.
-Adverse consequences could come from the test users and not the test makers.
-This could also come as a result of construct underrepresentation or irrelevance. Validity is compromised if as a result of these problems, participants are to show their knowledge, and consequences of that ensue.
-Fairness of test use is an issue. So are negative consequences on teaching and learning.
-It is important to note that not all negative consequences are unintended and make the use of the test invalid. Negative consequences that matter are associated with bias in scoring, interpretation, and use of the test. They may also be associated with differential value systems

Front 18

What is comparable validity?

Back 18

-Comparable validity – the test is only fair if it yields comparably valid scores from person to person, group to group, etc.
-Overestimating or underestimating the skills of one group
-Measuring different constructs in different groups when claiming to measure the same constructs.

Front 19

What are some aspects of testing fairness?

Back 19

-Absence of bias
-Equitable treatment of all examinees
-Opportunitiy to learn
-Equality of outcomes

Front 20

What are some questions to keep in mind about bias?

Back 20

Are the differences between groups due to bias in the test, or to true differences in the groups? Not all group differences are the result of test bias

Front 21

When does bias occur?

Back 21

Bias occurs when deficiencies are inherent in the test that give rise to different meanings for test scores for different groups of people.

Front 22

What is equitable treatment in testing?

Back 22

Everyone has the same opportunities to demonstrate knowledge of the constructs on the test.

Front 23

Opportunity to learn?

Back 23

Opportunity to learn:
-If not all students have been afforded the opportunity to learn the material on the test, there is inherent unfairness in the interpretation and use of the test.
-This is especially important when the test will be used for high-stakes purposes.

Front 24

Equality of Outcomes?

Back 24

Equality of outcomes:
-A test should not systematically under-represent the performance of any one group.
-There is no basis for stating that pass rates should be identical for different groups because there may actually be differences in the performances of some groups over others, but the test shouldn’t be the cause.

Front 25

Inequality V. Equity?

Back 25

-There may actually be differences in the groups (inequality), but that doesn’t mean the groups shouldn’t be treated fairly (inequity).
-This doesn’t mean that all groups should be treated the same, but means that all groups should be treated in ways appropriate to their characteristics and needs.
-Task conditions do not need to be exactly the same for evidence of comparable validity, but task conditions for all groups should represent the same construct-relevant conditions. Also, you must justify that scores obtained for the two groups have the same meaning.