Educational Research Chapter 5: Selecting Measuring Instruments

Front 1

Chapter 5: Selecting Measuring Instruments
1.1 Define the terms construct, variable, and operationalized variable.

Back 1

Constructs – abstractions that cannot be observed directly but are helpful when trying to explain behavior: Intelligence, Teacher effectiveness, Self concept

Operational definition – the ways by which constructs are observed and measured: Weschler IQ test; Virgilio Teacher Effectiveness Inventory; Tennessee Self-Concept Scale

Variable – a construct that has been operationalized and has two or more values

Front 2

Chapter 5: Selecting Measuring Instruments
1.2 Describe the relationships among a construct, variable, and operationalized variable.

Back 2

When constructs are operationally defined, they become variables

Front 3

Chapter 5: Selecting Measuring Instruments
2.1 Identify the unique characteristics of each of the four (4) measurement scales.

Back 3

Measurement scales:
Nominal – categories
Gender, ethnicity, etc.
Ordinal – ordered categories
Rank in class, order of finish, etc.
Interval – equal intervals
Test scores, attitude scores, etc.
Ratio – absolute zero
Time, height, weight, etc.

Front 4

Chapter 5: Selecting Measuring Instruments
2.2 Differentiate categorical from quantitative variables.

Back 4

Types of variables:Categorical or Quantitative
1. Categorical variables reflect nominal scales and measure the presence of different qualities (e.g., gender, ethnicity, etc.)
2. Quantitative variables reflect ordinal, interval, or ratio scales and measure different quantities of a variable (e.g., test scores, self-esteem scores, etc.)

Front 5

Chapter 5: Selecting Measuring Instruments
2.3 Differentiate an independent variable from a dependent variable.

Back 5

Types of variables: Independent or dependent
1. Independent variables are purported causes
2. Dependent variables are purported effects
Two instructional strategies, co-operative groups and traditional lectures, were used during a three week social studies unit. Students’ exam scores were analyzed for differences between the groups.
The independent variable is the instructional approach (of which there are two levels)
The dependent variable is the students’ achievement

Front 6

Chapter 5: Selecting Measuring Instruments
3.1 Define the term test and describe the unique characteristics of a standardized test.

Back 6

Important terms
Instrument – a tool used to collect data
Test – a formal, systematic procedure for gathering information

Standardized tests – tests that are administered, scored, and interpreted in a consistent manner

Front 7

Chapter 5: Selecting Measuring Instruments
3.2 Describe the differences between the terms test, assessment, and measurement.

Back 7

1. Test – a formal, systematic procedure for gathering information
2. Assessment – the general process of collecting, synthesizing, and interpreting information
3. Measurement – the process of quantifying or scoring a subject’s performance

Front 8

Chapter 5: Selecting Measuring Instruments
3.3 Describe the difference between selection and supply response formats.

Back 8

Selected response item format – respondents select answers from a set of alternatives
Multiple choice
True-false
Matching
Supply response item format – respondents construct answers
Short answer
Completion
Essay

Front 9

Chapter 5: Selecting Measuring Instruments
3.4 Discuss what is meant by a raw score and describe three statistical transformations of raw scores.

Back 9

Interpreting data
Raw scores – the actual score made on a test
Standard scores – statistical transformations of raw scores
-Percentiles (0.00 – 99.9)
-Stanines (1 – 9)
-Normal Curve Equivalents (0.00 – 99.99)

Front 10

Chapter 5: Selecting Measuring Instruments
3.5 Describe the differences in the interpretations of norm-referenced and criterion-referenced scores.

Back 10

Norm-referenced – scores are interpreted relative to the scores of others taking the test
Criterion-referenced – scores are interpreted relative to a predetermined level of performance
Self-referenced – scores are interpreted relative to changes over time

Front 11

Chapter 5: Selecting Measuring Instruments
3.6 Describe the unique characteristics of a performance assessment.

Back 11

Performance assessments – assessments that focus on processes or products that have been created

Front 12

Chapter 5: Selecting Measuring Instruments
4.1 Describe the differences between cognitive and affective tests.

Back 12

Types of instruments
Cognitive – measuring intellectual processes such as thinking, memorizing, problem solving, analyzing, or reasoning
Affective – assessing individuals’ feelings, values, attitudes, beliefs, etc.

Front 13

Chapter 5: Selecting Measuring Instruments
4.2 Describe the unique characteristics of achievement and aptitude tests.

Back 13

Types of instruments
Achievement – measuring what students already know
Aptitude – measuring general mental ability, usually for predicting future performance

Front 14

Chapter 5: Selecting Measuring Instruments
4.3 Discuss what is meant by the term self-report response and identify three problems associated with such responses.

Back 14

1) The researcher can never be sure that individuals are expressing thier true attitude, interest, values, or personality.
2) Response set-the tendency of an individual to continually respond in a particular way to a variety of instruments.
3) Bias-distortion of research data that renders the data suspect or invalid due to ethnicity, race, gender, language, or religious orientation.

Front 15

Chapter 5: Selecting Measuring Instruments
4.4 Define the term bias.

Back 15

Bias – distortions of a respondent’s performance or responses based on ethnicity, race, gender, language, etc.

Front 16

Chapter 5: Selecting Measuring Instruments
4.5 Describe the unique purposes of attitude, interest, values, and personality tests.

Back 16

Typical affective characteristics of interest
Values – deeply held beliefs about ideas, persons, or objects
Attitudes – dispositions that are favorable or unfavorable toward things
Interests – inclinations to seek out or participate in particular activities, objects, ideas, etc.
Personality – characteristics that represent a person’s typical behaviors

Front 17

Chapter 5: Selecting Measuring Instruments
4.6 Describe the difference between a non-projective test and a projective test.

Back 17

Nonprojective instruments include measures of attitude and interest.
Projective tests present an ambiguous situation and require the test take to "project" her or his ture feelings on the amiguous situation. Association and is exemplified by the inkbolt test. Only the specially trained can administer and interpret projective tests.

Front 18

Chapter 5: Selecting Measuring Instruments
5.1 Describe the unique characteristics of Likert, semantic differential, rating, Thurstone, and Guttman scales.

Back 18

Scales used for responding to items on affective tests
Likert: Positive or negative statements to which subjects respond on scales such as strongly disagree, disagree, neutral, agree, or strongly agree
Semantic differential: Bipolar adjectives (i.e., two opposite adjectives) with a scale between each adjective
Dislike: ___ ___ ___ ___ ___ :Like
Rating scales – rankings based on how a subject would rate the trait of interest
Thurstone – statements related to the trait of interest to which subjects agree or disagree
Guttman – statements representing a uni-dimensional trait

Front 19

Chapter 5: Selecting Measuring Instruments
6.1 Identify two (2) technical characteristics of tests that should be considered when using an instrument.

Back 19

Two concerns
Validity
Reliability

Front 20

Chapter 5: Selecting Measuring Instruments
7.1 Define validity and identify four (4) ways by which it can be estimated.

Back 20

Validity – extent to which interpretations made from a test score are appropriate
Four types
1. Content – to what extent does the test measure what it is supposed to measure a) Item validity, b) Sampling validity, c) Determined by expert judgment

Criterion-related: a) Predictive – to what extent does the test predict a future performance; b) Concurrent - to what extent does the test predict a performance measured at the same time; c) Estimated by correlations between two tests

3. Construct – the extent to which a test measures the construct it represents: a) Underlying difficulty defining constructs; b) Estimated in many ways

4. Consequential – to what extent are the consequences that occur from the test harmful: a) Estimated by empirical and expert judgment;

Front 21

Chapter 5: Selecting Measuring Instruments
7.2 Define content validity, discuss the importance of item validity and sampling validity to it, and identify ways by which it can be estimated.

Back 21

Content validity is the degree to which a test measures an intended content area. Content validity requires both item validity and sampling validity. Item validity is concerned with whether the test items are relevant to the measurement of the intended content area. Sampling validity is concerned with how well the test samples the total content are being tested.

Front 22

Chapter 5: Selecting Measuring Instruments
7.3 Define criterion-related validity, differentiate between predictive and concurrent validity, and identify ways by which each is estimated.

Back 22

Criterion-related validity is determined by relating performance on a test to performance on a second test or other measure. The second test or measure is the criterion against which the validity of the initial test is judges. Criterion-related validity has two forms: concurrent validity and predictive validity.
Concurrent validity is the degree to which scores on one test are related to scores on a similar, preexisting test administered in the same time frame or to some other valid measure available at the same time.
Predictive validity is the degree to which a test can predict how well an individual will do in a future situation.
Concurrent and predicitve validity are determined by correlating one test with another test or measure, either at the same time or in the future.

Front 23

Chapter 5: Selecting Measuring Instruments
7.4 Define construct validity and discuss ways by which it can be estimated.

Back 23

Construct validity is determined by a series of validation studies that can include content and criterion-related approaches. Both confirmatory and disconfimatory evidence are used to determine construct validation.
The validity of any test or measure can be diminished by such factors as unclear test directions, inappropriate teaching, subjective scoring, and failing to follow administration procedures.

Front 24

Chapter 5: Selecting Measuring Instruments
7.5 Define consequential validity and discuss ways in which it is investigated.

Back 24

Consequential validity is the extent to which an instrument creates harmful effects for the user. Examing consequential validty allows researchers to ferret out and identify tests that may be harmful to students, teachers, and other test users, whether the problem is intended or not.

Front 25

Chapter 5: Selecting Measuring Instruments
7.6 Describe the appropriate manner in which the estimates of test validity should be interpreted.

Back 25

Validity is the degree to which a test measures what it is supposed to measure, thus permitting appropriate interpretations of test scores. A test is not valid per se; it is valid for a partucular interpretation and for a particular group. Each intended test use requires its own validation. Tests are not simply valid or invalid; they are highly valid, moderately valid, or generally invalid. The four main forms of validity are content, criterion-related, construct, and sonsequential. They are viewed as interrelated, not independent aspects of valitity.

Front 26

Chapter 5: Selecting Measuring Instruments
7.7 Identify four (4) factors that contribute to reduced validity.

Back 26

Factors affecting validity
- Unclear test directions
- Confusing and ambiguous test items
- Vocabulary that is too difficult for test takers
- Technical Issues
- Overly difficult and complex sentence structure
- Inconsistent and subjective scoring
- Untaught items
- Failure to follow standardized administration procedures
- Cheating by the participants or someone teaching to the test items

Front 27

Chapter 5: Selecting Measuring Instruments
8.1 Define reliability and identify five (5) approaches to estimating it.

Back 27

Reliability – the degree to which a test consistently measures whatever it is measuring

Six reliability coefficients
1. Stability – consistency over time with the same instrument
2. Equivalence – consistency with two parallel tests administered at the same time
3. Equivalence and stability – consistency over time with parallel forms of the test
4. Internal consistency – artificially splitting the test into halves
5. Scorer/rater – consistency of observations between raters
6. Standard error of measurement (SEM) – an estimate of how much difference there is between a person’s obtained score and his or her true score

Front 28

Chapter 5: Selecting Measuring Instruments
8.2 Define stability and describe the way in which it is estimated.

Back 28

Stability – consistency over time with the same instrument
Test – retest
Estimated by a correlation between the two administrations of the same test

Front 29

Chapter 5: Selecting Measuring Instruments
8.3 Define equivalence and describe the way in which it is estimated.

Back 29

Equivalence – consistency with two parallel tests administered at the same time
- Parallel forms
- Estimated by a correlation between the parallel tests

Front 30

Chapter 5: Selecting Measuring Instruments
8.4 Define equivalence and stability and describe the way in which it is estimated.

Back 30

Equivalence and stability – consistency over time with parallel forms of the test
- Combines attributes of stability and equivalence
- Estimated by a correlation between the parallel forms

Front 31

Chapter 5: Selecting Measuring Instruments
8.5 Define internal consistency, identify three (3) ways by which it is estimated, and describe the unique characteristics of each estimate.

Back 31

Internal consistency – deals with the reliability of a single test taken at one time. It measures the extent to which the items in the test are sonsistent among themselves and with the test as a whole.
Split-half: determined by dividing a test into two equivalent halves, and using the Spearman-Brown formula to determine the reliability of the whole test.
Kuder-Richardson 20 and 21: deals with the internal consistency of tests that are scored dichotomosuly (right, wrong).
Cronbach's alpha: deals with the internal consistency of tests that are scored with more than two choices (agree, neutral, diagree or 0, 1, 2, 3)
Several coefficients – split halves, KR 20, KR 21, Cronbach alpha
All coefficients provide estimates ranging from 0 to 1

Front 32

Chapter 5: Selecting Measuring Instruments
8.6 Define scorer/rater reliability, discuss the difference between inter-judge reliability and intra-judge reliability, and identify the ways by which each is estimated.

Back 32

Scorer/rater – consistency of observations between raters
- Inter-judge – two observers
- Intra-judge – one judge over two occasions
- Estimated by percent agreement between observations

Front 33

Chapter 5: Selecting Measuring Instruments
8.7 Discuss the factors involved in the interpretation of a reliability coefficient.

Back 33

What constitutes an acceptable level of reliability differs among test types, with standardized achievement tests having very high reliabilities and projective tests having considerably lower realiabilities. If a test is composed of several subtests that will be used individually in a study, the reliabilty of each subtest should be determined and reported.

Front 34

Chapter 5: Selecting Measuring Instruments
9.1 Define standard error of measure and discuss its interpretation.

Back 34

Standard error of measurement (SEM) – an estimate of how much difference there is between a person’s obtained score and his or her true score. A small standard error of measurement indicates high reliability; a large standard error of measurement, low reliability.
The standard error of measurement allows us to estimate how much difference there is between a person's obtained and true score. Big differences indicate low reliability.

Front 35

Chapter 5: Selecting Measuring Instruments
10.1 Identify five (5) sources of information on published tests.

Back 35

Selection of a Test: Sources of test information
Mental Measurement Yearbooks (MMY)
Tests in Print
ETS Test Collection
Professional journals
Test publishers and distributors

Front 36

Chapter 5: Selecting Measuring Instruments
11.1 Identify three (3) psychometric factors and two (2) non-psychometric factors that should be considered when selecting a test.

Back 36

Issues to consider when selecting tests
Psychometric properties
- Validity
- Reliability
- Length of test
- Scoring and score interpretation

Non-psychometric issues
- Cost
- Administrative time
- Objections to content by parents or others
- Duplication of testing

Front 37

Chapter 5: Selecting Measuring Instruments
11.2 Describe two (2) of the authors' recommendations for constructing your own paper and pencil test.

Back 37

Designing your own tests
Get help from others with experience in developing tests
Item writing guidelines
Avoid ambiguous and confusing wording and sentence structure
Use appropriate vocabulary
Write items that have only one correct answer
Give information about the nature of the desired answer
Do not provide clues to the correct answer
See Writing Multiple Choice Items

Front 38

Chapter 5: Selecting Measuring Instruments
11.3 Identify three (3) types of selection items and three (3) types of supply items.

Back 38

Selection Items
1) Multiple Choice
2) Matching
3) True-False

Supply Items
1) Essay
2) Completion
3) Short-answer

Front 39

Chapter 5: Selecting Measuring Instruments
11.4 Identify four (4) considerations when administering a test as a part of your study.

Back 39

Test administration guidelines
Plan ahead
Be certain that there is consistency across testing sessions
Be familiar with any and all procedures necessary to administer a test

Front 40

Chapter 5: Selecting Measuring Instruments
12.1 Discuss the unique characteristics of Mental Measurements Yearbooks, Tests in Print, ETS Test Collection Database, ERIC/AE Test Locator, professional journals, and test publishers as sources of test information.

Back 40

Mental Measurement Yearbooks (MMY)
-The reviews in MMY are most easily accessed through your university library and the services to which they subscribe (e.g., EBSCO)-Provides factual information on all known tests-Provides objective test reviews-Comprehensive bibliography for specific tests-Indices: titles, acronyms, subject, publishers, developers-Buros Institute

Tests in Print-Tests in Print is a subsidiary of the Buros Institute-The reviews in it are most easily accessed through your university library and the services to which they subscribe (e.g., EBSCO)-Bibliography of all known commercially produced tests currently available-Very useful to determine availability

ETS Test Collection-Published and unpublished tests
-Includes test title, author, publication date, target population, publisher, and description of purpose-Annotated bibliographies on achievement, aptitude, attitude and interests, personality, sensory motor, special populations, vocational/occupational, and miscellaneous

Professional journals
Test publishers and distributors