Psychometrics And Test Interpretation

Front 1

What is the mathematical expression of Classical Test Theory?

Back 1

Test Score X = obtained Score T + Random Error

Front 2

What is the Standard Error of Measurement?

Back 2

The standard deviation of errors around an individual OBTAINED SCORE around the true score.

Front 3

What is a con to having a highly reliable test?

Back 3

It reduces sensitivity because the distribution is too leptokurtic.

Front 4

What is the general family of statistical methods used in Inferential Statistics?

Back 4

GENERAL LINEAR MODEL

Front 5

What are examples of the General Linear Model tests?

Back 5

t-Test

ANOVA/ANCOVA

Regression

Factor Analysis

Cluster Analysis

Linear Discriminant Function

Structural Equation Modeling

Multidimensional Scaling

Front 6

What is the major advantage of Item Response Theory?

Back 6

Observed Scores and True Scores are not test dependent (as in classical test theory). Instead, scores are ITEM DEPENDENT.

In other words, IRT is focused on item-level characteristics.

Front 7

What is the simplest IRT model?

Back 7

The Rasch Model:

Because it is only concerned with item thresholds and uses Item Characteristic Curves to do so.

Front 8

How does an Item Characteristic Curve work?

Back 8

It compares the probability of a correct answer against an underlying trait or ability. This then allows items to be examined for discrimination and difficulty levels.

Front 9

What do you get when you combine the Item Characteristic Curves of a measure?

Back 9

A Test Characteristic Curve

Front 10

What does a Test Characteristic Curve provide?

Back 10

Direct analysis of underlying ability.

Front 11

What is probability?

Back 11

The ratio of outcomes over an infinite number of trials.

Front 12

What is Bayes' Theorem?

Back 12

Front 13

What is variance?

Back 13

Standard Deviation squared.

It is the average of the squared differences of each observation from the mean of a distribution.

Front 14

What is the classical definition on of reliability?

Back 14

The ratio of true variance to total variance.

It acts as a measure of how much a measure is affected by random error.

Front 15

Name the 5 main types of reliability

Back 15

Test-retest

Alternate forms

Split-half

Internal consistency

Interrater

Front 16

What is the purpose of test-retest reliability?

Back 16

TEMPORAL STABILITY. To account for error variance that could affect consistency of test performance.

There is always a trade off between sensitivity and test-retest reliability.

Front 17

What is the purpose of Alternate Forms reliability?

Back 17

CONSISTENCY ACROSS FORMS. To assess content sampling and minimize error variance due to practice effects and to capture stability over time.

Front 18

What is the purpose of split-half reliability?

Back 18

To evaluate the internal consistency (i.e., content sampling) of a test in a single administration.

Front 19

What is the Spearman-Brown formula used for?

Back 19

In split-half reliability measures, the formula is used to determine the effect of lengthening a measure.

LENGTHENING HELPS IMPROVE INTERNAL CONSISTENCY BUT NOT STABILITY (i.e., test-retest reliability).

Front 20

What is the purpose of establishing Internal Reliability?

Back 20

TO DETERMINE IF ITEMS OF A TEST ALL ASSESS THE SAME COGNITIVE DOMAIN.

Assesses two sources of error:

Content sampling

Heterogeneity of the domain being assessed

Front 21

Does item heterogeneity or homogeneity improve internal reliability?

Back 21

Homogeneity improves reliability, but for it to be useful, the homogeneity of a test has to match the homogeneity of the construct being measured.

Front 22

What tests are used to establish Internal reliability?

Back 22

Kuder Richardson 20 for dichotomous items

Cronbach's Alpha for Likert-type items

Front 23

Name the various types of Validity

Back 23

Content Validity
Criterion Validity (Predictive and Concurrent)
Construct Validity (Convergent and Discriminant)

Front 24

What is content validity?

Back 24

The degree to which a test measures all facets of a domain of interest.

Not expressed mathematically, but is established by an 'expert' panel.

Front 25

What is predictive validity?

Back 25

How well a test predicts a predetermined criterion

Front 26

What is concurrent validity?

Back 26

How well a test correlates with a current established criterion (i.e., other validated tests measuring the same construct)

Can be expressed as a coefficient. The math for predictive and concurrent validity is the same.

Front 27

What is construct validity?

Back 27

How well a test measures a theoretical construct or trait.

Front 28

How do content and construct validity differ?

Back 28

Content validity is concerned with how well items of a test sample the domain of interest.

Construct validity is concerned with how well those items actually measure the domain of interest.

Front 29

What are convergent and discriminant validity?

Back 29

Forms of construct validity. Convergent validity is different items or tests that are supposed to correlated do. Discriminant validity is the opposite.

Front 30

What tool is used to establish construct validity?

Back 30

Multi-trait Multi-Method Matrix

Front 31

Define positive predictive power

Back 31

The proportion of the time we are correct when we say a disease is present based on a positive test result.

PPP=True Positive/(True Positive + False Positive)

Row-based index:

PPP = A/(A+B)

Front 32

Define negative predictive power

Back 32

The proportion of the time we are correct when we say someone does not have a disease based on a negative test result.

NPP= True Neg./(True Neg. + False Neg.)

Row-based index:

NPP= D/(D+C)

Front 33

Are positive and negative predictive power affected by the base rate & prevalence of a condition?

Back 33

YES!

Unlike Sensitivity and Specificity, which are fixed properties of a test.

Front 34

Define Sensitivity

Back 34

The number of people who DO have a disease that are correctly classified as having the disease by a test.

MINIMIZES FALSE NEGATIVES but may lead to more false positives

Front 35

Define Specificity

Back 35

The number of people who do NOT have a disease that are correctly classified as not having the disease by a test.

MINIMIZES FALSE POSITIVES but may lead to more false negatives

Front 36

Define Positive Liklihood Ratio

Back 36

A combination of sensitivity and specificity into a single index that indicates the ODDS THAT A POSITIVE TEST RESULT MEANS A PERSON HAS THE DISEASE

For example, if the ratio is 3.0, then it means the test result is three times more likely to be because the person has the disease than from someone who does not have the disease.

Column-based index

Front 37

Define Negative Liklihood Ratio

Back 37

A combination of sensitivity and specificity into a single index that indicates the ODDS THAT A NEGATIVE TEST RESULT MEANS A PERSON DOES NOT HAVE THE DISEASE.

For example, if the ratio is 3.0, then it means a the test result is three times more likely to have come from someone without the condition than someone with the condition.

Column-based index

Front 38

What happens if +/- Liklihood Ratios approach a value of 1?

Back 38

Then your test is no better than random guessing.

Front 39

Which are characteristics of the test itself: Sensitivity, Specificity, Liklihood Ratios, and/or Predictve Powers?

Back 39

Sensitivity, Specificity, and Liklihood Ratios

Predictive powers are NOT, but rather are properties of specific test scores in specific contexts.

Front 40

Does good Sensitivity and Specificity guarantee a clinically useful test?

Back 40

No! No! No! NEVER! No! Not. Ever. No! No!

This is because they are developed at the group level, and we are concerned with how tests apply to INDIVIDUALS.

Front 41

What classification method is used to classify INDIVIDUALS rather than groups?

Back 41

Positive and Negative Predictive Power

Front 42

What value do positive and negative predictive power need to exceed in order for a test to be useful?

Back 42

.50

.50 is chance level; Less than .50 means you will be wrong more than you are correct.

Front 43

Does positive predictive power increase or decrease with increased prevalence of a disease?

Back 43

INCREASE.

Negative predictive power will simultaneously decrease.

Front 44

What is the mathematical expression of prevalence?

Back 44

Prevalence = (True Pos. + False Neg.)/N

Front 45

What is the danger of using data from test validity studies to calculate PPP and NPP?

Back 45

Because the prevalence of diseases in validity studies is often 50% (artificially high) in order to facilitate other statistical comparisons (e.g., case vs. control analyses)

OR!

50% may be artificially low, and lead to false negatives, if you are in a setting where 100% of your sample has a condition of interest (e.g., 100% of patients in a severe TBI clinic have a TBI)

Front 46

What does the area under a ROC curve represent?

Back 46

Overall accuracy of a test (i.e., correctly classified portion of a sample).

Front 47

What is plotted on the y-axis of a ROC curve? The x-axis?

Back 47

y-Axis is True Positive Probability
x-Axis is False Positive Probability

Front 48

What does the slope of any point of a ROC curve represent?

Back 48

The POSITIVE LIKLIHOOD RATIO

Front 49

Do reliability coefficients need to be squared in order to determine variance?

Back 49

NO! The coefficient is the variance.

Only correlation coefficients need to be squared in order to obtain variance.

Front 50

What are the minimal acceptable standards for reliability coefficients?

Back 50

.80 to .90 for internal consistency

.70 or higher for test-retest

Front 51

What types of tests tend to have the highest reliability?

Back 51

Verbal tests

Front 52

What is the standard error of estimation?

Back 52

The standard error surrounding estimated true scores.

Front 53

What is an estimated true score?

Back 53

It corrects bias in an observed score by regressing it to the mean of the population.

This is necessary since observed scores above the mean are likely to be higher than true scores (and vice versa for observed scores below the mean).

Front 54

How is an estimated true score calculated?

Back 54

Estimated True Score (t') =

Mean Test Score+[test reliability (Observed Score-Mean Test Score)]

or

t'=test reliability(z score)

Front 55

What is the standard error of prediction?

Back 55

The likely range of observed scores expected upon retest using an alternative form.

Front 56

What types of tests produce stupidly high reliability coefficients in split-half reliability?

Back 56

Speed tests

Front 57

If a number of patients are at or near the lowest possible score of the test, then what problem is the test distribution said to have?

Back 57

A High Floor. More easier items are needed.

Front 58

If a test has a low ceiling, then what type of scores will patients produce?

Back 58

Scores close to the highest possible score on the test. More difficult items are needed.

Front 59

For a test to best differentiate between groups, what percentage of items should most people get correct?

Back 59

50%

Front 60

What is Type I Error?

Back 60

Alpha error:

Indicating that a condition is present when it is not

Front 61

What is Type II error?

Back 61

Beta error:

Indicating that no condition is present when it is.

Front 62

Describe the Deficit Measurement Model.

Back 62

The measurement of impairment when an examinee’s performance is significantly below an actual or estimated previous level of functioning.

Front 63

What comparison standards are used in deficit measurement?

Back 63

Normative comparison standards;

Individual comparison standards;

Criterion comparison standards

Front 64

What are the two primary types of normative comparison standards?

Back 64

Species Comparison & Population Averages

Front 65

How do species comparison standards work?

Back 65

They look at capacities shared by all healthy members of a species (i.e., language). A lack of these capacities is called a pathognomonic sign.

Front 66

How do comparisons to population averages work?

Back 66

Compares individual performances against the distribution of performances for healthy individuals in a population to identify deficient performances.

[If you don't know this, then you fail at neuropsychology...forever.]

Front 67

What are key weaknesses of population normative comparisons?

Back 67

It only tells you how the individual compares to the average population, it does not allow you to identify a key deficit in an individual.

Front 68

Describe individual comparison standards.

Back 68

Compares individuals to their premorbid performance.

Front 69

What is a criterion referenced comparison standard?

Back 69

A score on a pre-specified standard that differentiates between those who pass or fail. The goal is not to look at differences between examinees.

Focused on everyone achieving MASTERY or the SAME LEVEL OF PERFORMANCE.

Front 70

What does the Standard Error of Measurement quantify?

Back 70

Index of the amount of error expected in obtained scores due to the unreliability of a test.

Front 71

What is the advantage of a test with High Sensitivity?

Back 71

A NEGATIVE result rules OUT the diagnosis

SN-NOUT

Front 72

What is the advantage of a test with High Specificity?

Back 72

A POSITIVE result rules IN a diagnosis

SP-PIN

Front 73

What is the mathematical expression of Sensitivity?

Back 73

Sn = True Positives/(True Positives+False Negatives)

or

Sn = A/(A+C)

Front 74

What is the mathematical expression of Specificity?

Back 74

Sp = True Negatives/(False Positives + True Negatives)

or

Sp= D/(D+B)

Front 75

What is another name for pre-test probability?

Back 75

Prevalence

Front 76

What is the mathematical expression of disease prevalence (i.e., pre-test probability)?

Back 76

Prevalence = (True positives + False Negatives)/(True Pos. + False Pos.+ True Neg. + False Neg.)

OR

Prevalence = (A+C)/(A+B+C+D)

OR

Prevalance = People who actually have the condition/The entire population

Front 77

What is the mathematical expression of the Liklihood Ratio for a positive result?

Back 77

LR+ = (Sensitivity)/(1-Specificity)

Front 78

What is the mathematical expression of the Liklihood Ratio for a negative test result?

Back 78

LR- = (1-Sensitivity)/(Specificity)

Front 79

How are the pretest odds of a condition calculated?

Back 79

Pretest odds = (Prevalence)/(1-Prevalence)

Front 80

What is the difference between Pretest Probability and Pretest Odds?

Back 80

Probability: the proportion of people with a condition

Odds: a ratio of two probabilities. In this case, the the ratio is (probability of having a condition):(probability of not having a condition)

Front 81

What is the mathematical expression of pretest odds?

Back 81

Pre-test Odds = (prevalence)/(1-prevalence)

Front 82

What is post-test odds for a positive test result equal to(i.e., synonymous with)?

Back 82

Positive Predictive Value

Front 83

What is the practical definition of post-test odds?

Back 83

The added value of a test result over pre-test probability (i.e., prevalence). Can be positive or negative value.

Front 84

What is the mathematical expression of post-test odds given a positive result?

Back 84

Post-test odds = (Pre-test odds) X (LR+)

Front 85

What is the mathematical expression of post-test probability for a positive result?

Back 85

Post-test probability = (Post-test odds)/(Post-test Odds + 1)

Front 86

What is the mathematical relationship between probability and odds?

Back 86

Probability = (odds)/(1+ odds)

Odds = (probability)/(1-probability)

Front 87

How is parametric statistical modeling used to define impairment?

Back 87

Compares performance relative to normative population through an extrapolation of the central limit theorem. BASED ON NORMATIVE COMPARISON STANDARD

Front 88

How is Bayesian statistical modeling used to define impairment?

Back 88

Through the use of a probability model and corrective variables to improve the accuracy of prediction. BASED ON INDIVIDUAL COMPARISON STANDARD.

Front 89

How does choosing lower cut offs (i.e., -1.5 SD vs. -1 SD) affect specificity?

Back 89

A lower cut off will increase specificity (i.e., less false positives but more false negatives).

Front 90

How does choosing a higher cut score (i.e., -1 instead of -1.5) affect sensitivity?

Back 90

Sensitivity will increase (i.e., a negative test result will rule a disease out), but this results in more false positives.

Front 91

When examining discrepancies between scores (e.g., VCI vs. PRI), is it more meaningful to look at statistical significance or base rates for the difference?

Back 91

BASE RATES!

A statistically significant difference may still occur in 25+% of the population, meaning it is common.

Front 92

Are patient's with low IQ more likely to have more variability in their scores?

Back 92

NO, but they do tend to have more scores that normally fall in the impaired range.

In contrast, those with high IQ tend to have greater variability in scores, but less that fall in the impaired range.

Front 93

TRUE OR FALSE:

Pediatric tests always have the same psychometric properties across age groups.

Back 93

FALSE

Contributors to this problem include:

Different start/stop points on tests;

Remarkable variability due to developmental differences;

Occasionally limited normative sample sizes;