Csr Eppp Stats, Test Construction

Front 1

When assesing a test's split half reliability, what effect will incrasing the heterogenity of the sample (S's) have?

Back 1

Increase the reliability

Front 2

What is eta squared

Back 2

A measure of effect size describing the amount of variablity in the criterion accounted for by the predictor

Front 3

What is Cohen's d

Back 3

Cohen's d is a measure of effect size when comparing the mean difference between two groups

Front 4

ANOVA:regression as Cohen's d:_________

Back 4

eta squared

Front 5

Professor Sharp argues against raising the legal drinking age from 18 to 21 on the ground that doing so will only encourage 18-, 19-, and 20-year olds to drink. Apparently, Sharp is familiar with which of the following theories:


individuation
psychological reactance
inoculation
expectancy-value

Back 5

reactance

Front 6

How do you calculate a CI for a S's score on a test

Back 6

to calculate CI you need
S's score
SEM of the test
level of confidence desired

e.g. for a test w/ SEM of 5, if a S scored 105 and you wnated a 68% CI:
68% CI = within 1 SEM
105 +/- 5
68% CI = 100-110

for a 95 % CI (2 SEM)
105 +/- (2x5)
105 +/- 10
95 % CI = 95-115

99% CI (3 SEM)
105 +/- (3x5)
105 +/- 15
99% CI = 90-120

Front 7

What is the formuula for SEM?

Back 7

sigma/sqrt N

Front 8

What formula is

sigma [1-sqrt(1-r2xy)

where sigma=sd of criterion (Y) scores
and
rxy= correlation coefficient (an estimate of criterion validity)

Back 8

Standard Error of Estimate

It is used to create CI's for estimates of performance on a criterion (Y variable) derived from a regression equation involving the predictor (X variable) score

CI = Yhat +/- SEe(# of SEe from mean assoc. w/ desired CI)

68% CI = 1 SEE
95% CI = 2 SEE
99% CI = 3 SEE

Front 9

What is criterion related validity

Back 9

Criterion Related Validity is the extent to which a predictor (X) reliably estimates performance on a criterion (Y) using a regression equation

Front 10

What is construct validity

How is it assessed?

Back 10

Construct Validity refers to the extent to which a predictor measures the concept it is deisned to.

Assess it using facor analysis or multitrait/multimethod analysis

Front 11

What are the 3 types of Construct validity?

Back 11

1. Discriminant validity
2. Convergent validity

Assess it using
1. Factor Analysis
2. Multitrait Multimethod Matrix

Hint 11

Constuct Validity = DC FM

Construction of the Antena towers on Wisconsin DC recpeption of FM radio stations

Front 12

What are the 2 types of Criterion Validity

How do you assess them

Back 12

predicitive and concurrent

Predictive validity is the assessed by calculating the Standard Error of Estimate (SEest)

SEest=SDy[sqrt(1-r2xy)]

where SDy = SD of criterion scores (Y scores)
and rxy2=validity coefficient (correlational coefficient indicating how much of the variance in Y is predicted by variability in X)

Hint 12

Crtierion Validity is PC

Front 13

Reliability of a test= sum of what two things

Back 13

Communality + specificity

where communality= % of variance in DV (Y) explained by all of the IV (X)

where Yhat=X put into the regression equation

specificity = 1-error

Front 14

What is Content Validity?

For what kind of tests is it important

Back 14

Content validity is important for:
1. Mastery tests (how much does S know about a specific Domain - like the EPPP)
2. Achievement tests (same as mastery?)
3. Developing tests of the KSA needed for a particular job (Use Job Analysis to determine the KSA)

4. job samples

It can be evaluated by asking "experts" to examine the test and see if it seems representative of the knowledge in the domain.

AND by testing to see:

Does it discriminate between novices and experts in the field

Hint 14

Content Validity =
1. Discrimination between experts and novices
2.Experts evaluation of test, 3. Correltion w/ established measures,

Content Validity=DEC

Front 15

What are the 3 mnemonics for teh 3 types of validity

Back 15

Construct DCFM towers
Criterion is PC
Content = ED and content validity is often all that elementary and secondary mastery tests are about

Hint 15

Think about progression of my time at AU

Construct(ion) of DCFM towers on the way in

Criterion=PC (the criterion for being liked on college campuses is are you PC)

ED content = what was the content of my education at AU?

Front 16

What is the nature of the relationship between a test's reliability and validity

Back 16

A test's validity coefficient cannot exceed the square root of its reliability coefficient

Hint 16

Memorize it!

Write out the formula for this to help figure out difficultly worded questions!


Hints:
1. reliability (r2xx) is already squared

2. the relaionship involved involves taking the square root of something

Front 17

In terms of percentile ranks, where do 1sd, 2sd and 3sd above the mean fall?

Back 17

84
97.7
99

Hint 17

different from normal distribution

Front 18

If the assumptions for a One-Way ANOVA are violated, what test could you use instead

Back 18

Kruskal-Wallis

Front 19

What are the 3 assumptions when using a parametric test?

Back 19

1. NORMAL POPULATION DISTRIBUTION (not too serious as long as can compensate by using sample size >= 25-30)

HOMOSCEDASTICITY - variances of the two populations are equal - if you violate this assumption, it's usally not too serious)

3. INDEPENDENT OBSERVATIONS*
very serious - Incrases likelihood of Type I and Type II error

Independent Observations

Front 20

What 2 assumptions do Parametric and Non-parametric tests share?

Back 20

1. Random Selection from Population

2. Independent Observations

Front 21

If you have ordinal data w/ 2 IV's what test do you use to see if there are stat. sig differences between the groups?

Back 21

Mann-Whitney U

Front 22

If you have Nominal Data, what test do you use to compare cell means?

Back 22

Chi-Square

Front 23

If the assumption or normality is violated for a one-way ANOVA, what non-parametric test could you use instead?

Back 23

Kruskal-Wallis

Front 24

Non-parametric test to compare 2 indpendent groups?

Back 24

Mann-Whitney U

Hint 24

U looks like two lines of data if you eliminate the bottom

Front 25

Non-parametric test to compare 2 dependent (repeated-measures) groups?

Back 25

Wilcoxan Matched Pairs Signed Rank Test

Front 26

What test would you substitute for an independent measures t-test if the assumption of normality were violated or the data were ordinal?

Back 26

Mann-Whitney U

Front 27

What test would you substitute for a repeated measures t-test (or RM ANOVA w/ 1 IV) if the data were Ordinal or if the assumption of normality was violated?

Back 27

Wilcoxan Matched-pairs signed rank test

Front 28

If you want to use 2 or more continuous predictor variables (IV's) to predict preformance on nominal criterion (e.g. Yes/No or DSM-IV TR Dx.) what test should you use?

Back 28

Discriminant Function Analysis

Front 29

If the relationship between 2 or more discreet IV's (not contiuous) and a continuous Y variable, what type of test should you use?

Back 29

Logistic Regression

Front 30

If you want to test a pre-defined theory about causal relationship(s) between two vairables, and the relationship goes in one way (X causes Y, but not vice versa), what test (type of modling) should you use?

Back 30

Path Analysis

Hint 30

Think of Life Path - it goes only one way

Front 31

For what is the Spearman-Brown formula used?

Back 31

correction for split-half reliability calculation

Hint 31

Split half - Spearman Brown - alliteravie,
rhythm of saying them together

Front 32

If you are assessing inter-item consistency (reliability), and variable is continuous, you would use:

Back 32

Cronbach's alpha

Hint 32

e.g. score on MOCI for MA Thesis

Front 33

If you are assessing inter-item consistency and the variable is dichotomous (e.g. yes/no), what would you use to correct for the overestimaton you would expect using Cronbach's alpha?:

Back 33

Kuder Richardson-20

Hint 33

Kuder-Richardson 20 has a 2 in it and the DV on this kind of test has only 2 possible answers

Front 34

To assess inter-rater reliability, what two things could you use?

Back 34

1. simple percent agreement
2. Kappa statistic

Front 35

How can you increase a test's reliability?

Back 35

1. Increase Homogenity of test-items

2. Increase Heterogeneity of Subjects taking test

3. Increase test length

Front 36

What are the three types of Validity

Back 36

CRITERION (use regression, Standard Error of Estimate)
1. Predicdtive
2. Concurrent

CONTENT (Ask an expert what s/he thinks)
1. Expert's opinion
2. Does it discriminate between experts and novices in the field?


CONSTRUCT
Two Types
1. Convergent
2. Discriminant
Two Ways to Assess
1. Factor Analysis
2. Multi-trait, Multi-method

Hint 36

Three C's w/ two types of validity each




1. Content (expers' opinion)
2. Construct
convergent validity, discriminant validity
factor analysis; multi-trait, multimethod
3. Criterion
Predictive Validity (regression, Standard Error of Estimate)
Concurrent Validity

Front 37

Validity of a Predictor is limited by it's

Back 37

Reliability

Validity <= (sqrt Reliability)

Hint 37

Think about the formula for calculating the Standard Error of Estimate (for Confidence Interval's when predicting score on Criterion from predictor using a regression equation)

SEest = SDy*[1-(sqrt r2xy)]

Front 38

Validity of Predictor is less than or equal to the sqrt

and is less than or = to:

Back 38

Validity of Predictor (rxy) is less than or equal to the square root of it's reliability of the predictor
rxy <= sqrt(r2xx)

and is less than or equal to the square root of the product of the reliability of the predictor times the reliability of the criterion
rxy <= sqrt(r2xx*r2yy)

Front 39

If you want to calculate the correlation between two variables which are both continuous, which of the following would you use?

Biserial
Point Biserial
Rho
Pearson r

Back 39

Pearson r

Front 40

If you want to calculate the correlation between 2 variables and one is continuous and the other is a true dichtomoy, what test would you use?

Back 40

Point Biserial

Hint 40

If something is pointed, it is specific

Front 41

If you want to calculate the relationship between two rank-ordered variables, which of the following would you use?

Pearson r
Biserial
Point Biserial
rho

Back 41

rho

Front 42

If you want to calculate the relationship between two variables, one of which is continuous and the other of which is an artificial dichotomy

Back 42

Bserial

Front 43

In Item Response Theory, when one creates an Item Characteristic Curve (ICC), what is on the X-axis and what is on the Y-axis

Back 43

X-axis: S's ability level (low to high)

Y-axis: % of S's who get item right

Front 44

When using ICC, what does the slope indicate?

Back 44

Disciminative ability for S's high and low in the ability being assessed (sharp slope discriminates well, flat slope does not)

Front 45

When using ICC, how do you know what the chance of guessing correctly is?

Back 45

Y-intercept

Front 46

When using ICC, how do you determine how difficult an item is?

Back 46

at what level of ability (where on the X-axis) do 50% of S's get the item right