Reading...
Play button
Play button
Use LEFT and RIGHT arrow keys to navigate between flashcards;
Use UP and DOWN arrow keys to flip the card;
H to show hint;
A reads text to speech;
28 Cards in this Set
- Front
- Back
|
Test Construction
|
A tsts is a systmtc mthd of measrng a sampl of bhvr. Tst Cnstrctn nvolvs spcfyng th tst's prpos, gnrtng tst itms, admnstrng th itms 2 a smpl of xamnees 4 th prpos of itm anlys, evalutng th tst's reliblty nd valdty, nd estblshng nrms. Selctng itms 2 nclud n a tst nvolvs cnsdrng ea itm's relevnc, dffclty lvl, nd ablty 2 dscrmnt btwn xamnees w/dffrnt lvls of th chrctrstcs bng measrd.
|
|
|
Test Construction
Item Analysis Relevance |
Content Appropriateness: Does the itm actuly asses th cntnt or bhvr domn tht th tst is dsgnd 2 evaluat? Taxonomic Level: Does th itm rflct th approprit cgntv or ablty lvl? Extraneous Abilities: To wht xtnt does th itm reqir knwldg, skils nd abltys outsid th domn of ntrst?
|
|
|
Test Construction
Item Analysis Item Difficulty |
p (itm difficulty) value = Total # of examinees passing th item divided by the Total # of examinees. Th valu of p ranges from 0 to 1.0 w/lrgr valus ndicatng ezr itms. Whn p is = to 1, ths means tht th itm ws answrd crrctly by all xamnees; whn p is 0, ths ndicats tht non of th xamnees answrd th itm crrctly. 4 mny tsts, itms w/modrat dffclty lvls (p valu clos to .50) r rtnd. Ths strtgy usfl b/c it ncreses tst scors varblty, hlps nsur tht scors wil b nrmly dstrbtd, provds mxmum dscrmntn btwn xamnees, nd hlps mxmz th tst's rliblty. Th optml dffclty lvl is affctd by svrl fctrs. One fctr is th lklihd tht an xamnee cn chuz th crrct answr by gussng, w/th prfrd lvl bng hlfwy btwn 100% of xamnees answrng itm crrctly nd th lvl of succss xpctd by chnc alon. As an xampl, 4 tru/fals itms, th prblty of obtng a crrct answr by chnc alon is .50, so prfrrd dffclty lvl is 75%. Anthr fct is gol of tstng. If th gol is 2 chuz a crtn # of xamnees, th optml dffclty lvl crrspnds 2 th proprtn of xamnees 2 b selctd. 4 a grdute schl admssn tst, if only 15% of applcnts r 2 b admittd, itms wil b chsn so th th avrg itm dffclty lvl 4 th ntir tst is abt .15. Kp n mnd tht, n most cses, a p valu of .50 is optml. One xcptn is th cse of a tru/fals tst, for wch th optml p valu is .75.
|
|
|
Test Construction
Item Analysis Item Discrimination |
Rfrs 2 th xtnt 2 wch a tst itm dscrmnts (dffrntiats) btwn xamnees hu obtn hi v. lo scors on th ntir tst or on an xtrnl critrn. One wy 2 meas itm dscrmntn is 2 calcult an itm dscrmntn ndx (D). Ths rquirs idntfyng th xamnees n th tryout smpl hu obtnd th hghst nd th lwst scors on th tst (oftn th upr 27% nd th lwr 27%) nd, 4 ea itm, sbtrctng th % of xamnees n th lwr-scorng grp (L) from th % of th xamnees n th uppr-scorng grp (U) hu answrd th itm crrctly: D=U-L.
Th itm dscmntn ndx rngs frm -1.0 to +1.0. If all xamnees n th uppr grp nd none n th lwr grp answrd th itm crrctly, D is = to +1.0. If non of th xamnees n th uppr grp nd all xamnees n th lwr grp answrd itms crrctly, D=-1. For mst tsts, an itm w/a dscrmntn ndx of .35 or hghr is cnsdrd accptbl. As notd abv, itms w/modrt dffclty lvl (arund .50) hv th grtst potntl 4 mxmum dscrmntn. |
|
|
Test Construction
Item Analysis Item Resonse Theory |
tst cnstrctn usuly bsed on 1 of 2 thrys: clsscl tst thry or itm rspns thry (th latnt trait modl). Th stds of evaltn dscrbd n ths sctn - itm dffclty nd dscrmntn, rliblty, nd valdty - r assctd w/clsscl tst thry, wch vus an obtnd tst scor as rflctng a cmbntn of "truth" nd eror. Crtcs of ths thry point out tht it suffrs frm sevrl shrtcmngs. 1 of th bggst prblms is tht it is dffclt 2 equat scors obtnd on dffrnt tst tht hv bn dvlpd on th bsis of clsscl tst thry. A totl scor of 50 on an Englsh tst duz nt ncssrly mean th sam thng as a scor of 50 on an arthmtc tst or a dffrnt Englsh tst.
|
|
|
Test Construction
Item Analysis Item ResPonse Theory |
Accrdng 2 advcts, itm rspns thry (IRT) ovrcms thes prblms nd hs sevrl othr advntgs. Th itm chrctrstcs (paramtrs) drivd frm IRT r cnsdrd 2 b sampl nvarint; i.e., thy r th sam acrss dffrnt smpls. Als b/c tst scors r rprtd n trms of an xamnee's lvl on th trait bng measrd (rthr thn n trms of a totl tst scor), it is pssbl 2 equat scors frm dffrnt sets of itms nd frm dffrnt tsts. Anthr advntg is tht th use of IRT maks it ezr 2 dvlp cmptr-adptv tsts, n wch th admnstrtn of sbsqnt itms is bsed on th xamnees's prfrmnc on prvus itms. Whn usng IRT, an ITEM CHRACTERISTIC CURVE (ICC) is cnstrctd 4 ea itm by plottng th proportn of examnees n th tryout smpl hu answrd th itm crrctly agnst ethr th totl tst scor, prfrmnc on an xtrnl critrn, or a mthmtcly-drivd est of a ltnt ablty or trait. Th curv prvids nfo on th r/s btwn an xamnee's lvl on th ablty or trait measrd by th tst nd th probilty tht h/she wil rspnd 2 th itm crrctly. Th varus IRT mdls prduc ICCs tht prvid nfo on ethr one, two, or three parmtrs. IRT is vry cmplx, nd it is mpssbl 2 dscrb n a fw pargrhs. 4 th xam, hv "IRT lnkd w/ICC" nd knw hw dffclty lvl, dscrmntn, nd prblty of gussng crrctly r ndictd by th ICC.
|
|
|
Test Construction
Reliability Classical Test Theory |
From th prspctv of clsscl tst thry, an xamnee's obtnd tst scor (X) is cmposd of 2 cmponts, a tru scor cmponts (T) nd an eror cmpont (E): X=T+E. Th tru scor cmpont rflcts th xamnee's status w/rgrd 2 th attrbt tht is measrd by th tst, whl th eror cmpont rprsnts msmnt error. Msmnt eror is rndm eror: it is du 2 fctrs tht r irrelvnt 2 wht is bng measrd by th tst nd tht hv an unpredictbl (unsystmtc) effct on an xamnee's tst scor. Th scor you obtn on th lcsng xam (X) is lkly 2 b du both 2 th knwldg u hv abt th topics adrssd by xam itms (T) nd th effcts of rndm fctrs (E) such as th wy tst itms r wrttn, any altratns n anxty, attn, or mtvtn u xprnc whl tkng th tst, nd th accurcy of yur "educted guesses."
|
|
|
Test Construction
Reliability Classical Test Theory |
Whnvr we admnstr a tst 2 xamnees, we wud lk 2 knw hw mch of thr scors rflct "truth" nd hw mch rflcts eror. It is a meas of rliblty tht prvids us w/an est of th proportn of varblty n xamnees' scor tht is du 2 tru dffrncs amng xamnees on th attrbt(s) measrd by th tst. Whn a tst is rlibl, it prvids dpndbl, cnsistnt rslts nd, 4 ths resn, th trm cnsistncy is oftn gvn as a synym 4 rliblty.
|
|
|
Test Construction
Reliability The Reliabilty Coeficient |
A tst's rliblty (tru scor variblty) cud b measrd drctly. Hwvr, ths is nt pssbl, nd rliblty must b estmtd. Ther r svral wys 2 est a tst's rliblty. Ea nvolvs assesng th cnsistncy of an xamnee's scor ovr tme, acrss dffrnt cntnt smpls, or acrss dffrnt scorers nd is bsed on th asumptn tht variblty tht is cnsistnt is tru scor variblty, whl variblty tht is ncnsistnt rflcts msmnt (rndmn) eror. Most mthds 4 estmtng rliblty prduc a RELIABILITY COEFFICIENT, wch is a crrltn cffcnt tht rngs n valu frm 0.0 to +1.0. Whn a tst's rliblty cffcnt is 0.0, ths means tht all varblty n obtnd tst scors is du 2 msmnt eror. Cnvrsly, whn a tst's rliblty cffcnt is +1.0, ths ndcats tht all varblty n scors rflct tru scor varblty. Th rliblty cffcnt is symblzd w/ th lttr "r" nd a sbscrpt tht cntns two of th sam lttrs or #s (e.g., "rxx"). Th sbscrpt ndcats tht th crrltn cffcnt ws clcultd by crrltng a tst w/itslf rthtr thn w/sum othr measr.
|
|
|
Test Construction
Reliability The Reliabilty Coeficient |
Rgrdlss of th mthd usd 2 clcult a rlblty cffcnt, th cffcnt is ntrprtd drctly as th proprtn of vrblty n obtnd tsts scors tht rflcts tru scor varblty. E.g., a rliblty cffcnt of .84 ndicat tht 84% of varblty n cors is du 2 tru scor dffrncs amng xamnees, whl th rmaing 16% (1.00-.84) is du 2 msmnt eror. Note tht a rliblty cffcnt duz nt prvid nfo abt wht is actuly bng measrd. A rliblty cffcnt only measrs whthr th attrbt measrd - whtvr it is - is bng assesd n a cnsistnt, precise wy. Whthr th tst is actuly assesng wht it ws dsgnd 2 measr is addrssd by anlys of th tst's valdty.
|
|
|
Test Construction
Reliability The Reliabilty Coeficient |
Rmembr tht, n cntrst 2 othr crrltn cffcnts, th rliblty cffcnt is nvr squrd 2 ntrprt it bt is ntrprd drctly as a measr of tru scor varblty.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Test-Retest |
Th tst-retst mthd nvolvs admnstrng th sam tst 2 th sam grp of xamnees on two dffrnt ocasns nd thn crlltng th two sets of scors. Whn usng ths mthd, th rliblty cffcnt ndcats th dgre of stblty (cnsistncy) of xamnees' scors ovr tim nd is aka th cffcnt of stblty. Th prmry sorcs of msmnt eror 4 tst-retst rliblty r any rndm fctrs rltd 2 th tim tht passes btwn th two admnstrtns of th tsts. Ths tim smplng fctrs nclud rndm fluctns n xamnees' ovr tim (e.g., chngs n anxty or mtvtn) nd rndm variatns n th tstng situatns. Tst-retst rliblty is approprit 4 dtrmng th rliblty of tsts dsgnd 2 meas attrbts tht r rltvly stbl ovr tim nd tht r nt affctd bt rptd msmnt. It wud b apprpiat 4 a tst of aptitud, wch is a stbl chrctrstc, but nt 4 a tst of mood, snc mood flucts ovr tim, or of creatvty, wch mit b affctd by prevus xposur 2 tst itms.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Alternate (Equivalent, Parallel) Forms of Reliabilty |
2 asses a tst's ALTERNATE FORMS of rliblty, 2 equvlnt forms of th tst r admnstrd 2 th sam grp of xamnees nd th 2 sets of scors r crrltd. Altrnt forms of rliblty ndcats th cnsistncy of rspndng 2 dffrnt itm smpls (th two tst forms) nd, whn th forms r admnstrd at dffrnt tims, th cnsistncy of rspndng ovr tim. Th altrnt forms rliblty cffcnt is als clld th cffcnt of equvlnc whn th two frms r admnstrd @ abt th sam tim nd th cffcnt of equvlnc nd stblty whn a rltvly lng period tim seprats admnstrtn of th two frms. Th prmry sorc of msmnt eror 4 altrnt forms rliblty is cntnt smplng, or eror ntroducd by an ntractn btwn dffrnt xamnees' knwldg nd th dffrnt cntnt assesd by th itms ncluded n th two frms. Whn admnstrtn of th 2 frms is sepratd by a period of tim, tim smplng fctrs als cntrbt 2 eror. Lik tst-retst reliblty, altrnt forms reliblty is not appropriat whn th attrbt measrd by th tst is lkly 2 fluct ovr tim nd th forms wil b admntrd @ dffrnt tims or whn scors r lkly 2 b affctd by rptd msmnt (e.g., by practc effcts). altho altrnt frms rliblty is cnsdrd by sum xprts 2 b th most rigorus (nd bst) mthd 4 estmtng rliblty, it is nt oftn assesd du 2 th dffclty n dvlpng frms tht r truly equvlnt.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Internal Consistency |
Rliblty cn als b estmtd by measrng th ntrnl cnsistncy of a tst. SPLIT-HALF rliblty nd COEFFICENT ALPHA r two mthds 4 evalutng ntrnl cnsistncy. Both nvolv admnstrng th tst onc 2 a sngl grp of xamnees, nd both yld a rliblty cffcnt tht is aka th COEFFICIENT of INTERNAL CONSISTENCY.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Internal Consistency - Split-Half Reliabilty |
2 dtrmn a tst's split-half rliblty, th tst is splt n2 equl halvs so tht ea xamnee hs two scors (one on ea hlf of th tst). Scors on th 2 halvs r then crrltd. Tsts cn b splt n svrl wys, bt prbly th most cmmn is 2 dvid th tst on th bsis of odd-vs evn-#ed itms. A prbllm w/th splt-hlf mthd is tht it prducs a rliblty cffcnt tht is bsed on tst scors tht wer drivd frm 1/2 of th ntir lngth of th tst. If a tst cntns 30 itms, ea scor is bsed on 15 itms. B/c rliblty tnds 2 dcres as th lgnth of a tst dceses, th splt-hlf rliblty cffcnt usuly undrestmts a tst's tru rliblty. 4 ths resn, th splt-hlf rlblty cffcnt is ordnrly crrctd usng th SPEARMAN_BROWN PROPHESY FORMULA, wch prvids an estmt of wht th rliblty cffcnt wud hv bn hd it bn bsed on th full lngth of th tst.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Internal Consistency - Coefficient Alpha |
Cronbach's cffcnt alpha als nvolvs admnstrng th tst onc 2 a sngl grp of xamnees. Hwvr, rthr thn splttng th tst n hlf, a spcl frmula is usd 2 dtrmn th avrg dgre of ntr-itm cnsistncy. 1 wy 2 ntrprt cffcnt alpha is as th avrg reliblty tht wud b obtnd frm all pssbl splits of th tst. Cffcnt alpha tnds 2 b cnsrvtv nd cn b cnsdrd th lwr boundry of a tst's rliblty. Whn tst itms r scord dichotmsly (right or wrong), a varitn of cffcnt alpha kwn as th KUDER-RICHARDSON FORMULA 20 (KR-20) cn b usd.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Internal Consistency |
Cntnt smplng is a sorc of eror 4 bth splt-hlf rliblty nd cffcnt alpha. 4 splt-hlf rliblty, cntnt smplng rfrs 2 th eror rsltng frm dffrncs btwn th cntnt of th 2 hlvs of th tst (i.e., th itms ncluded n 1/2 may bttr fit th knwldg of sum xamnees thn itms n th othr hlf); 4 cffcnt alpha, cntnt (itm) smplng rfrs 2 dffncs btwn ndvdul tst itms rthr than btwn tst hlvs. Cffcnt alpha als hs as a sorc of eror, th hetrognty of th cntnt domain. A tst is hetrognus w/rgrd 2 cntnt domain whn its itms measr svrl dffrnt domains of knwldg or bhvr. Th grtrs th hetrognty of th cntnt domain, th lwr th ntr-itm crlltns nd lwr th mgntud of cffcnt alpha. Cffcnt alpha cud b xpctd 2 b smllr 4 a 200-itm tst tht cntains itms asesng knwldg of tst cnstrctn, ststcs, ethics, i/o psych, clncl psych nd psychopthlgy thn 4 a 200-itm tst tht cntains qustns on tst cnstrctn only.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Internal Consistency |
Th mthds 4 assesng ntrnl cnsistncy rliblty r usfl whn a tst is dsgnd 2 meas a sngl chrctrstc, whn th chrctrstc measrd by th tsts flctuats ovt tme, or whn scors r lkly 2 b affctd by rptd xpsur 2 th tst. Thy r not appropriat 4 assesng th rliblty of speed tsts b/c, 4 ths tsts, thy tnd 2 prduc spurusly hi cffcnts. (4 speed tsts, altrnt forms rliblty is usuly th bst choic).
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Inter-Rater (Inter-Scorer, Inter-Observer) Reliability |
Inter-rater rliblty is of cncrn whnvr tst scors dpnd on a ratr's jdgmnt. A tst cnstrctr wud wnt 2 mak sur tht an essay tst, a bhvrl obsrvtn scal, or a prjctv tst hv adqut intr-rtr rliblty. Ths typ of rliblty is assesd ethr by clcultng a crrltn cffcnt (e.g., KAPPA STATISTIC or cffcnt of cncrdnc) or by dtrmng th % agrmnt btwn 2 or mre rtrs. Altho th latr tchnq is frqntly usd, it cn lead 2 errneus cnclusns snc it duz ny tak n2 accnt th lvl of agrmnt tht wud hv ocurd by chnc alon. Ths is a prblm 4 bhvrl obsrvtn scals tht rquir rtrs 2 rcrd th frqncy of a spcfc bhvr. N ths situatn, th dgre of chnc agrmnt is hi whnvr th bhvr hs a hi rate of ocurnc, nd % agrmnt wil prvid an nflatd est of th meas's rliblty.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Inter-Rater (Inter-Scorer, Inter-Observer) Reliability |
Sorcs of eror 4 ntr-rtr rliblty nclud fctrs rltd 2 th rtrs such as lack of mtvtn nd rtr biases nd chrctrstcs of th measrng dvice. An ntr-rtr rliblty cffcnt is lkly 2 b lo whn rtng ctgres r nt xhustv (i.e., dn't nclud all pssbl rspnses or bhvrs) nd/or r not mutuly xclusv. Th ntr-rts rliblty of a bhvrl rtng scal cn als b affctd by cnsnsul obsrvr drift, wch ocurs whn 2 or mre obsrvrs wrkng 2gthr nflunc ea othr's rtngs so tht thy both assgn rtngs n s simlrly idiosyncrtc way. Unlk othr sorcs of eror, cnsnsul obsrvr drift tnds 2 rtfcly nflt ntr-rtr rliblty. Th rliblty (nd vldty) of rtngs cn nb mprvd n svrl wys, bt, ovrl, th bst wy 2 prvid rtrs w/trng tht mphszs th dstnctn btwn obsrvtn nd ntrprtn.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Study Tip |
4 th xam, hv th Spearman-Brown frmuls lnkd w/splt-hlf rliblty, KR-20 lnkd w/cffcnt alpha, nd th kappa ststc lnkd w/ntr-rtr rliblty. Als knw tht altrnt forms rliblty is th mst thoro mthd 4 estmtng rliblty nd tht ntrnl cnsistncy is nt approprt 4 speed tsts.
|
|
|
Test Construction
Reliability: Factors That Affect The Reliability Coefficient |
Th mgntud of th rliblty is affctd als by th lngth of th tst, th rng of th tst scors, nd th prblty tht th crrct rspns 2 itms cn b slctd by gesng.
|
|
|
Test Construction
Reliability: Factors That Affect The Reliability Coefficient - Test Length |
Th lrgr th smpl of th attrbt bng measrd by a tst, th lss th rltv effct of msmnt eror nd th mre lkly th smpl wil prvid dpndbl, cnsistnt nfo. Cnsqntly, a gen rul is tht th lngr th TEST LENGTH, th lrgr th tst's rliblty cffcnt.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Inter-Rater (Inter-Scorer, Inter-Observer) Reliability |
Inter-rater rliblty is of cncrn whnvr tst scors dpnd on a ratr's jdgmnt. A tst cnstrctr wud wnt 2 mak sur tht an essay tst, a bhvrl obsrvtn scal, or a prjctv tst hv adqut intr-rtr rliblty. Ths typ of rliblty is assesd ethr by clcultng a crrltn cffcnt (e.g., KAPPA STATISTIC or cffcnt of cncrdnc) or by dtrmng th % agrmnt btwn 2 or mre rtrs. Altho th latr tchnq is frqntly usd, it cn lead 2 errneus cnclusns snc it duz ny tak n2 accnt th lvl of agrmnt tht wud hv ocurd by chnc alon. Ths is a prblm 4 bhvrl obsrvtn scals tht rquir rtrs 2 rcrd th frqncy of a spcfc bhvr. N ths situatn, th dgre of chnc agrmnt is hi whnvr th bhvr hs a hi rate of ocurnc, nd % agrmnt wil prvid an nflatd est of th meas's rliblty.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Inter-Rater (Inter-Scorer, Inter-Observer) Reliability |
Sorcs of eror 4 ntr-rtr rliblty nclud fctrs rltd 2 th rtrs such as lack of mtvtn nd rtr biases nd chrctrstcs of th measrng dvice. An ntr-rtr rliblty cffcnt is lkly 2 b lo whn rtng ctgres r nt xhustv (i.e., dn't nclud all pssbl rspnses or bhvrs) nd/or r not mutuly xclusv. Th ntr-rts rliblty of a bhvrl rtng scal cn als b affctd by cnsnsul obsrvr drift, wch ocurs whn 2 or mre obsrvrs wrkng 2gthr nflunc ea othr's rtngs so tht thy both assgn rtngs n s simlrly idiosyncrtc way. Unlk othr sorcs of eror, cnsnsul obsrvr drift tnds 2 rtfcly nflt ntr-rtr rliblty. Th rliblty (nd vldty) of rtngs cn nb mprvd n svrl wys, bt, ovrl, th bst wy 2 prvid rtrs w/trng tht mphszs th dstnctn btwn obsrvtn nd ntrprtn.
|
|
|
Test Construction
Reliability Methods for Estimating Reliability: Study Tip |
4 th xam, hv th Spearman-Brown frmuls lnkd w/splt-hlf rliblty, KR-20 lnkd w/cffcnt alpha, nd th kappa ststc lnkd w/ntr-rtr rliblty. Als knw tht altrnt forms rliblty is th mst thoro mthd 4 estmtng rliblty nd tht ntrnl cnsistncy is nt approprt 4 speed tsts.
|
|
|
Test Construction
Reliability: Factors That Affect The Reliability Coefficient |
Th mgntud of th rliblty is affctd als by th lngth of th tst, th rng of th tst scors, nd th prblty tht th crrct rspns 2 itms cn b slctd by gesng.
|
|
|
Test Construction
Reliability: Factors That Affect The Reliability Coefficient - Test Length |
Th lrgr th smpl of th attrbt bng measrd by a tst, th lss th rltv effct of msmnt eror nd th mre lkly th smpl wil prvid dpndbl, cnsistnt nfo. Cnsqntly, a gen rul is tht th lngr th TEST LENGTH, th lrgr th tst's rliblty cffcnt.
|
