Nce Research And Evaluation

Front 1

causal-comparative (sometimes call an ex post facto study) has these essential characteristics:

Back 1

1. researchers observe and describe some current condition
2. researchers look to the past to try to identify the possible causes of the condition.
(is non-experimental)

Front 2

directional vs. nondirectional hypothesis.

Back 2

directional = researchers predict which group will be higher

Front 3

program evaluation

Back 3

applied research (versus basic research, which is understanding underlying theories)

Front 4

formative evaluation

Back 4

1. info is collected in the process of implementing a program
2. collecting info on the progress toward the ultimate goals (can prevent disappointment by changing midway if not working)

Front 5

summative evaluation

Back 5

when evaluators collect info about participants' attainment of the ultimate goals and the end of the program

Front 6

If a researcher studies every member of a population, this research is conducting a ______.

Back 6

census

Front 7

Researchers obtain biased samples when they use...

Back 7

samples of convenience

Front 8

simple random simple

Back 8

every member of a population is given an equal chance of being included in a sample

Front 9

sample error

Back 9

error created by random sample

Front 10

systematic simpling

Back 10

seen as equivalent to simple random sampling; every nth individual is selected.

Front 11

The technical term for discussing the magnitude of sampling errors is...

Back 11

precision (results are more precise when researchers reduce sampling errors)

Front 12

stratified random sampling

Back 12

divide a population into strata (men, women)
select the same PERCENTAGE (versus number)from each strata

Front 13

cluster sampling

Back 13

draw groups (or clusters) of participants instead of drawing individuals.
To be unbiased, the clusters or groups must be drawn at random

Front 14

purposive sampling

Back 14

purposively select individuals whom they believe will be good sources of information. (popular for qualitative research)

Front 15

snowball sampling

Back 15

useful when attempting to locate participants that are hard to find (find 1 who might lead you to others, for example, heroine addicts)

Front 16

mortality

Back 16

when subjects drop-outs, increases bias and should be reported in results

Front 17

T or F - increasing sample size decreases precision

Back 17

false

Front 18

instrument

Back 18

the generic term for any type of measurement device

Front 19

instrumentation

Back 19

terms used as the heading for the section of a report where the measurement devices used in the research are described

Front 20

valid

Back 20

the extent it measures what it is designed to measure and accurately performs the function it is purported to perform.

Front 21

to determine content validity of an instrument, researchers...

Back 21

make judgment on the appropriateness of its content

Front 22

predictive validity

Back 22

to what extent does the test predict the outcome it is supposed to predict?

Front 23

validity coefficient

Back 23

correlation coefficient used to express validity: range from 0 to 1 (1=perfect validity)

Front 24

concurrent validity

Back 24

an independent measure of the same trait that the test is designed to measure (a different test) given at about the same time that the test is administered.

Front 25

concurrent and predictive validity falls under a general term,

Back 25

criterion validity

Front 26

construct validity

Back 26

relies on subject judgments AND empirical data; offers only indirect evidence. (looking at constructs, like depression or self-esteem)

Front 27

A test is said to be reliable if it yields...

Back 27

consistent results

Front 28

A test with high reliability may have ____ validity.

Back 28

LOW

Front 29

When evaluating instruments, reliability/validity is more important than reliability/validity. (which)

Back 29

Validity is more important than reliability.

Front 30

when researchers use reliability coefficients to describe agreement between observers, they are usually called...

Back 30

interobserver reliability coefficients

Front 31

When researchers use correlation coefficients to describe reliability, they call them...

Back 31

reliability coefficients

Front 32

when studying interobserver reliability, researchers uaully obtain the two measurements...

Back 32

at the same time

Front 33

in ________ reliability, researchers measure at two different points in time.

Back 33

test-retest

Front 34

How is parallel-forms reliability determined?

Back 34

administering one form of the test to examinees and about a week or two later, administering the other form to the same examinees, thus yielding 2 scores per examinee.

Front 35

How high should a reliability coefficient be?

Back 35

Most published tests have a reliability of .80 or higher, so researchers should strive to select or build instruments that have coefficients at least this high, especially if researchers plan to interpret scores for individuals.

Front 36

For group averages based on groups of participants of about 25 or more, instruments with reliability coefficients as low as _____ can be serviceable.

Back 36

.50

Front 37

split-half reliability

Back 37

(usually odd-even split) one test is split to provide 2 scores for each individual. These are correlated to estimate internal consistency (checks on consistency within the test itself)

Front 38

An alternative to the split-half method for estimating internal consistency is _____.

Back 38

Cronbach's alpha (formula for essentially averaging a ton of split-half possiblities (like even-odd, first half/second half, and so on).

Front 39

Having high internal consistency is desirable when a research has developed a test designed to measure... (and provide example)

Back 39

single unitary variable
example: if a researcher build a test on the ability to sum one-digit numbers, students who score high on some of the items should score high on the other items and vice versa; whereas testing with multiple variables like math, social studies, and science would equal very low internal consistency)

Front 40

test-retest reliability and parallel forms reliability measure the ...., while internal consistency methods (split-half and alpha) measure ...

Back 40

consistency of scores over time

consistency among the items within a test at a single point in time

Front 41

tests designs to facilitate a comparison of an individual's performance with that of a norm group are called...

Back 41

norm-referenced tests (NRTs)

Front 42

In a norm-referenced test, if an examinee takes a test and earns a percentile rank of 64, the examinee knows that she scored...

Back 42

higher than 64% of the individuals in the norm group

Front 43

tests designed to measure the extent to which individual examinees have met performance standards are called...

Back 43

criterion-referenced tests (CRTs)

Front 44

On a criterion-referenced test, interpretation of an examinee's test score is _______ of how other students perform.

Back 44

independent

Front 45

NRTs are intentionally built to be of _____ difficulty. Specifically, items that are answered correctly by about ____% of the examinees in tryouts during test development are favored.

Back 45

medium difficulty
50%

Front 46

If the purpose is to describe specifically what examinees know and can do, ______ tests should be used.

Back 46

criterion-referenced tests

example: have students mastered the essentials of simple addition

Front 47

If the purpose is to examine how a local group differs from a larger norm group, _____ tests should be used.

Back 47

norm-referenced tests.

example: How well are students in New York doing in reading in comparison to the national average.

Front 48

An achievement test measures...

Back 48

knowledge and skills individuals have acquired.

Front 49

An aptitude test is designed to...

Back 49

predict some specific type of achievement.

Front 50

The most widely used aptitude tests typically have _____ validity and _____ reliability.

Back 50

low to modest validity (validity coefficients of about .20 to .60) and high reliability (reliability coefficients of .80 or higher)

Front 51

An intelligence tests is designed to...

Back 51

predict achievement in general, not any one specific type.

Front 52

The most popular intelligence tests are (2) things.

Back 52

1) culturally loaded
2) measure knowledge and skills that can be acquired with instruction (with questions such as, "How far is it from N.Y. to L.A.)

Front 53

Name 3 basic approaches to reducing social desirability in participants' responses in order to measure without the influence of social desirability.

Back 53

1) administering personality measures anonymously
2) observe behavior unobtrusively (without the participants' awareness--to the extent this is ethically possible.
3) projective techniques (which have controversial validity problems, time consumptions, etc)

Front 54

Scales that have choices from "strongly agree" to "strongly disagree" are known as...

Back 54

Likert-type scales

Front 55

true experimental designs are characterized by

Back 55

random assignment to treatments

Front 56

pretest sensitization, or reactive effect of testing

Back 56

changes in the experimental group may be the result of a combination of being exposed to the pretest and the treatment

Front 57

posttest-only randomized control group design

Back 57

an experiment without a pretest (to make-up for the pretest sensitization)

Front 58

Solomon randomized four-group design

Back 58

2 pretest groups (treatment, control group)
2 no-pretest groups (treatment, control group)
to determine if pretest sensitization effected the experiment
(probably need 48 or more people in sample to divide into 4)

Front 59

explanations for changes other than the treatment are...

Back 59

threats to internal validity

Front 60

What is "history" in regard to a threat to internal validity?

Back 60

other environmental influences on the participants between the pretest and posttest (reading a self-help book during the same time period)

Front 61

What is "maturation" in regard to threat to internal validity?

Back 61

participants matured during the period between pre and post-test, and the increase is due to maturation and not the treatment.

Front 62

What is "instrumentation" in regard to threat to internal validity?

Back 62

possible changes in the instrument (measurement procedure) from the time it was used as a pretest to the time it was used as a post-test. (different observers, for example)

Front 63

What is "statistical regression" in regard to threat to internal validity?

Back 63

occurs only if participants are selected on the basis of their extreme scores (if you select only the really low scores, on a post-test they will score higher purely because of the nature of random errors)

Front 64

What is "intact groups" in regard to threat to internal validity?

Back 64

individuals for groups aren't selected at random - they are left in "intact groups" (UI student group vs. WSU student group). (indicated by a dotted line between groups)

Front 65

What is "selection" in regard to threat to internal validity?

Back 65

When researchers do not assign participants to the 2 groups at random, there is a very strong possibility that the two groups are not initially the same in all important respect. Selection can interact with all other threats to internal validity (selection-history interaction, selection-maturation interactions, etc)

Front 66

all threats to internal validity can be overcome using a...

Back 66

true experimental design, in which participants are assigned at random to experiment and control conditions.

Front 67

When asking if a results generalize, the researcher is essentially asking what?

Back 67

Is it accurate to assume that the treatment administered to the experimental group will work as well in the population as it did in the sample?

Front 68

What are some threats to external validity? (how well it generalizes)

Back 68

selection bias
reactive effects of experimental arrangements
reactive effect of testing (pretest sensitization)
multiple-treatment interference

Front 69

What is selection bias (in regards to being a threat to external validity) and how can it be controlled?

Back 69

a biased sample of participants; control this threat by selecting participants at random

Front 70

What is reactive effects of experimental arrangements? control for it?

Back 70

if the experimental setting is different from the natural setting in which the population usually operates, the effects that are observed in the experimental setting may not generalize to the natural setting.
Control: conduct experiments under natural conditions, when possible

Front 71

What is reactive effect of testing? also known as pretest sensitization?

Back 71

the possibility that the pretest might influence how the participants respond to the treatment

Front 72

What is multiple-treatment interference?

Back 72

when a group of participants is given more than one treatment; treatments given earlier in an experiment might affect the effectiveness of treatments given later to the same participants.

Front 73

internal vs. external validity

Back 73

internal: "Is the treatment in this particular case response for the observed changes?"

external: "To whom and under what circumstances can the results be generalized?"

Front 74

pre-experimental designs

Back 74

limited value for investigating cause-and-effect relationships because of their poor internal validity

Front 75

list 3 pre-experimental designs

Back 75

one-group pretest-posttest design
one-shot case study
static-group comparison design

Front 76

what is the one-shot case study?

Back 76

one group is given a treatment (X) followed by a test (O)

X O

Front 77

static-group comparison design

Back 77

two groups but participants are not assigned to the groups at random (the dashed line between the two groups indicates they are intact groups)

X O
--------
O

Front 78

When it is not possible to assign participants at random, pre-experimental designs should be AVOIDED. Instead, _______ designs should be used, which are...

Back 78

quasi-experimental designs, which are of intermediate value for exploring cause-and-effect

Front 79

Two widely used quasi-experimental designs are...

Back 79

nonequivalent control group design
equivalent time-samples design

Front 80

What is the nonequivalent control group design?

Back 80

two intact groups (not assigned at random as indicated by the dashed line):
O X O
--------
O O

(even though not assigned at random, researchers often use some sort of matching to increase the internal validity)

Front 81

What is the equivalent time-samples design?

Back 81

has only one group; treatment conditions are alternated (preferably on random-basis)

X0O X1O X0O X1O X1O
(major disadvantage=multiple treatment interference)

Front 82

Hawthorne Effect

Back 82

"attention effect"

(lights in factory, whether down or up, performance went up because participants knew research were paying attention to them)

Front 83

How can a researcher control for the Hawthorne Effect?

Back 83

Use 3 groups:
1) experimental group
2) control group that receives attention
3) control group that receives no attention

Front 84

What is the John Henry Effect?

Back 84

refers to the possibility that the control group might become aware of its "inferior" status and respond by trying to outperform the experimental group.

Front 85

descriptive statistics vs. inferential statistics

Back 85

descriptive: summarize data so they can be easily understood

inferential: draw inferences about the effects of sampling errors on the results that are described with descriptive statistics

Front 86

Statistics come from ______.
Parameters come from ______.

Back 86

Statistics come from Samples.
Parameters come from Populations.

Front 87

significance tests determine the probably that...

Back 87

the null hypothesis is true

Front 88

Suppose a researcher conducted a significance test and found that the probability of the null hypothesis is a correct hypothesis is less than 5 in 100. How would this be stated?

Back 88

p < .05, where p stands for the word probability

Front 89

An alternative way to say that a researcher has rejected the null hypothesis is to state that the difference is...

Back 89

statistically significant (if a researcher states that a difference is statistically significant at the .05 level, it is equivalent to stating that the null hypothesis has been rejected at that level.

Front 90

What are the 4 scales (or levels) at which researchers measure?

Back 90

nominal
ordinal
interval
ration

(noir)

Front 91

What is the nominal level?

Back 91

the "naming" level
(words, not numbers)
For example: male, female, married, single, etc.

Front 92

What is the ordinal level?

Back 92

the measurements places participants in order from high to low (putting in order doesn't say the difference between them)

Front 93

What is the interval scale?

Back 93

has equal distances but no absolute zero (a zero on a multiple-choice exam doesn't mean the person has 0 knowledge)

Front 94

What is the ratio scale?

Back 94

has equal distances and an absolute zero (for example, weight)

Front 95

univariate analysis

Back 95

how participants vary (hence the root variate) on only ONE variable.

Front 96

bivariate analysis

Back 96

examines relationship between two nominal variables (for example, whether there is a relationship between student gender and preference for a candidate)

Front 97

What is chi-square?

Back 97

The usual test of the null hypothesis for differences between frequencies, in other words, used to obtain the probability that the null hypothesis is correct.

Front 98

Chi-square and degrees of freedom are NOT descriptive statistics. They can be thought of as sub-steps in the mathematical procedure for obtaining the value of >>>, which

Back 98

p, which if p is less than .05, the null hypothesis can be rejected. (if p > .05, the null hypothesis can NOT be rejected)

Front 99

Type I Error vs. Type II Error

Back 99

(RA)
(R) Type I: REJECTING the null hypothesis when it should have been accepted.
(A) Type II: ACCEPTING the null hypothesis when it should have been rejected.

Front 100

positive vs. negative skew of distribution

Back 100

positive: tail on right
negative: tail on left

Front 101

mean
median
mode

Back 101

mean=average
median=middle score
mode=most frequently occurring score

Front 102

a synonym for the term averages is...

Back 102

measures of central frequency

Front 103

the standard deviation describes ______.

Back 103

variability

Front 104

The most widely used coefficient is the...

Back 104

Pearson product-moment correlation coefficient, usually called Pearson r

Front 105

direct(positive) relationship vs. inverse (negative) relationship

Back 105

direct(positive): as one goes up, the other goes up
indirect(negative): as one goes up, the other goes down

Front 106

coefficient of determination

Back 106

to think about correlation in terms of percentages, square pearson's r.

Example.
If r=.50, the coefficient of determination would be .25; which means that .50 is 25% better than a Pearson r of 0.00

Front 107

The "t test" is often used to test...

Back 107

the null hypothesis regarding the observed difference between TWO means

Front 108

statistically significant indicates that the...

Back 108

null hypothesis has been rejected.

Front 109

Name 3 things that would cause a t test to yield low probability?

Back 109

sample size: the large the sample, the lower the value of p.
size of the difference between means: the larger the difference, the less likely the difference is due to sampling errors
the amount of variation in the population: the smaller the variability, the lower the p.

Front 110

An alternative test (to the t test) to test the null hypothesis between two sample means is...

Back 110

the analysis of variance, the ANOVA.

Front 111

Instead of t (like with the t test) the ANOVA yield a statistic called

Back 111

F (as well as degrees of freedom, sum of squares, mean square, and a p value). As with the t test, the only value of interest to the typical consumer is the value of p.

Front 112

By convention, when p equals .05 or less, researchers ______ the null hypothesis and declare the result to be ______ _______.

Back 112

reject the null hypothesis
statistically significant

Front 113

t-test vs. ANOVA

Back 113

t-test can compare only two means; a single ANOVA can compare a number of means.

When two means are compared, both will result in the same p -value.

Front 114

meta-analysis

Back 114

set of statistical methods for combining results of previous studies.